How Navigator Bench stays independent

Bench runs are paid from AI Navigator’s own monthly evaluation budget. The engine refuses any comparison whose estimated cost would go past what is left of the monthly cap, and pauses a comparison that starts costing far more than estimated.

Deterministic checks decide a run when a task has a single right answer. Where judgment is needed, two judge models score the answer against a written rubric, and a judge never scores a model from its own provider. There is one judge per family (Anthropic, OpenAI, Google); each answer goes to the two families other than its maker’s:

Judges are blinded: model and lab names are removed from the answer they read. If the two judges disagree, a person decides the run, and nothing is published while such a run is undecided. 5% of the runs both judges agreed on are sampled for a person to audit. Where a task carries known good and known bad answers, the judges grade those inside every comparison too, and a judge that grades one wrongly blocks the comparison from being published.

Judge models on this server: Anthropic: anthropic/claude-sonnet-5.5; OpenAI: openai/gpt-6.1-sol; Google: google/gemini-3.8-flash. Every judge score records the judge model it came from.

Before results are released, a person grades a sample of about 50 stored answers blind: no model name, no judge scores. We then measure how often the judges’ pass/fail calls match the person’s, with Cohen’s κ alongside. A release is only possible when that agreement is at least 85% (our launch bar), and every release publishes its calibration result: overall, per judge family and per difficulty tier.

No release has been published yet, so no calibration result is public yet.

To show what a good answer costs without AI, we pay a customer-support contractor to answer the same tickets the models answer. They are paid an hourly rate that we publish with the results, and a ticket’s cost is that rate × the time they actively spent on it (the page counts time only while a ticket is open and they are working). They see exactly what the models see: the same public tickets and the same instructions. They never see the held-out tickets. Their answers go through the same automatic checks and the same two judges, and they appear as a “Human (contractor)” reference line next to the models, never ranked against them. We store only an alias for them, nothing personal.

Errors are corrected publicly. Originals are kept and marked, never silently rewritten or rescaled. A published release cannot be edited: a fix is a new release that supersedes the old one, both stay public, and the old one is clearly marked “superseded by” the new one. Every correction is logged with what was wrong, what changed and which releases it affected, and a published correction is never deleted; later edits are kept as revisions.

Corrections log · Changelog · Releases · Methodology

AI Navigator earns affiliate commissions on some tool links elsewhere on the site. Bench pages, Bench API responses and Bench downloads carry no affiliate links. An automated test renders every Bench page and scans every Bench API response and download for affiliate links and tracking markers; any hit fails the test suite that every change has to pass.