Navigator Bench methodology

This page describes two versions of the methodology.

Methodology v2: the Navigator Index (Index v2.0, draft). This is what the Bench board shows: one score per model across private tasks in eight categories, with ranks the data can actually support.
Methodology v1 (classic). It scored the earlier task-by-task comparisons in the classic view. It is kept so those results can still be read; the Navigator Index does not use it.

Methodology v2 · current

The Navigator Index (Index v2.0, draft)

Each run gets a score from 0 to 1 (partial credit). A task's score is the mean over all its runs (every input and repeat). Category score = 100 × the mean task score over the category's active tasks. Navigator Index = the weighted mean of the eight category scores (weights above, summing to 100).

Coding · weight 15
Data analysis · weight 15
Extraction · weight 10
Reasoning · weight 15
Research & agentic · weight 15For now these are single-call multi-step tasks: one model call returns plan → steps → deliverable, checked step by step. Tool-using, browsing and long agentic tasks are deferred.
Summarization · weight 10
Support · weight 10
Writing · weight 10

Ranks are paired: every pair of models is compared on the same tasks (an exact sign-flip test of the Index difference), with Holm's correction over all pairs so the chance of any false "win" on the board is at most 5%. Each model shows the range of ranks the data allows (best = 1 + models significantly better; worst = number of models − models significantly worse) and is marked tied when that range is not a single rank.

A model is shown as dominating another only when it is significantly better on the Index (paired) AND costs less per successful task. The headline follows the same rule. The task bank is private: prompts, data and answer keys are not published, so they cannot be trained on. Inputs and outputs of runs in a published cycle are shown on their receipts; keys, hidden tests and check details never are.

Judges (methodology v2)

Two blinded judges from different companies than the model's own (Anthropic, OpenAI, Google); disagreements go to a person; known-good and known-bad sentinels catch a misgrading judge. A judge never scores a model from its own company. How judges are assigned.

Judge models on this server: Anthropic: anthropic/claude-sonnet-5.5; OpenAI: openai/gpt-6.1-sol; Google: google/gemini-3.8-flash. Every judge score records the judge model it came from.

Limits

Five tasks per category is few: a category interval is often ±10–20 points, so category scores are descriptive, not rankings.
At 40 tasks the Index interval is roughly ±4–7 points. Paired ranks separate a 12-point gap almost always and an 8-point gap about half the time; models 4–6 points apart are usually shown as tied.
Tasks are single model calls with the data in the prompt. Tool use, browsing and long agentic sessions are not measured yet.
Judge scores are calibrated against a person but remain judgement, not ground truth.
Prices are OpenRouter list prices for the pinned provider at run time; your own costs may differ.

Calibration (methodology v2)

Under methodology v2, a person grades a sample of stored answers blind, and the judges’ pass/fail calls are compared with theirs (agreement and Cohen’s κ, overall, per judge family and per difficulty tier). A release needs agreement of at least 85%. Only counts are published here, never the graded answers.

No release has been published yet, so no calibration result is public yet.

Methodology v1 · classic comparisons

Navigator Bench methodology v1 (classic)

Everything below applies only to the classic task-by-task comparisons, not to the Navigator Index above.

A v1 ranking answers one question: which model fits best for the inputs you gave it, meaning one task, one monthly volume and one budget. It is not a claim that the model is best in general.

What was tested

The active v1 library has 16 tasks in coding, extraction, reasoning, summarization, support, writing. These are active library definitions. A row here is not a score and not a cost.

Invoice field extraction · extraction · tier 1 · v1
Receipt total extraction · extraction · tier 1 · v1
Support intent classification · support · tier 1 · v1
Repair broken JSON · coding · tier 1 · v1
Meeting action items · summarization · tier 1 · v1
Five-bullet brief · writing · tier 1 · v1
Earnings call summary · summarization · tier 2 · v1
Customer changelog draft · writing · tier 2 · v1
Explain a bug in plain language · coding · tier 2 · v1
Tone-matched support reply · support · tier 2 · v1
Ambiguous ticket reply · support · tier 2 · v1
Conflicting status update · writing · tier 2 · v1
Schema-bound extraction · extraction · tier 1 · v1
Explain a subtle loop bug · coding · tier 2 · v1
Bat and ball · reasoning · tier 1 · v1
Long support-thread summary · summarization · tier 2 · v1

Three tiers

Tier 1, Deterministic check: Scripted checks against the fixture. No model judges the output.

Tier 2, LLM judge: A pinned judge model scores a blinded output against a versioned rubric. Methodology v1 uses a single judge model, google/gemini-2.5-flash-lite, with rubric rubric-v1. The judge model is configuration. It is recorded on each tier-2 score. Changing the judge or the rubric means a new methodology version. This pass does not change either. (Methodology v2 uses the two-judge panel described above instead.)

Tier 3, Human spot check: A person records that they read the output. A note that only restates evaluation status is an automated metadata check, not this tier. A human note is labeled “Human spot check”. A metadata-only note is labeled “Automated metadata check”.

Dimension scores and the overall

In short: the judge scores each answer on several dimensions and gives an overall from 0 to 5. If any dimension scores 3 or lower, the overall we display is capped at 4 out of 5 and labeled “mixed”, even when the judge’s stored overall is 5. The stored score is never rewritten.

The exact rule: The judge returns an explicit overall from 0 to 5 under rubric-v1. That stored overall is kept. If any dimension is 3 or lower and the stored overall is above 4, the displayed overall is capped at 4. The row is labeled a mixed result. A displayed 5/5 is allowed only when every stored dimension is above 3 and the output is complete. Fit quality uses this displayed overall. Judge model and rubric-v1 are unchanged.

Dimension scores are shown next to the overall on receipts and comparison tables.

Incomplete outputs

Flags are derived from the stored output, provider finish_reason when it was stored, and the max-token budget. The output text is not edited.

Fields: output_incomplete, completeness_confidence, completeness_reason. An incomplete output is marked needs review and is not shown as a clean 5/5.

Cost

Amounts are the exact stored decimal strings. No rounding is applied. candidate_cost_usd is the sum of measured run costs. judge_cost_usd is the sum of stored tier-2 judge ledger rows, or null when none were recorded. total_cost_usd equals candidate plus judge only when every candidate cost is present and a judge cost was recorded.

Public names: candidate_cost_usd, judge_cost_usd, total_cost_usd.

Known limits (v1)

Judge scores are comparative signal, not ground truth.
A displayed 5/5 is not a universal winner and is not a claim that the output was flawless if a completeness flag is set.
A ranking is the best fit for the inputs given (one task, one monthly volume, one budget), not the best model in general.
Token-size hints on tasks are not prices. Measured costs come from stored OpenRouter amounts.
AI Navigator does not pay for v1 runs (its v1 budget is $0), so a featured v1 comparison only runs on an API key that the person requesting it brings and pays for.

Changelog (v1)

2026-09-30. Methodology v1 approved. Tiers, fit weights, and rubric-v1 pinned. Judge id is configuration, recorded on each score.
2026-10-01. Display rule added without a new judge or a new rubric. Stored overalls are not rewritten. If any dimension is 3 or lower, a stored 5 is shown as 4 and labeled mixed. Completeness flags are derived from the stored output. Metadata-only spot-check notes are labeled Automated metadata check. Public costs split candidate runs, judge calls, and the total.

These are the v1 entries. The full changelog adds every methodology v2 change and every release; errors are logged in the corrections log.

Back to the Bench