Bench / Changelog
Navigator Bench
Changelog
Every change to how Bench scores models, and every published release, newest first. Methodology entries are the records kept with each methodology version; nothing here is rewritten after the fact. Errors are logged separately in the corrections log.
Methodology v2
Navigator Index (Index v2.0, draft weights) added; Support suite results unchanged. Every run now records a partial-credit score from 0 to 1 (deterministic: pass/fail or the weighted share of scored checks; judged: the mean of (overall − 1) / 4 over the two judges; hybrid: wD·D + wJ·J, default 0.6 / 0.4; multi-step: w·deliverable + (1 − w)·steps completed). A task without its own pass rule keeps the existing rule; Index tasks declare one (score thresholds, check groups, judge average on the 1–5 scale, veto dimensions). New check types: numeric tolerance, set F1, Kendall tau, grid cells, length caps and hidden unit tests in a locked code sandbox (a sandbox failure is an infrastructure error, never the model's). Index = weighted mean of 8 category scores (100 × mean task score); 95% intervals from a stratified task bootstrap (B = 2,000, small-sample correction); paired ranks (every pair compared on the same tasks, sign-flip test with Holm’s correction, so any false win is at most 5%; each model shows its rank range); dominance only when significantly better and cheaper per successful task. Monthly cycles publish immutable releases; a model added mid-cycle supersedes the release. The Index task bank is private.
Methodology v2
Amended before first use (still no v2 runs or publications). Clustered intervals: repeats of one input and inputs of one task are correlated, so the suite and difficulty-tier 95% intervals are now a percentile cluster bootstrap over tasks (B = 2,000; seed = sha256 of "navigator-bench-v2/bootstrap/1|<comparison id>|<model entry>|<scope>|<slice>"), widened where needed to contain the Wilson interval on the effective sample size; per-task intervals are Wilson on n_eff = n / (1 + (m − 1) · ICC), m = mean decided repeats per input, ICC = the model entry's pooled within-input one-way ICC across its tasks, bounded to [0, 1] (1 when it cannot be estimated). Tie bands and the cheapest-model lower-bound flag use these intervals; the naive Wilson interval is kept for reference. Passes every time (pass^k): an input passes every time when all of its repeats have a pass/fail verdict and all pass; inputs with any error, judge_error, contested or pending repeat are reported as not fully decided and left out; reported as k of N inputs (Wilson at task level, the task-cluster bootstrap above), next to pass rate everywhere, with an optional "passes every time ≥ X" filter on the cheapest-model bar (off by default). Effort rows: a lineup entry is (model, reasoning effort), so the same model at two efforts is two rows with separate results ("· low effort", "· high effort"); runs never pool across efforts. Mistral Large 4.0 (only "none" or "high") runs as both.
Methodology v2
Amended before first use (no v2 runs or publications existed). Temperature 0.3 where the model supports it; otherwise the model's default, recorded (temperature null, temperature_mode "model_default"). Reasoning effort is pinned per model in the registry; the standard lineup uses low (Mistral Large 4.0, which accepts only high or none, runs with none). Requests send only parameters the pinned endpoint lists in OpenRouter's supported_parameters, so require_parameters stays on. max_tokens = the task's answer budget + a reasoning allowance, so reasoning cannot starve the answer; an answer emptied or cut off by reasoning is INCOMPLETE. Reasoning effort and reasoning tokens are recorded per run and per judge score; cost stays the measured provider cost, which includes reasoning. Judges reason at low by default (BENCH_V2_JUDGE_<FAMILY>_EFFORT) and get JSON mode only where supported. Pinned providers are OpenRouter provider slugs.
Methodology v2
Methodology v2 engine. Repeats (5) × variants (3 public by default) per model and task, temperature 0.3, pinned OpenRouter providers, a two-judge cross-family panel with contested-run review, a deterministic 5% agreed-error audit, judge sentinels, Wilson intervals, tie bands, cost per successful task, and held-out fixtures kept out of git and out of every public response. v1 rows are not rescored.
Methodology v1
Display rule added without a new judge or a new rubric. Stored overalls are not rewritten. If any dimension is 3 or lower, a stored 5 is shown as 4 and labeled mixed. Completeness flags are derived from the stored output. Metadata-only spot-check notes are labeled Automated metadata check. Public costs split candidate runs, judge calls, and the total.
Methodology v1
Methodology v1 approved. Tiers, fit weights, and rubric-v1 pinned. Judge id is configuration, recorded on each score.