Navigator Bench
Schema extraction, loop bug & long-thread summary · 4 models
Rankings are for one task type, volume, and budget. They are not a universal winner.
Every cost on this page is a measured OpenRouter amount stored on a run. A blank cost was not reported. Estimates are not shown. Published comparisons · This comparison · Episode
Amounts are the exact stored decimal strings. No rounding is applied. candidate_cost_usd is the sum of measured run costs. judge_cost_usd is the sum of stored tier-2 judge ledger rows, or null when none were recorded. total_cost_usd equals candidate plus judge only when every candidate cost is present and a judge cost was recorded.
Candidate costs are the measured amounts on the receipts below (16 runs).
Depth at a glance
Active library: 16 tasks across 6 categories (coding, extraction, reasoning, summarization, support, writing). This comparison stores 4 tasks and 4 models. Last run .
Library size is the task list. Scores and costs below come only from stored runs.
Where models differed: On Long support-thread summary, models differed on conflict: google/gemini-3.1-flash-lite 5; google/gemini-2.5-flash 5; openai/gpt-4o-mini 3; deepseek/deepseek-chat-v3.1 5. Receipt 9129eec0 · Receipt 163cf4bf · Receipt 335158dc · Receipt afdd4ed3
Stored runs
| Model | Task | Tier | Score | Latency | Measured cost | Receipt |
|---|---|---|---|---|---|---|
| google/gemini-2.5-flash | Schema-bound extractionv1 | Deterministic check | 500 ms | 0.0002034 | Run c0c30fcf | |
| openai/gpt-4o-mini | Schema-bound extractionv1 | Deterministic check | 434 ms | 0.00004605 | Run feaa6e8b | |
| google/gemini-3.1-flash-lite | Schema-bound extractionv1 | Deterministic check | 775 ms | 0.0001235 | Run 7230f7a6 | |
| deepseek/deepseek-chat-v3.1 | Schema-bound extractionv1 | Deterministic check | 1191 ms | 0.00008221 | Run a4223a89 | |
| openai/gpt-4o-mini | Explain a subtle loop bugv1 | LLM judge | 419 ms | 0.000117 | Run 90b26224 | |
| google/gemini-3.1-flash-lite | Explain a subtle loop bugv1 | LLM judge | 535 ms | 0.00042025 | Run c84be2b5 | |
| google/gemini-2.5-flash | Explain a subtle loop bugv1 | LLM judge | 730 ms | 0.0005938 | Run 87540701 | |
| deepseek/deepseek-chat-v3.1 | Explain a subtle loop bugv1 | LLM judge | 396 ms | 0.00019331 | Run 1251ccce | |
| google/gemini-2.5-flash | Bat and ballv1 | Deterministic check | 484 ms | 0.000072 | Run fe9c3687 | |
| google/gemini-3.1-flash-lite | Bat and ballv1 | Deterministic check | 800 ms | 0.00004025 | Run 1649b8c0 | |
| openai/gpt-4o-mini | Bat and ballv1 | Deterministic check | 1086 ms | 0.0000177 | Run 5d90bb44 | |
| deepseek/deepseek-chat-v3.1 | Bat and ballv1 | Deterministic check | 373 ms | 0.00002881 | Run 709b8a52 | |
| google/gemini-3.1-flash-lite | Long support-thread summaryv1 | LLM judge | 626 ms | 0.0010485 | Run 9129eec0 | |
| google/gemini-2.5-flash | Long support-thread summaryv1 | LLM judge | 1152 ms | 0.0015609 | Run 163cf4bf | |
| openai/gpt-4o-mini | Long support-thread summaryv1 | LLM judge | 4/5 · mixedconflict 3 · coverage 5 · late_fact 3 · faithfulness 4Mixed resultHuman spot checkMethodology | 1045 ms | 0.000447 | Run 335158dc |
| deepseek/deepseek-chat-v3.1 | Long support-thread summaryv1 | LLM judge | 431 ms | 0.00096281 | Run afdd4ed3 |
Notable flags
- openai/gpt-4o-mini on Long support-thread summary. 4/5 · mixed.Mixed result: conflict, late_fact scored 3 or lower.
Measured cost vs quality
Fit for your inputs
Label: best_fit_for_inputs. The top rank is for this task, volume, and budget. It is not a universal winner. Methodology
Choose a task, volume, and budget to rank the stored runs for that task.