Navigator Bench
Bug explanation & support reply · 4 models
Rankings are for one task type, volume, and budget. They are not a universal winner.
Every cost on this page is a measured OpenRouter amount stored on a run. A blank cost was not reported. Estimates are not shown. Published comparisons · This comparison · Episode
Amounts are the exact stored decimal strings. No rounding is applied. candidate_cost_usd is the sum of measured run costs. judge_cost_usd is the sum of stored tier-2 judge ledger rows, or null when none were recorded. total_cost_usd equals candidate plus judge only when every candidate cost is present and a judge cost was recorded.
Candidate costs are the measured amounts on the receipts below (16 runs).
Depth at a glance
Active library: 16 tasks across 6 categories (coding, extraction, reasoning, summarization, support, writing). This comparison stores 4 tasks and 4 models. Last run .
Library size is the task list. Scores and costs below come only from stored runs.
Where models differed: On Tone-matched support reply, models differed on completeness: google/gemini-2.5-flash 3; openai/gpt-4o-mini 5; google/gemini-3.1-flash-lite 5; deepseek/deepseek-chat-v3.1 5. Receipt 0923ba40 · Receipt ad86ac26 · Receipt 63b9f0b5 · Receipt 532ae9ba
Stored runs
| Model | Task | Tier | Score | Latency | Measured cost | Receipt |
|---|---|---|---|---|---|---|
| google/gemini-3.1-flash-lite | Explain a bug in plain languagev1 | LLM judge | 629 ms | 0.00025025 | Run c619851e | |
| google/gemini-2.5-flash | Explain a bug in plain languagev1 | LLM judge | 1412 ms | 0.0005352 | Run 1ae2b197 | |
| openai/gpt-4o-mini | Explain a bug in plain languagev1 | LLM judge | 1585 ms | 0.0000957 | Run bc686e25 | |
| deepseek/deepseek-chat-v3.1 | Explain a bug in plain languagev1 | LLM judge | 530 ms | 0.00015081 | Run bf682c4a | |
| google/gemini-2.5-flash | Tone-matched support replyv1 | LLM judge | 4/5 · mixedaccuracy 5 · tone_match 5 · completeness 3 · policy_safety 5Mixed resultHuman spot checkMethodology | 470 ms | 0.0001416 | Run 0923ba40 |
| openai/gpt-4o-mini | Tone-matched support replyv1 | LLM judge | 535 ms | 0.00005445 | Run ad86ac26 | |
| google/gemini-3.1-flash-lite | Tone-matched support replyv1 | LLM judge | 810 ms | 0.00014075 | Run 63b9f0b5 | |
| deepseek/deepseek-chat-v3.1 | Tone-matched support replyv1 | LLM judge | 209 ms | 0.00008731 | Run 532ae9ba | |
| google/gemini-3.1-flash-lite | Ambiguous ticket replyv1 | LLM judge | 846 ms | 0.00017125 | Run 1f9abacf | |
| google/gemini-2.5-flash | Ambiguous ticket replyv1 | LLM judge | 435 ms | 0.0004158 | Run 45335487 | |
| openai/gpt-4o-mini | Ambiguous ticket replyv1 | LLM judge | 1158 ms | 0.00008265 | Run e33803c8 | |
| deepseek/deepseek-chat-v3.1 | Ambiguous ticket replyv1 | LLM judge | 1545 ms | 0.0001539 | Run 187597dc | |
| google/gemini-2.5-flash | Conflicting status updatev1 | LLM judge | 465 ms | 0.0000941 | Run 293f17b9 | |
| google/gemini-3.1-flash-lite | Conflicting status updatev1 | LLM judge | 484 ms | 0.00006775 | Run 260c89f2 | |
| openai/gpt-4o-mini | Conflicting status updatev1 | LLM judge | 3/5 · mixedaudience 5 · word_cap 5 · fact_coverage 5 · conflict_honesty 0Mixed resultHuman spot checkMethodology | 463 ms | 0.0000348 | Run 45ce890c |
| deepseek/deepseek-chat-v3.1 | Conflicting status updatev1 | LLM judge | 339 ms | 0.00005227 | Run e635c9ee |
Notable flags
- google/gemini-2.5-flash on Tone-matched support reply. 4/5 · mixed.Mixed result: completeness scored 3 or lower.
- google/gemini-3.1-flash-lite on Ambiguous ticket reply. 4/5 · mixed.Mixed result: tone scored 3 or lower.
- google/gemini-2.5-flash on Ambiguous ticket reply. 4/5 · mixed.Mixed result: tone scored 3 or lower.
- openai/gpt-4o-mini on Ambiguous ticket reply. 4/5 · mixed.Mixed result: tone scored 3 or lower.
- openai/gpt-4o-mini on Conflicting status update. 3/5 · mixed.Mixed result: conflict_honesty scored 3 or lower.
Measured cost vs quality
Fit for your inputs
Label: best_fit_for_inputs. The top rank is for this task, volume, and budget. It is not a universal winner. Methodology
Choose a task, volume, and budget to rank the stored runs for that task.