Schema-bound extraction
schema-bound-extraction · v1
Pull a nested JSON object with an enum, a nullable field, and an ISO date. Extra prose fails the JSON check. "N/A" is null. The gold object is an exact match.
Bench / Episodes / Schema extraction, loop bug & long-thread summary · 4 models
Navigator Bench episode
Schema extraction, loop bug & long-thread summary · 4 models
Schema-bound extraction, Explain a subtle loop bug, Bat and ball, Long support-thread summary · openai/gpt-4o-mini vs google/gemini-2.5-flash vs google/gemini-3.1-flash-lite vs deepseek/deepseek-chat-v3.1
As of Oct 1, 2026, 8:47 PM ET (runs stored from Oct 1, 2026, 8:46 PM ET). 16 stored runs. Methodology
A rank on this page is best_fit_for_inputs for one task, volume, and budget. It is not a universal winner.
Tasks
Explain a subtle loop bug
subtle-loop-bug · v1
Explain a loop that walks one past the end of an array and then adds undefined. The failure mode is NaN, not a single undefined return. This is a different snippet from the off-by-one in bug-explanation.
Bat and ball
bat-and-ball · v1
A classic price trap. The intuitive answer is $0.10. The amount that satisfies both sentences is $0.05. The check is exact JSON, so a prose answer fails even when the number is right.
Long support-thread summary
long-thread-summary · v1
Summarize a long ticket. The refund amount and the April 3 photo condition appear only in the last message, and they conflict with earlier figures. A summary that stops early will miss them. This fixture is long on purpose. It is not a measured score.
Models: openai/gpt-4o-mini, google/gemini-2.5-flash, google/gemini-3.1-flash-lite, deepseek/deepseek-chat-v3.1
Methodology: v1
Published:
Candidate runs total: 0.00595749
Judge cost: 0.0015292
Total (candidate + judge): 0.00748669
Amounts are the exact stored decimal strings. No rounding is applied. candidate_cost_usd is the sum of measured run costs. judge_cost_usd is the sum of stored tier-2 judge ledger rows, or null when none were recorded. total_cost_usd equals candidate plus judge only when every candidate cost is present and a judge cost was recorded.
Candidate costs are the measured amounts on the receipts below (16 runs).
Comparison 7696b7c0-7884-49a9-9ec6-3a230f1ac943
Funding visitor-key / video-demo
Stored runs
| Model | Task | Tier | Score | Latency | Measured cost | Receipt |
|---|---|---|---|---|---|---|
| google/gemini-2.5-flash | Schema-bound extraction | Deterministic check | 500 ms | 0.0002034 | Run c0c30fcf | |
| openai/gpt-4o-mini | Schema-bound extraction | Deterministic check | 434 ms | 0.00004605 | Run feaa6e8b | |
| google/gemini-3.1-flash-lite | Schema-bound extraction | Deterministic check | 775 ms | 0.0001235 | Run 7230f7a6 | |
| deepseek/deepseek-chat-v3.1 | Schema-bound extraction | Deterministic check | 1191 ms | 0.00008221 | Run a4223a89 | |
| openai/gpt-4o-mini | Explain a subtle loop bug | LLM judge | 419 ms | 0.000117 | Run 90b26224 | |
| google/gemini-3.1-flash-lite | Explain a subtle loop bug | LLM judge | 535 ms | 0.00042025 | Run c84be2b5 | |
| google/gemini-2.5-flash | Explain a subtle loop bug | LLM judge | 730 ms | 0.0005938 | Run 87540701 | |
| deepseek/deepseek-chat-v3.1 | Explain a subtle loop bug | LLM judge | 396 ms | 0.00019331 | Run 1251ccce | |
| google/gemini-2.5-flash | Bat and ball | Deterministic check | 484 ms | 0.000072 | Run fe9c3687 | |
| google/gemini-3.1-flash-lite | Bat and ball | Deterministic check | 800 ms | 0.00004025 | Run 1649b8c0 | |
| openai/gpt-4o-mini | Bat and ball | Deterministic check | 1086 ms | 0.0000177 | Run 5d90bb44 | |
| deepseek/deepseek-chat-v3.1 | Bat and ball | Deterministic check | 373 ms | 0.00002881 | Run 709b8a52 | |
| google/gemini-3.1-flash-lite | Long support-thread summary | LLM judge | 626 ms | 0.0010485 | Run 9129eec0 | |
| google/gemini-2.5-flash | Long support-thread summary | LLM judge | 1152 ms | 0.0015609 | Run 163cf4bf | |
| openai/gpt-4o-mini | Long support-thread summary | LLM judge | 4/5 · mixedconflict 3 · coverage 5 · late_fact 3 · faithfulness 4Mixed resultHuman spot checkMethodology | 1045 ms | 0.000447 | Run 335158dc |
| deepseek/deepseek-chat-v3.1 | Long support-thread summary | LLM judge | 431 ms | 0.00096281 | Run afdd4ed3 |
Notable flags
- openai/gpt-4o-mini on Long support-thread summary. 4/5 · mixed.Mixed result: conflict, late_fact scored 3 or lower.
B-roll
Race, radar, run replay, fit walkthrough, PNG stills, and CSV stay marked todo. The links below are the artifacts that exist for this comparison.
JSON packet · Scatter · Checklist · Episode packet · Race (not generated)
Measured cost vs quality
Open in the interactive Bench · Comparison page · Episode packet · Checklist