How to Read AI Benchmarks Reliably: Version, Harness, Effort and Cost
Use real 2026 examples to check benchmark revision, agent harness, tool permissions, reasoning effort, token cost and private-eval design before turning public scores into a model-routing decision.
On this page
Do not rank two models until the benchmark revision, harness, tool permissions and reasoning effort are comparable. A public score measures a model inside a test system. When those conditions differ, the numbers can still be useful evidence, but they are not a clean model A/B result and should not be subtracted as if they shared one scale.
Six checks before a public score becomes a routing decision
Use benchmark tables as evidence, not as a universal ranking. Before you repeat a score, check the experimental setup.
- Same benchmark name is not enough — confirm the exact version.
- Confirm the agent harness or scaffold used to run the model.
- Match reasoning effort, sampling settings and token budgets where possible.
- Check whether tools, browsing, code execution or vision were enabled.
- Separate vendor-reported scores from independent evaluations.
- Measure cost per successfully completed task, not only token price.
Section sources: Artificial Analysis — Intelligence Index v4.3 · DeepSeek API — Change Log · Google DeepMind — Gemini 3.8 Flash model card · Anthropic — Claude Fable 5.1
A benchmark revision can change what a percentage means
Gemini 3.8 Flash is a perfect example: Google's model card reports 89.4% on Terminal-Bench 2.1 but 19.1% on Terminal-Bench 4.0. The model did not suddenly become worse inside the same test; the test changed.
Therefore DeepSeek V4 Pro's 87.9 on Terminal Bench 2.1 cannot be compared directly with GPT-6 Astra's 57.9 on Terminal-Bench 4.0. Writing '87.9 is higher than 57.9' is mathematically true and analytically wrong.
| Example | Score | Why direct comparison fails |
|---|---|---|
| Gemini 3.8 Flash — Terminal-Bench 2.1 | 89.4% | Different task set/version |
| Gemini 3.8 Flash — Terminal-Bench 4.0 | 19.1% | Different task set/version |
| DeepSeek V4 Pro — Terminal Bench 2.1 | 87.9 | Not the same as v4.0 |
| GPT-6 Astra — Terminal-Bench 4.0 | 57.9% | Not the same as 2.1 |
Section sources: Google DeepMind — Gemini 3.8 Flash model card · DeepSeek — V4 Pro GA release · OpenAI — GPT-6 Astra
For agent benchmarks, the harness is part of the measured system
DeepSeek explicitly says its V4 Pro public Code Agent evaluations used DeepSeek Harness Minimal Mode. A coding harness decides how the model sees files, shell tools, edits, tool results and permissions. Change that interface and the same model can produce a different result.
This is why a model leaderboard and an agent-product leaderboard answer different questions. Model quality matters, but the scaffold can amplify strengths, hide weaknesses or introduce its own failures.
Section sources: DeepSeek — V4 Pro GA release · DeepSeek API — Change Log
Reasoning effort, safeguards and token budget can move the score
DeepSeek's published agent setup used max reasoning effort for several evaluations. OpenAI exposes multiple Astra reasoning levels from low through max. Anthropic notes that Fable 5.1 production safeguards can block or reroute some cyber and biology tasks, reducing scores on affected evaluations.
A fair report should therefore name the effort level and deployed configuration. If that information is missing, treat the number as incomplete evidence.
Section sources: DeepSeek — V4 Pro GA release · OpenAI API — GPT-6 Astra model · Anthropic — Claude Fable 5.1
Task economics require cost per accepted task, not token price alone
A model that costs ten times more per output token can still be cheaper per successful task if it finishes in one attempt while a cheaper model loops, retries or requires human repair. Conversely, a slightly weaker cheap model can dominate high-volume workloads when failures are easy to detect and retry.
Artificial Analysis increasingly reports cost per task alongside intelligence. For your own application, include token spend, wall-clock time and human intervention in the same evaluation sheet.
Section sources: Artificial Analysis — Intelligence Index v4.3 · Artificial Analysis — DeepSeek V4 Pro 0813
A small private DSH regression set is more useful than one universal leaderboard
For DeepSeek Harness, build a small private test set from work you actually do. Keep the workspace snapshot and preset fixed, then run each candidate model on the same tasks. Twenty representative tasks are enough to reveal many practical differences that public benchmarks miss.
Track final correctness, number of tool calls, failed commands, human approvals, time to completion, input/output tokens and whether the model recovered after the first error. Repeat the test when a provider silently updates an alias or routes an old model name to a new backend.
- 5 repository-reading / diagnosis tasks
- 5 controlled code-edit tasks with tests
- 5 tool-heavy tasks that require recovery after an injected failure
- 5 long-context tasks using real project documentation
Your eval should identify the cheapest model that reliably clears your quality threshold for each workload. It is completely reasonable to route hard tasks to a frontier model and routine tasks to a cheaper model.
Section sources: DeepSeek API — Models & Pricing · OpenAI API — GPT-6 Astra model · Anthropic — Claude Fable 5.1 · Google DeepMind — Gemini 3.8 Flash model card
Artificial Analysis — Intelligence Index v4.3 · DeepSeek API — Change Log · Google DeepMind — Gemini 3.8 Flash model card · Anthropic — Claude Fable 5.1 · DeepSeek — V4 Pro GA release · OpenAI — GPT-6 Astra · OpenAI API — GPT-6 Astra model · Artificial Analysis — DeepSeek V4 Pro 0813 · DeepSeek API — Models & Pricing