Coding AI Benchmark Template: Compare DeepSeek, GPT, Claude & Gemini in DSH
Benchmark coding models in DSH with the same repository, commit, mode, tools, prompt and acceptance tests; track pass rate, score, retries, time, API cost and human review cost.
On this page
This does not replace SWE-bench or Terminal-Bench. Public benchmarks give standardized reference points; this template answers which model is actually better for your repository, DSH setup and acceptance bar.
Freeze the variables before you compare models
Same repository baseline
Start every run from the same Git commit. Do not let model B inherit model A's patch.
Same DSH version
Record the DSH version and hold mode/preset, workspace, tools, permissions and environment constant.
Same task and prompt
Keep prompt, input files, screenshots/PDFs and acceptance criteria identical.
Fresh session every run
Avoid contaminating the A/B with an existing session's model record or prior context.
Fixed resource limits
Define retries, time limit, network access and tools before the run; do not grant one model extra help midway.
Repeat runs
When budget allows, run each model 3–5 times per task and record both pass rate and median performance.
SWE-bench evaluates real GitHub issues with test-based acceptance, while Terminal-Bench standardizes task environments, timeouts and resources. Both reinforce the same rule: hold the harness and acceptance conditions fixed when comparing models.
Use a small representative task pack, not one cherry-picked bug
| Task type | Suggested constraint | What it tests |
|---|---|---|
| Bug fix | One reproducible failure with objective tests | Can the model isolate the cause and make the smallest safe patch? |
| Targeted refactor | Behavior must remain unchanged | Can it improve structure without expanding scope or breaking tests? |
| Test generation | Known behavior with missing coverage | Can it understand behavior and add useful, non-brittle tests? |
| Multi-file change | Small feature or compatibility change | Can it plan across files without losing consistency? |
| Multimodal task | Optional screenshot/PDF/UI bug | Useful when comparing models that accept visual inputs. |
Start with 3 tasks × 3 repeats × 2 models = 18 runs. If the result is close, expand the task set or model set instead of paying for dozens of runs up front.
Standard benchmark prompt
Replace Task and Acceptance Criteria with your real task. Give every model exactly the same version.
You are participating in a reproducible Coding Agent Benchmark.
Repository base commit: <BASE_COMMIT>
Task: <TASK>
Constraints:
1. Inspect the repository and relevant tests before editing.
2. Do not change public APIs unless the task explicitly requires it.
3. Modify only files necessary to complete the task.
4. Run tests directly relevant to the change; if tests cannot run, state why accurately.
5. Do not delete, skip or weaken tests merely to make the run pass.
6. At the end report: files changed, tests run, known risks and whether acceptance criteria were met.
Acceptance criteria:
<ACCEPTANCE_TESTS>If the task genuinely requires public-API changes, network access or images, state that before the run. Do not relax the rules only after one model gets stuck.
100-point rubric: keep quality separate from acceptance
Acceptance should be decided by objective gates such as required tests passing, forbidden files untouched and no critical review issue. The 100-point rubric differentiates runs that all technically passed.
| Criterion | Points | How to score |
|---|---|---|
| Functional correctness | 40 | Required tests and acceptance checks pass; the task is actually solved. |
| Regression safety | 20 | Existing relevant tests still pass and the change does not introduce obvious breakage. |
| Scope control | 15 | The model changes only what the task requires and avoids unrelated rewrites. |
| Tool efficiency | 10 | Uses tools purposefully, avoids repeated loops and does not waste calls on irrelevant exploration. |
| Code quality | 10 | The patch is maintainable, consistent with the repository and avoids unnecessary complexity. |
| Final report accuracy | 5 | The summary correctly states files changed, tests run, unresolved risks and limitations. |
| Total | 100 | Do not let a composite score override hard acceptance gates. |
The cost metric that matters: accepted-task cost
Human review cost = review minutes ÷ 60 × reviewer hourly rate
Total cost = API cost + sandbox/infra cost + human review cost
Accepted-task cost = total cost across a model's runs ÷ number of accepted runs- Pass rate = accepted runs ÷ total runs.
- If accepted runs = 0, accepted-task cost is not zero; it is undefined because the model did not clear the production bar on that test set.
- When quality is comparable, prefer accepted-task cost over raw token price.
- When engineering review is expensive, human review time can dominate token price.
Download the score sheet
The CSV includes the model, provider, model ID, reasoning setting, DSH version, mode, base commit, accepted flag, 100-point score, wall time, tool calls, retries, tokens, API cost and human review fields.
Download the Coding AI Benchmark CSV template ↓
| Model | Accepted | Score | Time | API $ | Review min | Retries |
|---|---|---|---|---|---|---|
| DeepSeek V4.1 Flash | — | — / 100 | — | — | — | — |
| GPT-6 Astra | — | — / 100 | — | — | — | — |
| Claude Fable 5.1 | — | — / 100 | — | — | — | — |
| Gemini 3.8 Flash | — | — / 100 | — | — | — | — |
Do not collapse the result into one universal winner
Quality winner
Start with pass rate, then compare median rubric score among accepted runs.
Cost winner
Among models that clear your quality bar, compare accepted-task cost.
Speed winner
Use accepted runs only, then compare median wall-clock time.
Stability winner
Look at pass rate, retries, tool loops and variance instead of cherry-picking the best run.
For example, repository-scale batch work may favor a cheap model, the hardest debugging may justify a frontier model, and screenshot-heavy tasks may favor a multimodal model. A production system can have multiple winners by workload.
FAQ
How many runs should I do per model?
Three to five runs per task is a practical starting point when budget allows. It is not statistically definitive, but it is much more useful than judging a model from one lucky or unlucky run.
Why should every model use a fresh DSH session?
DeepSeek Harness records the model used by an existing session. A fresh session helps keep model identity and prior context from contaminating the comparison.
Can I compare different agent harnesses with this template?
Yes, but then you are benchmarking complete systems rather than models. For a model-only comparison, keep DSH version, preset, tools, permissions and workspace fixed.
What if a model has zero accepted runs?
Do not report an accepted-task cost of zero. The denominator is zero, so cost per accepted task is undefined for that test set; operationally, the model failed your acceptance bar.
Should I use public leaderboard tasks?
Public benchmarks are useful reference points, but a production decision is stronger when you also use held-out tasks from your own repository that reflect the work you actually ship.
Why this template does not copy a public leaderboard
One of SWE-bench Verified's strengths is evaluating real software issues in reproducible environments with tests deciding whether a patch resolves the task. Terminal-Bench 2.1 also demonstrates that benchmarks themselves need continuous validation because task definitions, resources and external dependencies can materially change results.
This private benchmark borrows that evaluation discipline without pretending that three to five internal tasks produce a score directly comparable with a public leaderboard.