Coding AI Benchmark Template: Record Tasks, Results and Costs
A reproducible exercise, fair-run rules, rubric and CSV record sheet—not unrun DeepSeek, GPT, Claude or Gemini rankings.
On this page
No measured model scores are published below. Generate the exercise or choose your own tasks, then record real runs before comparing. Fixture self-checks only establish that the exercise works.
Start with a reproducible repair exercise
Download the script and run it in a new local directory. It creates dsh-cart-task with defective code, 10 tests, a bilingual task brief and a blank run record. It uses no network, model calls or dependency installation, and refuses to overwrite an existing folder.
Download the exercise generator ↓
node create-cart-task.mjs
cd dsh-cart-task
node --test cart.test.mjsThis exercise requires Node.js on your computer. That is a prerequisite for this script, not a claim that DSH Desktop requires a separate Node.js installation. Inspect the downloaded script before running it.
- Save the baseline result: 6 passing and 4 failing tests are expected. The initial failure is the deliberate task defect, not an installation failure.
- In DSH, select the generated folder and an available model. Start a fresh session and provide the task in TASK.md. Allow changes to cart.mjs only; do not permit test changes.
- After the task, rerun the tests yourself and inspect the file changes. Acceptance requires 10 passing tests with the tests unchanged. A model's claim of success is not test output.
- After the run, fill in run-record.json. Leave unknown model IDs, DSH versions, timings or costs as null, not zero. Generate a clean directory for another model.
On 2026-09-30, using Linux / Node.js v22.16.0, the baseline had 6 passes and 4 failures; the one-line reference repair passed all 10 tests, and duplicate creation was refused. No DSH session or paid model was run, and no model cost or speed was measured.
Read the fixture validation record (includes the reference repair) →This small public task only checks the task–tools–tests–recording workflow. It is too simple to represent overall coding capability or rank models. Do not provide the reference repair to the model during a comparison.
Freeze variables before comparing models
Same repository baseline
Start from the same commit or clean fixture; do not inherit another model's patch.
Same execution environment
Fix DSH version, mode/preset, workspace, tools, permissions, OS and Node.js version.
Same task and prompt
Keep files, prompt and acceptance criteria identical and define them before running.
Fresh session every run
Do not inherit a previous model's session context, settings or conclusions.
Fixed resource limits
Define time, retries, network and tool permissions in advance; do not relax them for one model.
Record all outcomes
Include failures, timeouts and manual takeovers. Disclose repeat counts and sample limitations.
Use representative tasks, not one tiny exercise
| Task type | Acceptance focus |
|---|---|
| Bug fix | Reproducible failure and objective tests |
| Targeted refactor | Behavior unchanged and regression tests passing |
| Test generation | Known behavior with missing coverage |
| Multi-file change | Small feature and bounded scope |
| Multimodal task | Specified visual input and acceptance criteria |
Three tasks × three repeats × two models means 18 runs and is only a starting point. Set a budget first and do not generalize from a single success.
Use one standard task prompt
Repository baseline: <BASE_COMMIT>
Task: <TASK>
1. Inspect the repository and relevant tests first.
2. Modify only necessary files; preserve public APIs unless authorized.
3. Do not delete, skip or weaken tests.
4. Run acceptance tests; state accurately when a test cannot run.
5. Report changed files, test results, known risks and acceptance.
Acceptance criteria: <ACCEPTANCE_TESTS>Keep the 100-point rubric separate from acceptance
First require the specified tests to pass and forbidden files to remain unchanged. Then score quality. A composite score cannot override a correctness or safety failure.
| Criterion | Points | Standard |
|---|---|---|
| Functional correctness | 40 | Required tests and acceptance checks pass. |
| Regression safety | 20 | Relevant existing tests still pass. |
| Scope control | 15 | Only necessary files and behavior change. |
| Tool efficiency | 10 | Purposeful calls without repeated unrelated loops. |
| Code quality | 10 | Maintainable changes consistent with the repository. |
| Final report accuracy | 5 | The report matches the actual patch, tests and limitations. |
| Total | 100 | Does not replace acceptance gates |
Cost per accepted task includes failed runs and human review
Human review cost = review minutes / 60 × hourly rate
Total cost = API + infrastructure + human review cost
Cost per accepted task = cost of all model runs / accepted runs
Pass rate = accepted runs / all runsWhen accepted runs = 0, accepted-task cost is undefined. Missing costs are not zero, and failed-run costs must not be discarded. Keep calculator estimates separate from actual provider billing.
Download the run-record CSV template
Record model ID, provider, DSH version, mode, reasoning setting, base commit, acceptance, score, time, tool calls, retries, tokens and costs. Blank rows below are illustrative, not measured runs.
Download the Coding AI Benchmark CSV template ↓
| Model ID | Accepted | Score | Time | API $ | Review min |
|---|---|---|---|---|---|
| DeepSeek — exact ID | — | — / 100 | — | — | — |
| GPT — exact ID | — | — / 100 | — | — | — |
| Claude — exact ID | — | — / 100 | — | — | — |
| Gemini — exact ID | — | — / 100 | — | — | — |
Conclusions apply only to the recorded conditions
Check acceptance rate first, then compare cost and time at comparable quality. Disclose sample sizes, failures, retries and manual takeovers. Different workloads may favor different configurations; do not announce an unsupported universal winner.
FAQ
Why are no model scores published here?
This page has no published site-run DSH model comparison. Blank values mean unrecorded, not zero. Fixture validation is not a model test.
How many runs should I do?
Three to five per task is a starting point when budget allows, not a statistically sufficient sample. Expand tasks and repeats for close or variable results.
Can I compare different harnesses?
Yes, but the result compares complete systems rather than models alone. Record each harness version, tools and permission differences.
What if a model has zero accepted runs?
Cost per accepted task is undefined, not zero, because the denominator is zero. Record the failed outcomes.
What if API cost was not recorded?
Leave it null or unrecorded. Do not present estimates as measured costs or use missing data for a cost comparison.
Method references and sources
These sources explain versioning, environments and acceptance methods. They are not sources of site-run scores. Check the relevant project revision and do not combine scores obtained under different conditions.