Public leaderboards are a starting point. The questions I want a benchmark to answer are closer to our daily work: can a model write a React form that passes its Playwright test, implement a C# endpoint that honours its contract, write a Delphi unit test that compiles, or identify a fault from a device log?
Measurement status: This article describes a proposed benchmark and its scoring method. The percentages in an earlier version were illustrative placeholders, not measurements, and have been removed. Measured results are not yet published; this article does not establish a model ranking or a performance-based routing decision.
The seven tasks
The proposed suite covers seven kinds of work. Each needs a fixed set of cases and a scoring rule agreed before models are compared. Case counts should come from the versioned dataset used for an actual run.
| Task | How it would be scored |
|---|---|
| Build a React form from an OpenAPI schema, with a Playwright test | The submitted test and independent acceptance tests pass against the mock API; lint is clean |
| Implement a C# endpoint from an OpenAPI operation | Independent contract tests pass and the configured analyzers report no violations |
| Write unit tests for a Delphi unit | Tests compile and pass against the original unit, and fail against deliberately broken versions |
| Translate a Firebird query to Transact-SQL | Both engines return equivalent results on fixed fixtures, including edge cases |
| Find the root cause in a device log | The known cause appears in the top three suggestions, supported by the cited log lines |
| Triage a support ticket | Category and routing match the reviewed reference labels |
| Explain a legacy module | Developers rate correctness, completeness and supporting code references against a shared rubric, without seeing the model name |
Potential cases include forms and endpoints from our API descriptions, units from the codebase, queries from the migration, and resolved incidents and tickets with customer details removed. Development examples and the held-out evaluation set need to stay separate. Changing a prompt after inspecting a failed evaluation case should not turn that same case into fresh evidence of generalisation.
The seven tasks are deliberately separate. A combined score could hide a model that is useful for ticket routing but unreliable when explaining code. Avalonia and firmware are part of the wider agentic coding workflow; they are not measured by this proposed suite, so these tasks would not justify claims about those stacks.
The harness
A .NET console application running in Azure DevOps would fit this design. For each case, it would save the input, prompt version, model identifier, request settings, response, tool calls, scorer output, token usage and elapsed time. Each record needs a run ID so that a published total can be traced back to individual cases.
The initial protocol would run each case three times with the same task instructions and tool access. Record the actual settings supported by each model, including reasoning and output budgets, rather than describing different defaults as equivalent. Pin model versions where the provider supports it and record the evaluation date.
Report pass and fail counts for each run, along with variation between runs. Keep those repetitions linked to their original cases: repeating a case does not create a new independent case. Latency can be summarised with a median and a tail measure; cost should be calculated from recorded usage and the pricing in effect on the run date, including retries.
For coding tasks, score the first attempt separately from an attempt with compiler, analyzer or test feedback. Give every model the same retry limit and record the extra time and cost. Whether feedback helps, and by how much, is a question for the measurements.
Choosing models to compare
The comparison should include OpenAI and Anthropic models that are candidates for the same work. Each row in a future report needs the exact model identifier and configuration. A provider name or a broad model family is not enough to reproduce a run.
Choose candidates and acceptance criteria before seeing the scores. A new version gets a new evaluation record. It should not silently replace the version behind an earlier result.
What a results report must contain
There is no measured results table here yet. A report would need the following evidence before percentages or rankings are published:
- Dataset: version, task definitions, case counts and exclusions.
- Run configuration: date, model versions, prompts, settings, tool access and retry budget.
- Outcomes: per-case scorer records, pass and fail counts for each task and run, and examples of important failures.
- Human scoring: the explanation rubric, number of reviewers, individual ratings and disagreements.
- Resource use: recorded token usage, actual calculated cost and latency, with first attempts and retries distinguished.
- Uncertainty: variation across cases and runs, and the limits of the comparison.
Until that evidence is available, claims that one model leads on Delphi, that smaller models are close enough on triage, or that a retry loop closes the gap remain hypotheses. The earlier draft's claims about score changes after a model update and results-driven routing are also withdrawn.
How the results could guide routing
The intended use is to choose a model for each task against an agreed quality threshold, then consider cost, latency and review effort among the candidates that meet it. A cheap answer that needs extensive correction may not be cheap in practice.
The same evaluation could help assess upgrades. Compare a candidate version with the current version on the same held-out cases, inspect regressions, and make the decision from the recorded outcomes. No particular routing choice or upgrade is established by the design alone.
Questions for you
- Which tasks would you put in your own benchmark, and which can be scored without a human?
- What evidence do you require before a model update enters a development workflow or product?
- How do you account for retries and human review when comparing costs?
Email me at bjorn.nostdahl@nostdahl.com. I would like to compare notes on task selection and scorers.