Coding Bench is under active development. Dataset, harness, and leaderboard releases will be announced here.
Most coding benchmarks measure how well a model writes a single function. CodingBench asks a harder question: can an agent take a product spec, work in a real development environment, and deliver a working repository from scratch?
Patch benchmarks assume the project already exists. Function-level suites assume a single turn and a clean sandbox. Neither matches how people actually use coding agents:
Natural language spec → multi-turn implementation → tests → ship.
That workflow — Vibe Coding — is where products live or die. CodingBench is built to measure it.
📄 Full overview: doc/coding-bench-promo-blog.md
Every case traces back to a real, high-quality open-source repository. Requirements, acceptance criteria, and scoring anchor to behavior from production-grade code — not prose a model invented in a vacuum. Real data keeps the distribution honest: messy dependencies, implicit conventions, and engineering trade-offs that only show up in shipped software.
An agent receives a natural-language product document and a starter skeleton. It plans, writes code, runs commands, reads errors, and iterates — the same loop users experience in Claude Code, Codex, OpenCode, and similar tools.
PRD / task description → agent writes code → deterministic tests judge the result
No hidden shortcuts. No pre-built project waiting to be patched. The evaluation scenario and the product scenario are the same shape.
The harness provides a persistent terminal sandbox — real shell, filesystem, and toolchains — and records everything the agent does. Plug in your agent stack of choice. The benchmark measures the agent you ship, in the environment your users use.
CodingBench captures the entire execution trajectory: every command, file edit, and test run from first prompt to final submission.
- Failure attribution — misread spec, wrong architecture, or late integration failure?
- Process quality — debug loops, wasted token spend before meaningful progress
- Training signal — recoverable errors vs. fundamental misunderstanding
When an agent derails, you can see the moment it derailed — not just that it did.
Building hundreds of repo-scale cases by hand is impossible. CodingBench's automated data-generation pipeline extends far beyond publishing a benchmark.
| Output | Value |
|---|---|
| Benchmark cases | Fully automated production from curated open-source seeds — specs, environments, hidden judges, release gates — at scale without manual authoring |
| SFT ground truth | Verified [state + task + feedback] → patch trajectories grounded in real test output, sharing the same distribution as evaluation |
| RL environments | Ready-made envs with defined initial state, shell tool-use action space, and deterministic reward from hidden acceptance tests |
Training data and evaluation data from the same source — closing the train-eval gap in agentic coding.
| Question | What CodingBench gives you |
|---|---|
| Can this model build a project from a spec? | End-to-end success rate on real repo-scale tasks |
| How far did it get before failing? | Milestone-level functional coverage — partial credit for real progress |
| Is it cost-competitive? | Token and wall-clock cost per case |
| Why did it fail? | Full trajectory logs for diagnosis and model improvement |
| Can we train on the same distribution we test? | Pipeline-exported SFT and RL assets |
Scoring is deterministic and black-box — no LLM-as-judge. Hidden tests are injected only after submission, keeping the benchmark fair and the leaderboard defensible.
Dataset, case catalog, harness, and participation instructions will follow at launch.