Most AI coding benchmarks measure the wrong thing.
HumanEval tests function completion. SWE-bench tests bug repair in an existing repo. Both treat the codebase as a given — the scaffolding is already there, and the model fills a gap. That's useful, but it's not software engineering. It's autocomplete on steroids.
RepoZero, from a joint PKU and Baidu team, tests something genuinely harder: given only API documentation and a handful of example tests, can a model build the entire repository from nothing? No starter code. No existing structure. Just a spec and a blank directory.
The answer, for every model tested including frontier ones, is: not reliably.
What Is RepoZero
The benchmark frames the task as repository reproduction. The setup is:
- Take a real, existing open-source repository
- Strip all code, leaving only API documentation and sample I/O tests
- Ask the model to reconstruct the entire repository from scratch
- Validate with a hidden test suite — output must match the original exactly
That last point matters. The validation isn't "does this look reasonable" or "does an LLM judge think it's correct." It's execution-based: the generated code runs, and its outputs are compared against the ground truth byte-for-byte. Either it passes or it doesn't.
This is a significant step up in rigor from most existing benchmarks. LLM-judged evaluations have well-documented reliability problems. Execution-based validation doesn't.
The Cross-Language Challenge: Py2JS and C2Rust
To prevent models from simply regurgitating memorized GitHub code, RepoZero introduces a clever constraint: generate the repository in a different language than the original.
Two tracks:
- Py2JS — Reproduce a Python library's logic in JavaScript
- C2Rust — Reproduce a C/C++ codebase in Rust
The rules are strict: no calling the original language's runtime, no bridge tools, no external dependencies. The model must actually understand the algorithm and re-express it in the target language's idioms. A Python sorting library full of list comprehensions and generator patterns needs to become idiomatic JavaScript — not a mechanical transliteration.
This design choice is smart. It forces genuine comprehension over pattern matching, and it makes data contamination far less likely. Even if a model has memorized the Python source, that doesn't help it write the JavaScript implementation.
ACE: The Self-Testing Loop
Alongside the benchmark, the paper proposes Agentic Code-Test Evolution (ACE) — a method for improving generation quality through iterative self-verification.
The loop runs as follows:
- Generate initial code from the API specification
- Generate tests derived from the same spec
- Execute the tests against the generated code
- Analyze failures — where does the code diverge from spec?
- Fix the code based on error analysis
- Repeat until tests pass or iteration limit is reached
This is a meaningful departure from one-shot generation. The model acts as its own QA engineer, surfacing its own bugs and correcting them before submission.
The results support this approach: adding ACE consistently raises pass rates across all models tested. How much depends on the model and task, but the direction is unambiguous — the ability to self-verify and self-correct is a real capability multiplier.
Benchmark Results: Even Top Models Struggle
The numbers are sobering.
On the hardest RepoZero tasks, top frontier models — including Claude Sonnet 4.6, GPT-4o, and various open-source competitors — achieve pass rates in the 20–40% range. Some simpler tasks see higher rates; the hard end of the distribution is brutal.
To put that in perspective: a developer handed the same API spec and told to implement it would realistically hit near 100% given enough time. The gap between human and model performance here isn't a rounding error — it's a yawning chasm.
ACE narrows the gap meaningfully. Multi-round code-test iteration can push pass rates up several percentage points on mid-difficulty tasks. But even with ACE, the hardest tests remain largely unsolved.
What It Means for Developers
The paper identifies four main failure modes, and they're instructive:
Long-context forgetting. A 20-file repository has a lot of state to track. Models lose the thread — a type defined in utils.py is handled inconsistently by the time the model writes core.py. The model technically "knows" the spec, but can't maintain coherent application of it across thousands of lines.
Module coordination errors. Function A's output format doesn't match what Function B expects as input. In a single-function benchmark, this never comes up. In a multi-module repository, it breaks things everywhere.
Output inconsistency. Non-deterministic behavior, race conditions in async code, floating-point handling — the generated code produces different outputs on repeated runs. The hidden test suite catches this; one-shot human review often doesn't.
Test coverage gaps. The model's self-generated tests (in the ACE loop) miss edge cases that the benchmark's hidden test suite covers. The model essentially tests for the happy path and misses the boundaries.
If you're building AI coding tools, these failure modes are a useful taxonomy. They suggest that raw generation power matters less than: (1) maintaining long-range coherence across files, (2) enforcing interface contracts between modules, and (3) adversarial self-testing that goes beyond the obvious cases.
Bottom Line
RepoZero is a well-designed benchmark for a real capability that existing evaluations under-measure. The execution-based validation is credible. The cross-language design prevents memorization artifacts. The ACE framework is a practical method worth exploring in production agent workflows.
The 20–40% ceiling on today's best models isn't a reason to dismiss AI coding tools — it's a reason to be precise about what they can and can't do. They're good at filling gaps. They're not yet good at building systems.
The gap is real, it's now measurable, and it's the right thing to be working on.
Resources
- RepoZero Homepage — benchmark overview and leaderboard
- GitHub Repository — code, datasets, and evaluation scripts
- arXiv Paper — full technical report
RepoZero is open-source and available at github.com/JesseZZZZZ/RepoZero. The benchmark and evaluation framework are publicly accessible.
