Hugging Face Reproduced 2,200 ICML Papers. Here’s What They Found

In July 2026, Hugging Face asked a simple, uncomfortable question: if you actually tried to reproduce every paper from a major AI conference, what would you find? Over 19 days, 1,221 community members provided an answer — and it wasn’t entirely reassuring.

The ICML 2026 Open Reproductions challenge ran from July 15 to Aug. 2, pairing human participants with their own coding agents — tools like Claude Code, Codex, Cursor and OpenResearch’s orx — to verify the specific scientific claims in accepted papers from the International Conference on Machine Learning. By the time the challenge closed, participants had published 6,816 Trackio logbooks covering 2,226 papers, roughly 34% of the entire conference. An automated judge running on GLM-5.2, an open-weights model, then evaluated each logbook and issued per-claim verdicts: verified, falsified, toy-scale or inconclusive.

What the Numbers Actually Say

The headline results are mixed. Fifty-one percent of examined papers had at least one claim independently verified — 266 were fully reproduced with every extracted claim confirmed, and 632 more were partially reproduced with nothing falsified. That adds up to 3,978 individual claims confirmed through real experiments across 35,908 total judgments.

But 23% of examined papers had at least one claim falsified or contested. Forty-nine papers had all of their claims falsified with nothing verifiable. Perhaps most striking: 242 papers received opposite verdicts from different independent reproduction teams, which Hugging Face frames not as a methodological flaw but as evidence that reproducibility is inherently adversarial — the same paper can look solid or shaky depending on how carefully you look and for how long.

Several of the confirmed falsifications are documented in detail. One accepted spotlight paper — notable because its reviewer explicitly wrote, “My low confidence score is because I did not check all the proofs carefully” — contained a flaw in a core mathematical proof about a paging algorithm’s robustness guarantee. A participant’s logbook measured an additive term growing logarithmically rather than staying constant; Hugging Face’s own re-implementation extended the analysis to k = 1,024 and confirmed the discrepancy at roughly nine sigma. In a separate paper on attention mechanisms, three independent teams found counterexamples to a claimed theorem — but only after hundreds of simulation steps, explaining why shorter-horizon checks had missed them entirely. A third paper’s theory section analyzed reverse KL divergence while the released code, which produced all the paper’s actual results, computed forward KL instead. The authors confirmed the issue and uploaded a corrected version to arXiv.

Why the Conference System Is Under Pressure

The scale of ICML 2026 itself is part of the story. The conference received 23,918 submissions and accepted 6,352 papers — roughly double the previous year. Reviewers are volunteers, and their capacity hasn’t scaled with submission volume. The same AI tools making it faster to run experiments and write papers are also, at least in theory, available to help check them. Hugging Face’s argument is that coding agents can attempt a paper reproduction in an afternoon that would cost a human reviewer a weekend — and can do it in parallel, thousands of times over.

This effort is structurally different from earlier reproducibility initiatives. Conference reproducibility checklists, like those introduced at NeurIPS and ICML in recent years, rely on authors self-reporting. The Hugging Face challenge placed the burden on the broader community, using agents to independently verify claims after publication. No comparable effort — not Papers With Code, not OpenReview, not the ML Reproducibility Challenge — has attempted agent-powered, claim-level verification across an entire major conference.

What This Means If You’re a Student or Early-Career Researcher

The barrier to entry for this hackathon was low: a free Hugging Face account, a coding agent (Claude Code and Codex are accessible to students at low or no cost), and $20 in HF compute credits that Hugging Face provided to each participant. Every contributor with at least one verified logbook received a participation certificate — a concrete portfolio artifact. The 274 published full agent-trace datasets and thousands of logbooks now function as a free, searchable library showing exactly how to implement and test the claims in specific papers — useful for thesis work, capstone projects or self-directed ML study.

There’s also a skills lesson embedded in how the challenge went wrong. Agents working alone made characteristic mistakes: misreading units, terminating experiments before scale-dependent effects appeared, and occasionally constructing confident falsifications on top of simple conversion errors. In one documented case, a participant claimed a 2x slowdown that turned out to be a units mismatch — per-trajectory versus per-batch-of-50 — and the participant’s own data actually confirmed the paper’s claimed 8x speedup. The reproductions that held up best came from human-agent pairs where the human was actively steering, catching errors and making qualitative judgment calls the agent couldn’t.

That gap — between an agent that can run experiments and a person who can tell when the experiment is asking the wrong question — is where practical skill in AI research lives right now. Learning to supervise, correct and critically evaluate agent-run experiments isn’t just good scientific practice. It’s increasingly what research teams are looking for.

Read the full Hugging Face writeup here.

Source: Hugging Face