Hugging Face's open challenge set 1,221 participants and their agents on the papers the machine-learning conference ICML had accepted in 2026; among the refutations it confirmed is a spotlight paper whose robustness proof gives way at a step entrants pinpointed.
At least one claim was falsified or contested in 496 of the papers put through the ICML 2026 Open Reproductions challenge — 23% of those examined.
In machine learning the conference is where work is published and vetted, so an ICML acceptance carries the weight a journal's does elsewhere. The challenge ran from July 15 to August 2, 2026, and was open to anyone, each entrant free to choose their own agent framework and their own compute.
Within that group, 49 papers lost every claim with nothing verified at all, and on 242 papers independent reproduction teams reached opposite verdicts on the very same claims.
The other side of the ledger is larger. For 1,103 papers — 51% of those examined — an independent reproduction confirmed at least one claim: 266 came through entirely, every claim extracted from them standing up, and 632 reproduced only in part but with nothing falsified. A further 502 produced evidence that went no further than toy scale, and 280 could not be settled in either direction, usually because the artifacts needed to reproduce them were missing.
The volume being audited has outrun the capacity to check it. ICML drew 23,918 submissions and accepted 6,352 papers, about twice the previous year's figure, while reviewing at most conferences falls to unpaid volunteers who may lack either the hours or the specialist knowledge, and the pool able to review has not grown with the submissions. One accepted spotlight paper drew strong scores alongside a review that said, "My low confidence score is because I did not check all the proofs carefully."
That paper, Towards Optimal Robustness in Learning-Augmented Paging, is among the confirmed refutations. One entrant's logbook tracked the additive error term climbing and pinned down precisely which step of the proof gives way, and a re-implementation by the organizers pushed the sweep out to k = 1,024, where the growth held at about nine sigma.
Not every claimed falsification survived. Thirty-five entrants filed formal claims that a result had been disproved, and each went through a hostile re-check: the paper read again beside the logbook, the mathematics worked through afresh or the experiment rebuilt from what the paper itself described. One logbook reported a method running 2x slower than its baseline, but the fault was a miscalculation inside the reproduction, which set time per trajectory against time per batch of 50; put on a common footing, that participant's own figures bore out the 8x speedup the paper had claimed.
Agents left running unsupervised fell into local loops and misjudged behavior that depends on scale — several "verified" rulings on the paging paper came from checks that halted before the growth could show — and now and then assembled an entire falsification out of mismatched units. Results held up best where a person stayed at the wheel, redirecting the agent, pushing back on an assumption, or ruling an experiment's premise faulty before any compute went into it.
Authors are writing back. A theorem in Attention's forward pass and Frank-Wolfe — that token particles converge on the origin whenever the origin sits inside their convex hull to begin with — drew counterexamples from three separate teams, the first violations surfacing at t = 224, around t = 3,800, and t = 6,416 steps, which accounts for earlier checks stopping at a limited horizon and coming back as confirmations. Those authors acknowledged the counterexample the day it turned up and are putting together a fix.
Organizers set about contacting the authors behind every confirmed finding, laying out the evidence and asking whether they agreed or thought the analysis mistaken. When the challenge's results were released, authors had accepted what had been found on several papers, two corrections were being prepared for arXiv, and every logbook, verdict, trace and artifact from the exercise had been published.
Cover image: “In Peer Review We Trust” by Sarahmirk, Wikimedia Commons, CC BY-SA 4.0, resized, re-encoded.
