AI agents falsified 49 papers at ICML: research validation is now an

AI agents independently reproduced 34% of all papers at ICML 2026—2,200 papers in 19 days—with 1,200 participants. The positive headline: a narrow majority of papers had at least one major claim verified, with 266 fully reproduced by multiple independent agents. This is the largest systematic reproduction effort ever attempted for a major AI conference, and

the organizers produced fully auditable logs using tracko, enabling review at the code, trace, and artifact level. The more telling signal is the failure rate. 496 papers (23% of those attempted) had at least one claim falsified or contested. 49 papers were essentially fully falsified, with all major claims shown to be problematic. Notably, several

authors confirmed the issues after reviewing the audit logs and are issuing corrections. This suggests the problem is not merely theoretical—some accepted papers contain genuine errors that peer review missed. The key operational insight: human-in-the-loop agents outperformed fully autonomous agents on complex verification tasks. The winning entries combined agent automation (quantization, proof checking) with human

judgment (visual evaluation via custom UIs). This mirrors emerging production patterns where agent-assisted quality assurance proves more reliable than pure automation. However, the headline figure of 34% coverage is inflated by selection bias—participants chose easier papers, and failed attempts were less likely to be logged. The 49 fully falsified papers likely represent the floor, not