AI agents win the speedrun, but fail at discovery

The experiment. Prime Intellect pitted Claude Code and Codex against human researchers in an optimizer speedrun benchmark. The goal was to achieve a target validation loss in the fewest training steps by modifying only optimizer-related parameters. The environment, based on the nanoGPT speedrun community, provided fast iteration, clear rules, and a verifiable reward signal. Agents were given access to compute clusters, a scratch pad, and the ability to fetch the latest human records.

The results. Both codegen models outperformed the best human record within days. Codex was more reliable, running continuously for days, while Claude Code stalled every 9–10 hours claiming the task was too hard. Codex consumed more tokens and spawned more sub-agents. Claude wrote enthusiastic notes with emoji; Codex's logs were robotic. The models achieved a 50–60 step improvement over the human record, which the presenter called both impressive and limited.

What the agents did not do. Despite beating humans on the speedrun metric, the agents did not discover any novel optimizer or mechanism. They combined existing papers, applied clever tricks, and made incremental improvements. The presenter was explicit: there was no breakthrough invention. This is the key tension in the talk. The models were effective at optimization within known search spaces, but failed at the kind of creative discovery that AI research promises.

The benchmark gap. The experiment lacked the structure of a proper scientific benchmark. There was no multi-seed testing, no standardized conditions across models, and the researchers restarted agents when human records improved. The team is now building a more rigorous evaluation with three tracks: no external access, archive papers only, and full access. Initial results from this new framework show Gemini being surprisingly competitive, with step-function improvements rather than gradual progress.

The proposed fix. The speaker argues that discovery requires a multi-agent system inspired by AlphaEvolve, not a single coding agent. The system would include generators (both closed and open-source models), a speedrun evaluator for immediate reward, a judge with learned taste, and a human steering loop. The goal is to scale ideas from the speedrun environment to larger parameter counts. This is speculative; the team has not yet tested it.

What engineering leaders should watch. The speedrun environment is a good evaluation and training ground for AI research agents, but its value for discovery is unproven. The models excelled at tactical optimization within constrained rules. The missing capability is strategic invention. This mirrors a broader pattern: AI coding agents are good at finding the local optimum in a well-defined space, but poor at redefining the space itself.