Agent RL: Real Work, Real Infra, Custom Evals

The central argument here is a paradigm shift: we’re no longer just training models with RL on single-turn Q&A tasks (like math problems); we’re training *agents* that operate in multi-step, stateful environments over long horizons. This mirrors how real software work happens—coding, debugging, deploying over hours or days. The talk dissects the four-layer stack needed: 1) **Environments**, which provide tasks, state management, and reward signals (heuristics or LLM-as-judge rubrics), are proliferating from Nvidia’s Nemo Gym to Hugging Face Spaces. The catch is most are still toy environments, not full replicas of Excel or enterprise tools. 2) **Training frameworks** are shifting to asynchronous architectures (e.g., Mistral’s model) to avoid GPU idle time when rollouts vary from minutes to hours. This decoupling of generation from training is critical for horizon scaling. 3) **Evaluations** are fundamentally broken. Public benchmarks saturate in months, are susceptible to reward hacking, and don't capture real-world, long-horizon capability. The advice is blunt: teams must build proprietary evals that match their actual tasks. 4) **Scaffolds**—like recursive self-aggregation or parallel-sequential hybrid inference—are still underexplored but show that training *with* the scaffold outperforms static model prompting. However, the claim that scaffolds will be “washed away by scale” is a live debate. The real signal for engineering leaders is that cost and domain specialization are now defensible reasons to train your own agent, as post-trained small models (like Cursor’s Composer) can match frontier performance at a fraction of the token cost. The technique is accessible but the details are sparsely documented, mostly locked in Chinese tech reports, creating an open recipe vacuum.