Stop Prompting: Build an Environment, Train a Small Agent in Hours

This session shifts focus from reward functions to the environments that agents act within, presenting a practical framework for post-training agents using OpenM and TRL. The key signal is that environments are far more versatile than most teams assume; they are not just for reinforcement learning but can power evaluation, distillation, and even supervised fine-tuning.

The speaker introduces a clear capability cycle where a discovered capability moves through harness, benchmark, environment, and training, emphasizing that building an environment should be an early step for any task. Two practical experiments demonstrate this: training a small 1.6B model on basic Python problems using a simple run-python tool, and training an 8B model

on the open-code coding harness using a capture proxy to integrate with TRL. The former shows rapid reward improvement within 30 minutes, validating that small models can be quickly specialized. The latter shows reward climbing from near zero to one in about 10 training steps, indicating that even complex coding harnesses are viable for targeted

post-training. The tradeoff between white-box and black-box environment integration is a central operational insight. In white-box mode, the trainer owns the loop, ideal for simple tasks. Black-box mode, using a capture proxy, allows integration with complex existing harnesses like open-code. A critical warning: reward hacking is a real and hard problem, with no silver bullet