How to improve coding agents without golden answers or regression

Sam Denton presents Applied Compute's framework for continual learning using a distillation spectrum. The key insight is that distillation can operate across four quadrants defined by offline/online traces and offline/online hinting, each serving different enterprise maturity levels.

The most practical finding is that offline traces with offline hints can improve specific agent behaviors without degrading base performance. Applied Compute demonstrated this by teaching a Qwen model to submit SWE-bench tasks faster, increasing task completion from 22% to 60% with no regression.

For production systems, online hinting outperforms offline hinting significantly. When teaching a coding agent to format hyperlinks in a specific way, online hints achieved 80% compliance versus offline hints which barely moved the metric. The tradeoff is that online hinting requires a replayable environment or live production traffic.

The engineering challenge is that most enterprises can't provide golden answers for distillation. Applied Compute's approach uses LLM judges to create hints dynamically and uses per-step hinting rather than full-rollout distillation. The relevance mask technique prevents catastrophic forgetting by selecting which tokens to learn from.