Scaling Holds, Evals Are Broken, and the Engineer’s Role Is Shifting

Mark Chen, OpenAI’s research chief, flatly rejects the ‘pre-training is dead’ narrative, insisting scaling laws have held for 10 orders of magnitude. Engineering and data innovations repeatedly break bottlenecks. For leaders, this means compute demand and model capabilities will keep growing—any assumption of a plateau is premature.

He warns of an evals crisis: benchmarks are saturated, and overfitting is easy. OpenAI separates eval and model teams for adversarial measurement. For any engineering org, relying on public benchmarks risks deploying brittle models. Custom, task-specific evals, regularly refreshed, are essential to measure real-world capability.

He describes a shift to ‘vibe researching,’ where models handle implementation while researchers provide direction and taste. Similar to AI coding tools, senior engineers will increasingly act as reviewers and architects rather than coders. Teams must hire for design taste and system thinking, not just implementation speed. The bottleneck is moving from production to curation.

OpenAI manages research via directive compute allocation: a few high-conviction bets per org get dedicated resources, with flexible pools for exploration. This portfolio approach forces regular hard calls to kill failing projects—a discipline many teams lack. The lesson: allocate aggressively to big bets but build a culture that rapidly disengages when evidence turns negative.