Agent Harnesses Stop Adding Coding IQ. Now They Manage Cost and Risk.

The argument. For coding evals, the agent harness no longer adds intelligence. Jediah Katz, who worked on Cursor's agent harness and now works at SpaceXAI, cites a UC Berkeley study that ran seven frontier models across harnesses including Codex and Claude Code and found no significant score difference on SWE-bench Light. Apple and EPFL found the same on MLEBench. Prompts and guardrails that once taught a model how to explore a codebase or check its work are now native model behavior.

The flywheel. Katz explains this with a data flywheel. The harness makes harder coding tasks reachable, more users attempt them, their coding data reaches model labs, and the model eventually absorbs the harness's tricks. He says Cursor removed guardrails it used to need, including a small model dedicated to applying code edits and coding-specific instructions. The uncomfortable implication for tool vendors is that the smarter models get, the less differentiation a coding wrapper can claim.

The counterpoint. The harness is not dead, Katz argues, because it shifts to new work. He points to Grokbot, SpaceXAI's assistant for knowledge work, which uses long-lived coordinator agents that orchestrate a fleet of worker agents. There, his team is adding guardrails back for tasks models have not mastered yet, such as making slide decks, booking travel, and deciding when to act versus ask. The flywheel then starts again on a new frontier.

The real constraints. The parts a model cannot absorb are operational. Long-running agents need cloud infrastructure because a laptop going to sleep breaks multi-day tasks. Permissions must be scoped to specific agents and people, and guardrails must stop a catastrophic action or a runaway API bill. Both studies Katz cites found significant cost-per-task differences across harnesses even when scores matched. That turns harness choice into a cost and control decision.

Efficiency work. Katz says his recent focus is Grokbot's harness efficiency: cache hit rate, token efficiency, and model mix. He reports meaningfully lower cost and faster responses with no quality loss, but gives no numbers. For engineering leaders this is the most actionable part of the talk. Agent usage at scale makes harness operations, not model IQ, the main variable a team can actually manage.

What to watch. Katz predicts the product will keep moving faster than the model, harness internals will be hidden from users to reduce change management, and 90% of agent work will run off the laptop within six months. The prediction reads like vendor enthusiasm, but the direction matches what he describes inside Grokbot and Cursor. Teams should watch the shift from "which harness is smartest" to "which harness is cheapest, safest, and easiest to change."