Your Model’s Best Feature Won’t Survive a Bad Harness

Open-source models are only 4 months behind the frontier, but progress isn't just about model intelligence. Without reasoning, performance plateaued; now doubling time is 3.5 months. Yet real-world accuracy regresses from serving issues—incorrect prompts, deleted traces, flawed quantization—not model weakness.

One striking example: Claude Code suffered a multi-week accuracy dip because the harness erased the reasoning trace on the second call. Similarly, human evaluators note that dynamic quantization—smartly choosing which layers to compress—can recover accuracy, but poor quantization or tool-calling loops in smaller models degrade output.

The model itself is only one piece. Treating it as a replaceable component and investing in monitoring, harness correctness, and quantization strategy yields stabler, cheaper results than chasing benchmark scores. Today’s open-source catch-up leans on distillation and RL, but operational excellence—not waiting for the next release—is the long-term lever.