Context compaction is a trap when caching is cheap

Towards AI's experiments on their AI tutor reveal a counterintuitive finding: not compacting context often outperforms compaction. The team tested sliding windows, summarization, selective retention, and full history across Gemini and DeepSeek models. Full history consistently achieved higher recall (95% vs 32% for compacted) and lower cost per turn due to aggressive prompt caching.

Caching discounts of up to 50x mean that resending the same context is far cheaper than regenerating summaries. Compaction destroys cache locality, forcing the model to re-process new tokens at full price. The result is higher latency, more tool calls, and worse memory recall.

The tradeoff shifts when context exceeds local hardware limits. On a MacBook, the 32K window forced compaction, lowering recall to 33%. For documents, hybrid retrieval (dense + BM25) maintained 100% accuracy, outperforming pure semantic search at scale.

Teams should not default to compaction. Instead, measure cache hit rates, use cheaper models with generous caching, and only compact when context windows are genuinely constrained by hardware or API limits. The real bottleneck is workflow design, not model quality.