Context Bloat Will Drain Your AI Budget—Fix It with Just-in-Time Tools

The talk’s main signal is that attaching MCP servers “just in case” creates hidden cost and latency. A demo shows a prompt using 20 tokens jumps to 3,500 tokens when three unused MCP servers are connected—a 175x increase. This context bloat affects every call, draining budgets and increasing reasoning time without added value.

The proposed fix combines two infrastructure patterns: an MCP gateway that filters tools by identity, reducing the exposed tool surface for each caller, and an MCP registry that lets agents discover relevant tools at runtime. Together, they shift from static tool attachment to dynamic, just-in-time provisioning.

However, the evidence is entirely demo-scale. There’s no production data, no cost comparisons, and no discussion of failure modes. Identity propagation for agents (on-behalf-of) remains incomplete. The registry’s semantic search is currently rudimentary—regex-based in the Anthropic implementation—and embedding-based improvements are still speculative.

For engineering leaders, the takeaway is that architecture, not prompting, will control token economics. Investing in API-level governance now can prevent tool bloat as MCP adoption scales. The gateway-registry pattern mirrors service-mesh lessons, but it’s early; treat the approach as a promising direction, not a proven solution.