Credit Karma's production agents show governance is the real bottleneck

The problem. Harish Gaggar, senior staff engineer at Intuit Credit Karma, argues most AI agents fail in production because of missing controls, not missing model quality. His talk describes an internal platform where agents are registered, built through a dev portal, deployed to Kubernetes, and forced through an MCP gateway for authentication and routing. The concrete signal for engineering leaders: the bottleneck is governance and infrastructure, not prompt design.

The architecture. The platform combines LangGraph for agent logic with in-house MCP servers and an MCP gateway. Every agent call gets a JWT, routes through the gateway, and reaches MCP servers only after authentication. Tools are reviewed by security before production. No direct access to endpoints is allowed. Gaggar also adds workload identity, secret management, memory tables, and trace storage via SQLite locally and Cloud Spanner in production.

The workflow. Agents classify requests as read, modify, or sensitive operations. Read paths execute automatically; write or delete paths require human approval. Every decision lands in audit logs. Tracking and eval tables store queries, responses, model, token cost, latency, safety, and quality scores per run, which lets teams build ROI dashboards across hundreds of agents. This is the part most engineers miss: agent quality is a data problem, not a demo problem.

The evidence. The strongest number is token consumption. A JFrog Artifactory query used 24,000 tokens, then 2,400 with the context optimizer, a 90% cut. A Slack bot query dropped from 24,000 to 1,500. Gaggar claims typical savings of 70 to 78%. This is the most actionable piece, but it is a demo, not a benchmark. No production cost data, no team productivity numbers, no response quality comparison after pruning are shown.

The trade-off. The proposed model is heavy. A dev portal that generates repositories, security reviews every tool, a central gateway, mandatory HITL, and per-run eval tables require platform team investment and slow iteration. For a small startup this is overkill. For an enterprise with hundreds of agents, the alternative, ungoverned agents calling internal APIs directly, is worse. The real question is how much governance your risk profile demands.

The caveat. The talk is part architecture review, part book promotion. Gaggar ends by mentioning his free Kindle MCP book, and many claims are asserted rather than measured. Still, the direction is credible: agents will move through a controlled API gateway, with tool-level security, memory, and evaluation as required platform services. Teams should copy the pattern, not the vendor approach.