The Production AI Playbook: Deploying Agents at Enterprise Scale — Sandipan Bhaumik, Databricks

Most organizations start their AI projects by fixating on model selection, which is precisely the wrong approach. Sandipan Bhaumik, drawing from enterprise delivery experience at Databricks, argues that the production readiness gap is defined by three missing layers: observability, evaluation, and governance. Teams that demo well but fail in production almost always skipped these pillars.

The framework is structured around five engineering pillars. First, evaluation must be defined before any code is written—specifically, business-aligned metrics like deflection rates for chatbots, not just abstract accuracy. The talk highlights three layers of evaluation that most teams miss: deterministic checks (regex, classic ML), semantic LLM judges, and behavioral tracing (e.g., duplicate or wasteful API tool calls). The third layer is especially overlooked and expensive at scale.

Second, observability via full tracing is positioned not as nice-to-have but as a regulatory and operational requirement. The speaker illustrates this with a banking overdraft waiver flow, showing how each decision point must be logged to diagnose failures and avoid blind customer appeasement. Third, data foundations are a bottleneck: enterprise data was built for forgiving humans, not unforgiving agents, making data quality and cataloging a dominant engineering concern.

Fourth, agent orchestration complexity explodes beyond a single agent. The talk contrasts orchestrator-worker (central control) with choreography (event-driven, lower latency) and includes human-in-the-loop patterns. Finally, governance includes prompt versioning as formal change management, model change impact testing, and production incident playbooks integrated with ITSM systems.

A retail banking case study reinforces the point: the team spent week 7 of an 8-week engagement on model selection, after building evaluation datasets from 200 real human-agent interactions. This inversion eliminated the typical multi-week model debates. Six weeks post-launch, the observability and evaluation stack caught a stale policy document causing CSAT drops—an issue that would have been invisible without tracing.