Mission-Critical Generative AI in Action • Scott Shaw • YOW! 2025

Scott Shaw, from CommBank, shares hard-won lessons from running GenAI at massive scale (80B tokens/week). The core problem: most prototypes stay experimental because teams hit unpredictable latency, crushing inference costs, frequent model deprecation, and safety concerns that stall production. His solution is a minimal viable platform built on three pillars—a central gateway that provides one API, load balancing, and auditing; tuned guardrails that filter inputs and outputs against safety and compliance needs; and an evaluation framework that collects, labels, and statistically compares model performance.

Without this platform, teams repeatedly spend months revalidating models every time a frontier model reaches end-of-life (often less than a year). The talk also redefines engineering practice: instead of test-driven development, adopt evaluation-driven development with statistical assertions, and instead of starting with the most powerful model, pick the smallest, cheapest model that meets quantitative thresholds. This requires data pipelines, synthetic bootstrapping, and continuous human-in-the-loop supervision.

For leaders, the signal is clear: enthusiastic experimentation must be channeled into disciplined processes with platform support, or the gap between prototype and production remains a grave of wasted investment. The shift demands that every engineer develop model rigor, a risk mindset, and mathematical literacy in probability and statistics.