Uber’s AI Agent Lesson: Redundant QA Gates Stop Reward Hacking

Uber’s computer vision team built a multimodal agent that enhances food photos—but the real innovation is how they keep it trustworthy. Instead of a single model, they chain multiple agents with evaluation gates that check faithfulness, completeness, and realism. Failures like adding imaginary shrimp or deleting sauce get caught before a customer sees them.

The system uses a “Swiss cheese” model of redundant QA layers, balancing creativity with safety. A router decides whether to enhance or skip; an editing agent iterates with feedback; a final gate applies policy checks. This multi-stage design minimizes the chance of a bad edit slipping through, even if individual agents fail.

Beyond static evaluation, Uber built a closed-loop auto-tuning pipeline triggered by drift detection. A diagnoser agent analyzes mismatches between human labels and agent output, then rewrites configurations for specific agents—no human required. This keeps the system aligned as marketplace data shifts, but with quick rollback built in.

The talk emphasizes that logging everything in a flat, queryable format was foundational, enabling cross-functional debugging. For engineering leaders, the key tradeoffs are balancing precision vs. recall in routing, managing compute costs of unnecessary enhancements, and the risk of agents oversteering toward generic outputs when penalized.