Roblox's fix for the trust gap between AI code and production shipping

The trust paradox. Andrew Swedllo, an engineering director at Roblox, opens with a sharp observation: most teams ship AI-generated code, but almost nobody trusts it. In an audience poll, nearly everyone had shipped AI code recently, but only three people trusted that code completely. This gap between generation and trust is the central problem his team set out to solve with a project called "Prompt to Prod" — the ability to go from a natural language prompt straight to production without human intervention.

The infrastructure iceberg. The bulk of the work had nothing to do with AI models. Before Roblox could attempt autonomous deployments, the team spent weeks adding CLIs, APIs, and MCP integrations to systems that had none. Playwright was used to convert UIs into agent-friendly interfaces. Test coverage, canary deployments, auto-rollback, and telemetry were all prerequisites. Swedllo makes the point explicit: the hard part of prompt-to-prod is plumbing and infrastructure, not model selection.

Institutional knowledge as code. To make agents act like Roblox's best engineers, the team built "exemplars" — YAML-encoded rules extracted from 1.75 million code review comments across 700,000 PRs over three years. These were clustered, curated by repo owners, and fed into an alignment engine. The result: an automated code review agent reached 60-70% suggestion acceptance rates, outperforming the 55% acceptance rate of human reviewers. The system works because it surfaces the judgment of top engineers, not generic model knowledge.

Policy as a bottleneck. Removing human code review was a policy fight. Compliance and security teams initially said no. Swedllo built the case with data showing AI code review suggestions outperformed humans, then negotiated a policy change for low-risk changes. He argues that teams producing 10x more code cannot review it with the same headcount. The real constraint is organizational, not technical — and most teams are not yet confronting this policy question.

New metrics for a new world. Traditional DORA metrics and lines of code are becoming meaningless when AI generates most of the output. Roblox now measures agent quality through rigorous evals (30-40% of their AI work), feature velocity classified by LLMs analyzing PR intent, and long-running turn time — the duration an agent can work autonomously without human re-steering. Currently the P50 turn time is 40 seconds, far from the 8-hour goal needed for true 24/7 AI.