Letting AI approve low-risk PRs cut Duolingo's median merge time

The signal. Duolingo's DevEx AI team runs an LLM-based PR risk assessment in production. Every pull request is scored low, medium, high, or undetermined from title, verification steps, diff, and risk definitions. Low-risk changes that pass repo config get auto-approved by a bot, merging with no human reviewer. Roughly 10% of all PRs now merge this way, and median time to merge across the org dropped from about 18 hours to 12 hours over six months.

The guardrails. The system is deliberately conservative. Low risk covers markdown, small straightforward changes, and routine config updates. Medium and high risk catch API version bumps, auth, logging, permissions, infrastructure, and core features. Repo config can restrict the bot to code owners, exclude directories, disable AWS changes, skip compliance-heavy repos, and block new engineers during onboarding. Eligible authors can still opt out and request human review.

The precondition. Training came before automation. Duolingo ran lab-style workshops, built AI observability dashboards tracking usage, cost, and models by team, held office hours, ran a Slack channel and bi-weekly meetups, and used vendor relationships for early access to Cursor features like team rules. AI tool adoption went from 80% to 100%. That literacy changed behavior: engineers reported misclassifications to a Slack channel instead of rejecting the bot.

The feedback loop. Wrong risk labels flow into a dedicated Slack channel, become an eval set with expected labels and keyword extraction, and are used to fine-tune the risk prompt. The prompt grows with developer feedback and stays backwards compatible. A couple of messages per week is the maintenance load. This loop is what makes the system trustworthy, and it depends on engineers being willing to engage with the model rather than fear it.

The missing data. There is no strong defect-rate signal. Sarah says about 200 PRs flow per day and one or two auto-approved changes get reverted, which is hard to compare with normal human-review outcomes. She also says the bot has not, to her knowledge, been involved in a site incident. That is reassuring, but it is not a controlled comparison showing automated low-risk approvals are as safe as human ones.

The real takeaway. The cascade effect matters most: accelerating the easy 10% lowered median merge time for all PRs, freeing reviewers for riskier work. The project pivoted Meta's diff risk score: instead of gating risky changes, Duolingo removed friction for safe ones. Copy the flywheel, not just the bot: find changes people already rubber-stamp, guardrail a conservative classifier, and create a visible correction channel. The bot is the last step, not the first.