Engineering brief
The hidden cost of building your own AI code reviewer
At a glance
The brief is enough
- Relevance
- Practical value
- Radar
- Time horizon
- Now
- Warnings
- None
Greptile processes 1% of all GitHub PRs. Their CTO explains why naive AI code review costs $3-$25 per PR, misses bugs, and produces wrecked codebases — and what it actually takes to do it right.
AI-generated code needs AI review, but naive implementations cost more and catch less than expected.
Watch if you're evaluating a code review vendor or considering building an in-house review agent.
Summary
The proposal. Daksh Gupta, co-founder of Greptile, presents a framework for engineering teams that want to build their own AI code review agent. His company processes about 1% of all GitHub pull requests, working with codebases from Nvidia to Coinbase. The talk targets platform teams or individual engineers tasked with building internal review tools, especially as 'vibe coding' creates a validation bottleneck.
Why naive approaches fail. A simple agent hooked to a PR webhook will miss bugs, misunderstand codebase context, cost between $3 and $25 per pull request, and produce nitpicky comments. The nitpick problem is worse than it sounds: annoying comments breed human complacency, and coding agents will blindly fix every nitpick, producing painfully defensive code. Bug rates are also deceptive — only one in a thousand PRs contains a catastrophic bug, making it hard to know when your review agent is secretly failing.
The prompt and data fix. Teams should build a catalog of bug types from historical incidents and tune prompts to look for those specific patterns while explicitly defining what constitutes a nitpick. The hardest part is the counterfactual: review agents can confirm bugs but never prove their absence. Gupta recommends creating a dataset of historical PRs that caused outages, running every agent version against it, and continuously updating the set as bug topologies shift.
Model inversion. A surprising finding is that Claude and GPT models produce different bug types — Claude is three times more likely to produce off-by-one errors, while GPT models produce more auth bypasses. Using the same model to review its own generated code misses more bugs. Gupta reports a 10% higher catch rate when inverting models. Detecting which model wrote the code is straightforward 80% of the time through branch naming or commit signatures, but the remaining 20% requires heuristics or telemetry.
Context and autonomous testing. Proper codebase context requires more than grep, which Greptile claims is imprecise and accounts for a vast share of inference tokens spent by coding agents. Their approach builds an internal knowledge base — a 'living wiki' — that indexes flows, diagrams, and file relationships. For UI changes, autonomous testing in sandboxed environments using computer vision can catch rendering bugs that static analysis misses. This is technically hard and usually requires a forward-deployed engineering team to set up.
My take
I appreciate the practical skepticism here. The conventional wisdom is that throwing an LLM at PRs is trivially easy, and Gupta correctly identifies the failure modes that only appear at scale: nitpicky comments that wreck codebases, the counterfactual problem, and silent bug regression. The model inversion finding is genuinely useful, I hadn't considered that Claude and GPT have different failure signatures. But the real takeaway for engineering leaders is organizational: maintaining a good review agent requires continuous investment in datasets, prompt tuning, and model selection. That's a team, not a script. The cost per PR also matters: $3-$20 per review adds up fast across hundreds of daily PRs. I'd be careful about the autonomous testing claim; saying end-to-end QA is 'nearly solved' is the kind of statement that gets people into trouble.
What to do with this
- Audit your current AI code review costs per PR, including inference and engineering time.
- Build a dataset of historical PRs that caused incidents and test your review agent against it.
- Implement model inversion: if Claude wrote the code, use GPT to review it, and vice versa.
Watch the video
This video is blocked due to your privacy settings. To watch this video, please accept YouTube marketing cookies.
Related breakdowns
AI Code Needs More Than Static Analysis — Verifying Intent Is the
Guitar from Sonar claims to catch AI-generated code bugs by understanding context. A workshop demo shows promise, but real-world proof is missing.
Code Mode Agents: Less Token Bloat, More Sandbox Complexity
Code mode agents can write and compose tools on the fly, slashing token usage but introducing sandbox management burdens. Arize AI shares production results…
Stop Shipping Agents on Vibes: Build a Regression Loop
Arize's Xander Song shows Harbor, a sandboxed evaluation framework that turns agent changes into verifier-scored regression tests. The catch: realistic data.
Get TL;DW
Too Long; Didn't Watch.
A concise breakdowns of the AI and devtools videos that actually matter for engineering leaders.
Free. Weekly. No hype.
Video and thumbnails remain the property of their respective creators. tldw.news provides editorial analysis, commentary, and discovery links to original content.