Engineering brief
AlphaFold won a benchmark, not the problem. Scaling needs problem definition first.
At a glance
The brief is enough
- Relevance
- Practical value
- Time horizon
- Now
- Warnings
- None
AlphaFold matched deposited PDB structures; it did not solve protein dynamics. The same gap shows up in every AI project where a benchmark stands in for the real problem, and calibration plus problem definition decide whether scale helps.
AI wins are often benchmark wins. Teams that confuse them with solved problems build on uncalibrated proxies and weak foundations.
Watch if you're setting AI strategy and want senior perspective on when scale, data, or problem definition will move outcomes.
Summary
The headline claim. Pushmeet Kohli is blunt about AlphaFold. It matched structures that researchers had already deposited in the PDB, and that turned out to be useful. It did not solve protein folding in the deeper sense: the true ground state, the distribution of shapes, and dynamics remain open. Proteins are disordered and context-dependent. The lesson for any engineering leader is that a benchmark win is not a solved problem.
The scaling caveat. Sal Candido challenges the naive version of scaling laws. They are not automatic; finding one is the actual research. A model can only pull information that exists in the data. His concrete example: training protein language models on messy metagenomic sequences, many of which are not even full proteins, improves the ability to design real proteins. But that can tempt teams into generating more of the wrong data instead of asking what data the problem needs.
The bitter lesson. Kohli reframes Rich Sutton's bitter lesson beyond data versus modeling. The original trap is tribalism: researchers who call themselves modelers ignore data generation, and data people ignore modeling. AlphaFold invested in modeling because the PDB took decades of global investment and could not be augmented cheaply. For the virtual cell, the opposite was true: the data did not exist yet. Define the problem first, respect constraints, and fail fast.
The craft tradeoff. AlphaFold 2 was not pure scale. It encoded physics and biochemistry into the architecture, which made it data-efficient. Kohli argues that curated data and coverage are their own art. Candido adds the counterpoint: craft still matters in scaling, through algorithmic work, bespoke architectures, and faster inference, and a slightly wrong inductive bias can hold a model back once data grows. The strategy depends on where you are on that curve.
The uncertainty lesson. Reliability is not accuracy. AlphaFold 2 scored around 90 GDT on its benchmark, but its confidence measure, pLDDT, was poorly calibrated. A model that is correct on average but overconfident on failures is dangerous because humans cannot tell which answers to trust. He wants behavioral characterization before use. Interpretability, he argues, is in the eye of the beholder: a human may not read AlphaFold's reasoning, but a larger model might.
The practical call. Both researchers are cautious about AI in the clinic. AI already touches every part of drug discovery, but a 10x acceleration depends on harder biology and better models of disease, not just bigger models. Candido says basic research and understanding are still the foundation. For engineering organizations, the transferable signal is to name the real problem, check whether the benchmark measures it, and treat confidence calibration as a release criterion.
My take
I keep coming back to the gap between a benchmark and the actual problem. AlphaFold matched structures already sitting in the PDB, and that was genuinely useful. But calling it solved hid the distance between that benchmark and protein dynamics. Many AI teams are in the same spot: a leaderboard looks like progress until someone asks what the metric ignores.
The panel's strongest point is calibration. An accurate model with uncalibrated confidence shifts the cost of error downstream to the humans who have to decide whether to trust it. I'd make calibration a release criterion the way we treat uptime. The second lesson is harder to fund: data curation and problem definition are more constrained than compute. I would budget for them explicitly.
What to do with this
- Ask each AI project team to document the benchmark the model optimizes and what it ignores.
- Require confidence calibration checks before production rollout, and treat overconfident failures as release blockers.
- Audit data pipelines for curation and coverage; budget for collecting the data the problem needs, not just more compute.
Watch the video
This video is blocked due to your privacy settings. To watch this video, please accept YouTube marketing cookies.
Related breakdowns
Killing code review is only safe for small, agent-native teams
Mandatory code review is gone at AMP. Bet: trust, agents, and cloud dev make review gates obsolete. Catch: that math works for 20 people, not every org.
Digital twins are 85% accurate—but only if you collect the right data
Simile AI achieves 85% accuracy in digital twin behavior prediction, outperforming frontier models by 2-3x on niche populations. The tradeoff: proprietary…
Protein design works. Scaling it is the hard part.
Chai Discovery's models now predict protein structures within an atom's width. The bottleneck is no longer model capability — it's integrating with pharma's…
Get TL;DW
Too Long; Didn't Watch.
A concise breakdowns of the AI and devtools videos that actually matter for engineering leaders.
Free. Weekly. No hype.
Video and thumbnails remain the property of their respective creators. tldw.news provides editorial analysis, commentary, and discovery links to original content.