Transformers Lost to Linear Models—A Lesson for Tech Leaders

The push to build transformer foundation models for single-cell RNA data has hit a wall. Recent benchmarks, including NeurIPS 2024 papers, show these complex models often fail to outperform simple linear models. The 'genes as tokens' analogy breaks down: single-cell measurements are snapshots of a noisy, bursty process, not a coherent sequence.

This isn't just a biology niche. For engineering leaders, it's a live-fire demonstration that pouring compute into domain-specific foundation models without rigorous, domain-aware baselines creates expensive dead ends. The real bottleneck is data quality and the mismatch between modeling assumptions and biological reality, not model architecture.

Flow-matching models are emerging as a more natural fit. Instead of compressing cells into a latent vector and predicting mean gene counts, they learn the full distribution. Early results align better with ground truth, suggesting distribution-aware generative models may unlock scaling where transformers could not.

Scaling noisy data won't fix the snapshot problem. Incremental model improvements may not shorten the 10-year drug development pipeline unless paired with innovations across other measurement modalities and pipeline stages. Hype around virtual cells and digital twins remains aspirational.