Stop Predicting Tokens: Why Latent Space Training Could Change AI Economics

Matthieu Wyart argues deep neural networks overcome the curse of dimensionality by implicitly learning hierarchical abstractions from data, a process analogous to how physicists build coarse-grained variables. Unlike shallow networks that memorize, deep architectures discover latent structures like syntax trees or object hierarchies, which allows them to generalize creatively from limited examples.

The key insight is that networks don't need to memorize everything because they group configurations that predict similar contexts—a mechanism Wyart formalizes in a physics-inspired framework. This explains why deep networks can generate novel sentences or images: they learn the underlying generative rules, not just surface correlations.

Wyart's latest theoretical work suggests predicting in latent space rather than token space is dramatically more sample-efficient. Abstract concepts are learned faster because correlations between abstractions are stronger than correlations between abstractions and raw tokens. However, this remains untested at scale, and it's unclear whether such encoders can power generative decoding efficiently.

The practical implication is significant: current LLMs may be dramatically overfitting in their token-level prediction objectives. Teams should watch for emerging methods that predict in latent space, which could reduce data requirements by orders of magnitude—but adoption is stalled by infrastructure lock-in and the absence of proven generative decoders for these representations.