Why most AI benchmarks are quietly fake and what actually matters

The data market is undergoing a structural unbundling. Two years ago, vertically integrated giants like Scale AI handled everything. Today, specialists outcompete them at sourcing, environment building, reward design, and evals. This fragmentation is permanent because quality does not scale linearly with quantity, and labs now mandate 20-30 vendor diversification. The critical distinction is between

type one data (captured real workflows) and type two data (contrived expert examples). Most vendors sell type two and bill it as type one. The dirty secret is that contrived benchmarks only test isolated questions, not sustained reasoning across long dependent episodes. This creates widespread benchmark psychosis where single numbers cannot be trusted. Verifiability determines

which domains mature first. Coding succeeded because GitHub provided unit tests, community consensus on correctness, and infinite public examples. Biology, security, and law score low on all three verification axes, which is why their data markets remain fragmented and locked inside enterprise workflows. The upstream signal for the next application layer is which data vendors

labs are funding. Successful data companies are all pivoting to enterprise. Once enterprises stop renting lab intelligence and start owning their own models, they need an entirely new abstraction layer: routing small models, managing RL datasets across base model migrations, and building bespoke evaluation systems. The durable value accrues not to data itself but to