Versioned web search lets teams backtest agent retrieval. That changes AI evals.

The pitch. Ryan Ahern from Exa introduced ExaSnapshot at an AI engineer event: 400 billion historical web snapshots, searchable as of a date. A user can give a URL and ask how it looked then, or run a semantic search restricted to what would have been returned. Exa's positioning is that agents replaced the search box and need retrieval built for high volume, verbose queries, and low cost.

The use case. The financial example drives the argument. In early October, reporting put OpenAI's annualized run rate around $70 billion. Later stories revised that toward $50 billion because cloud provider revenue should be treated as pass-through. A date-restricted search lets a model ask what evidence existed before the market understood the accounting, including SEC footnotes or less prominent reporting. That turns web retrieval into a backtestable input.

The workflow. Ahern describes three patterns: traditional models needing historical unstructured data such as software prices or sentiment; existing models augmented with agentic features such as adverse media scans or equity research; and fully agent-run prediction systems. The core safeguard is preventing leakage. Without versioning, a live search for acquisition prediction finds the announcement and makes the model look perfect. With snapshot search, a team can test performance a day, seven days, or a month before the event.

The evidence. The strongest illustration is a workshop with 100 people building acquisition prediction features. High-scoring features included a shareholder calling for a sale, a share price 60% below its five-year high, and a feature for whether the company is named Dominion. Ahern uses that to make a point about overfitting: backtesting lets teams catch features that memorize the training set. But the talk offers no production results, no independent benchmark, and no details on coverage or date accuracy.

The trade-off. Exa owns the crawl, the index, the embedding model, and the retrieval stack, which promises low latency and token savings, but leaves customers dependent on a vendor's version history. A historical search is only trustworthy if Exa captured the page on the relevant date and can prove it. No coverage guarantees or accuracy numbers are given. For a typical engineering organization, the takeaway is not adopting this product now, but building eval hygiene that pins retrieval to dates.

The bottom line. The concept is stronger than the demo. Versioned web search addresses a failure mode: looking at today's web when you need to know what was knowable yesterday. That matters for regulation, risk, and any model trained on news or public data. Ask search and AI vendors whether historical retrieval exists, how complete it is, and how to backtest features before deployment. The answer may be Exa, or an internal eval fix.