Why Your AI Analyst Can't See Everything: The Permission Problem

The problem. Bridgewater Associates built a vault of 100 million documents—internal research, meeting notes, third-party reports—each with its own permission rules. Their AI analyst, meant to answer questions like "Why have token prices fallen?", needed to respect those permissions. Naively, they tried post-filtering: search all vectors, then discard results the user couldn't see. Recall collapsed.

Why naive fails. At 100 million documents, even a fast in-memory scan takes seconds per query. Post-filtering returns too few relevant results because the query's nearest vectors rarely belong to the user's allowed set. Users knew documents existed but the AI couldn't surface them. Bridgewater saw this break in production.

The engineering solution. Turbopuffer's native filtering inverts the approach. Their vector index automatically clusters similar vectors into groups of roughly 200. Permissions are indexed per cluster: if a user has access to even one document in a cluster, that cluster is searchable. At query time, the engine first selects eligible clusters (pre-filter), then scans only those clusters for nearest neighbors. This avoids the performance hit of scanning all allowed documents.

Production results. Across 450 million vectors (from 100 million documents with multiple embeddings), turbopuffer achieves a P90 latency of 120 milliseconds. The cost advantage comes from storing cold data on S3 and caching hot segments on SSD or memory. Bridgewater's production deployment confirms the pattern works at scale—a rare real-world validation for permissions-aware vector search.

Tradeoffs and gaps. Turbopuffer acknowledges nested group permissions remain difficult to handle automatically. Organizations with deeply nested hierarchies may need custom logic. Additionally, rebuilding permission indexes when access changes is not discussed—a critical operational detail for dynamic organizations. The talk focuses on search, not on how the AI agent itself interprets results.

What leaders should take away. The core insight is durable: post-filtering permissions in vector search is a combinatorial trap. Any team building internal AI tools should test recall at their target document count. If recall degrades, they likely need a search engine that bakes permissions into indexing, not just into query results. This is infrastructure, not a feature toggle.