The Agents Aren't Bad, Their Search Queries Are Broken

LLMs have rapidly improved reasoning, but retrieval tools lag behind, creating a 'knowledge gap' where models fail not because they can't think, but because they can't find the right documents. Mixedbread's benchmarks show a sharp drop from oracle performance when default code-optimized tools are used, while better search tools almost eliminate the gap.

The root cause is surprising: agents write garbled keyword-heavy queries like 'senator woman questions billionaires not a company' because they were fine-tuned on code search (grep) and web patterns, mimicking BM25-favored structure. They default to guessing keywords instead of using semantic search, making powerful retrieval useless.

Their solution reframes the query task to describing needed evidence, uses a multi-tool harness (overview, semantic, metadata filter, grep) with up to four parallel search rounds, and fine-tunes a small model on retrieval and trajectory rewards via supervised fine-tuning and RL, yielding dramatic gains on Oblique Congress and Match QA.

For engineering teams, the implication is clear: retrieval quality is now a prompt-design and tool-orchestration problem, not primarily a model-capacity problem. The approach shows that small, well-trained agents with proper search framing can outperform much larger setups, but the technique remains tied to a specific platform and is unproven in diverse production environments.