Stop Fine-Tuning Your Way Out of Bad Retrieval

The core argument. Most enterprise AI teams treat prompt, memory, and fine-tuning as interchangeable tools, escalating from one to the next when results are poor. Anant Srivastava argues this is an architectural accident. Each layer serves a distinct job based on the nature of the knowledge: behavior belongs in the prompt, facts belong in memory, and stable reasoning patterns belong in the model weights. Mixing them leads to expensive, brittle systems.

The accidental architecture. Srivastava illustrates with a support assistant that accumulates decisions over weeks: a product manager edits the prompt for tone, the support team adds refund documents to retrieval, an engineer stuffs the product catalog into the prompt, and the ML team fine-tunes on old support tickets. When the product catalog changes, the model still outputs old product names because those facts leaked into the weights via fine-tuning. Nobody owned the architecture — everyone owned a piece.

The diagnostic lens. The prompt is for small, stable, editable behavior — the persona and tone. Memory (both agent memory and external retrieval) is for grounding the model in current, large, citable knowledge that changes faster than you can retrain. It also handles access control. Weights are for patterns that have stopped changing — reflexes, not facts. The key axis is rate of change. Srivastava gives concrete examples: a code assistant should use memory for repo contents, not fine-tuning or stuffing into the prompt.

The fine-tuning trap. The most common mistake is fine-tuning to fix a retrieval problem. If the model isn't giving the right answers, the right fix is better chunking, metadata, and retrieval — not teaching it facts via fine-tuning. Fine-tuning should only happen when the information has stopped changing and humans have converged on a stable pattern, like for content moderation or claims processing where a stable center emerges. Even then, monitor for drift.

The circulation loop. Prompt, memory, and weights form a loop. The prompt generates signals that become durable memory. Memory feeds into prompts on session restart. Over time, retrieval patterns can migrate to weights via fine-tuning when stable. Once fine-tuned, you may no longer need to retrieve certain examples. This circulation is the architecture that teams must design, not accumulate.

The takeaway. The model itself is the easy part. The hard part is the harness around it — deciding what lives where and how information circulates. Most teams will benefit from auditing their current system against this lens. If your fine-tuning is compensating for bad retrieval, fix the retrieval first.