Why upgrading your LLM won't fix context problems

The proposal. Ricardo Ferrer, a software engineer at Redis, built an Alexa skill called MyJarvis to test the company's open-source Agent Memory Server. The goal was to create a conversational assistant that could remember context, set reminders, and answer household questions. What started as a simple LLM call quickly revealed that the hard problems were not about model capability but about how context is managed.

The first lessons. Upgrading from GPT-3.5 to GPT-4 increased cost but did nothing to fix the bad experiences: context poisoning, distraction, confusion, rot, and clash. Ferrer realized that prompting alone was insufficient. He needed intentional techniques like injecting date/time tools, adding short-term session memory, and implementing long-term memory via vector search. Each fix solved one problem but introduced new complexities, such as infinite loops in tool calls or memory TTL expirations.

The architectural layer. Ferrer gradually built a pipeline of context engineering patterns: query transformation for pronoun resolution, few-shot design to teach the model behavior, reranking using Cohere to prioritize relevant memories, and a content aggregator to filter top results. He also added knowledge bases separate from user memories, using an LLM to route between them. This pipeline turned a single LLM call into a multi-step orchestration, all running within Alexa's 8-second timeout on AWS Lambda.

The cost trap. A naive implementation that appended every interaction to context led to exponential token growth. Ferrer's family of three generated about 30 queries per day, but the memory structure ballooned costs. He switched to token window memory, which capped tokens at the model's limit but risked losing relevant old context. He then built a summarization-based chat memory that used another LLM call to condense important facts, keeping cost flat while preserving signal.

Semantic caching. Ferrer reduced redundant LLM calls by 75% using semantic caching, which checks whether a new query is semantically similar to a previous one. This required careful calibration—too tight a similarity threshold and valid rephrasings miss the cache; too loose and wrong answers are returned. He implemented a cache-aside pattern, querying the cache first and falling back to the LLM only on miss. This technique became essential for cost control and latency.

The bottom line. Ferrer argues that context engineering should be part of the architecture from the start, not added reactively. The Alexa skill worked, but only after systematically applying patterns like few-shot design, reranking, summarization, and caching. The main lesson: upgrading the model is rarely the fix. Engineering leaders should budget for context engineering patterns—they are what make AI applications actually usable in production.