Agent to-do lists waste tokens. Move the plan to the harness.

The core claim. Arize's in-product agent Alex began as a GPT-3.5 prototype in 2023 with a 50% hallucination rate and became a production system after multiple rebuilds. The team's conclusion is direct: the model was never the bottleneck. Nearly every improvement came from harness code, not from waiting for frontier models. That makes this a useful architectural thesis for any team building agents, even with the vendor context.

The harness owns the plan. Alex originally handed the model a to-do list and watched it spend most calls checking items off, with 20-plus iterations on the to-do tool itself. The team moved planning into the harness, rendered the next step into each model call, dropped the in-progress marker, and is now testing with no to-do tool at all. The lesson is that models are eager with lists; the system should own state and completion.

Memory is a file, not a graph. After trying vector retrieval, episodic and semantic stores, and temporal knowledge graphs, Arize landed on one markdown file per user, capped at 8,000 characters, loaded on every request and written by a background job after the session. It is small, exact, easy to audit, and hard to over-engineer. The choice is a quiet rebuke to the default reflex of reaching for a vector database.

Model choice is a default. Arize defaults to "fast and good enough," with Sonar 4.5 as the current model, and invests the saved effort in the harness. Because customers can bring their own providers, the team built defensive stopping rules: an explicit finish call, bounce-backs, and force finish. They are also starting to auto-route tasks and train small SLMs for narrow surfaces like filter generation, where a full agent loop is overkill.

Production traces are the test. Alex is traced with Arize's own product, and live user sessions become regression datasets and experiments. The team notes agent bugs rarely appear as explicit errors: 43 identical tool calls in one turn, or a 227-second response, are the real signals. Signal automates the reveal, while humans still own golden labels, evaluator alignment, and speed-versus-quality tradeoffs.

The tradeoffs. The talk is convincing but shaped by the vendor. Arize sells the tracing and evaluation platform that makes these lessons visible, and the numbers are illustrative rather than benchmarked. The choice to keep memory as one small file also depends on Alex operating inside a defined product boundary. Even so, the survivor list, tracing, math in code, golden datasets, streaming, and skills is a strong starting point for agent teams.