Stop Inventing Bad Scripting Languages with Agent Tools

The breakthrough was replacing 15 sequential spreadsheet tools with a single JavaScript REPL that maintains state. This lifted accuracy from 50% to 92% on an internal financial analysis benchmark and eliminated timeouts entirely. The REPL let the agent interleave reasoning with shorter scripts, building on previous results rather than chaining many atomic calls.

Earlier attempts—multi-agent architectures, SQL, XML representations—all failed because they were too rigid, lacked context flow, or didn't align with how LLMs reason. The team discovered that when an agent makes many tool calls, it's effectively using a bad scripting language; giving it a real one proved far more efficient and easier to extend.

Equally important were high-fidelity feedback loops: a formula engine and a renderer let the agent verify and correct its work. Without them, the REPL alone would produce unreliable outputs. The verification loop's value increases with model capability, making it a durable investment even as interfaces evolve.

Domain knowledge prompts and deterministic evaluation were crucial supporting practices. The team replaced LLM judges with black-box spreadsheet comparisons where possible, reducing confounded results. And they caution that many apparent model failures are actually infrastructure bugs—teams must inspect traces before blaming the model.