Your Agent Improvement Strategy Is Incomplete Without Trace Mining

LangChain's Applied Research lead frames agent improvement as a data mining problem. The core insight is that agents generate massive trace data—tool calls, API interactions, outputs—and this data is the signal for continuous improvement. Most teams miss that reading and analyzing these traces at scale is currently expensive and technically challenging. The talk introduces a

three-step pipeline: ship the agent, collect traces, then mine them. Mining means using agents to read other agents' traces to find good/bad interactions, test counterfactuals (e.g., model swaps), and generate training data. The key tension is between determinism and autonomy—as agents become more autonomous, understanding their behavior becomes harder without systematic trace analysis. The practical

recommendation is to start with harness engineering (prompt/loop tweaks) which gives fast feedback, then fall back to fine-tuning when that ceiling is hit. Open models now make fine-tuning economically viable, especially for narrow vertical tasks. The talk also notes that trace data volume is on an exponential trajectory—what's large today will be trivial tomorrow.