AI Engineers: Stop Writing Features, Start Building the Loop

The role shift. Aparna Dhinakaran of Arize AI argues that the definition of an AI engineer is changing rapidly. Citing a recent Karpathy tweet about the funnel from 6 billion LLM users down to 5,000 frontier-lab engineers running thousands of agents, she claims most businesses sit in the middle. The new job is no longer coding individual AI features but constructing the feedback loop that writes, reviews, and improves those features.

The loop architecture. Dhinakaran describes a four-step cycle: build, eval, ship, monitor, improve. The AI engineer now builds the system that performs each step autonomously. Arize’s product Signal is presented as an example: an always-on agent that reads telemetry traces, identifies issues, and can auto-generate pull requests. The agent is triggered by cadence, events, or metrics, runs inside a sandbox with memory, and is designed to be durable against failure.

Telemetry for agents. A key shift: telemetry data must be structured for agent consumption, not human dashboards. The observability pipeline becomes the trigger and context for agentic workflows. Dhinakaran emphasizes that agents like Signal need access to repo, third-party APM data, and a proxy-controlled sandbox. Every run saves issues to memory and tears down credentials, while tracing the agent’s own actions for further evaluation.

The eval bottleneck. The hardest part, she says, is trusting the evaluator. AI engineers must build golden datasets, run evals offline and online, and verify that evaluator judgments align with human expectations. Arize has built a skill to help write evals, but the meta-problem of evaluating evaluators remains an open challenge for most teams.

The enterprise gap. Dhinakaran acknowledges that labs have clean, constrained data. Enterprise data is messy. The gap between what’s possible in demos and what works on real business data is wide. The talk suggests that AI engineers are the ones who must close that gap by designing systems that can handle real-world telemetry and still produce reliable fixes.

What’s missing. The talk is ultimately a vendor pitch for Arize Signal, heavy on architecture but light on production failure modes or cost metrics. The assumption that always-on agents are desirable ignores operational overhead: false positives, security risks, and the challenge of aligning incentives across agent and human workflows. Teams adopting this pattern need strong guardrails before letting agents auto-merge PRs.