LLMs as Robot Brains: Why Robotics R&D May Be Investing in the

The shift. A growing group of robotics researchers and startup founders now argue that general-purpose language models—not models fine-tuned specifically on robot data—may become the most effective way to control robots. This idea, which MIT Professor Philip Isola recently called the era of "robot-use agents," is built on a simple observation: frontier LLMs are already good at writing code, and controlling a robot is fundamentally a code-generation problem.

The historical arc. The conversation traces the lineage from RT-2, which fine-tuned a language model to output end-effector poses directly, through code-as-policies methods that let LLMs write Python functions to control robots one-shot, to the current generation where models like Astra can write complex policy graphs with built-in failure handling. The key architectural insight: early VLA (vision-language-action) models were essentially transductive—mapping inputs to actions without intermediate reasoning—while modern LLMs can chain-of-thought, write code, and use tools, which is far more sample-efficient.

The data lesson. Both founders argue forcefully for the bitter lesson: scale and diverse data beat domain-specific engineering. Waddle Labs is building a harness that lets LLMs control robots by treating robot APIs as tools, then consolidating successful traces into fast, reusable skills. RoboCurve evaluates every approach—LLMs, VLAs, world action models—and reports that current LLMs often match or exceed specialized models on spatial reasoning tasks, especially when given a good harness.

The latency bottleneck. Directly querying a frontier model for every robot movement introduces unacceptable latency. Astra's response times, while improving 2x per month, are still too slow for real-time control at useful throughput. The proposed solution: use the LLM to generate a deterministic code path once, with sensory checkpoints where a smaller VLM intervenes for failure detection and variation. This mirrors the sleep-and-compression cycle in biological intelligence—fast in-context learning to gather experience, then consolidation into faster policy weights.

The organizational blind spot. Most teams assume that robotics progress depends on better robot data collection and more VLA training. Both founders suggest the opposite: the fastest path may be to ignore robot data entirely and bet on frontier LLMs trained on computer-use traces, code, and images. This flips the standard R&D allocation argument—companies like Figure or Tesla should still fine-tune at scale, but for everyone else, LLM-as-controller may be the smarter bet by an order of magnitude.