Computer-use agents beat benchmarks but fail the reliability test

The pitch. Simular CEO Ang Li makes the case that computer-use agents are a distinct category: instead of calling APIs, they operate a full Windows desktop through screenshots and accessibility trees. The company's Agent S reached human-level performance on the OSWorld benchmark in December, and the same loop can automate insurance portals, healthcare data entry, and other workflows that have no modern API. That sounds like a breakthrough.

The reality check. The benchmark headline hides a reliability gap. Simular reports OSWorld 2 now tops out at around 20% for the best model, and its own pass@k tests show that running the same agent three times drops success by roughly 10 points, with more degradation when the environment changes. A single successful trajectory is not a repeatable process. That is the metric engineering leaders should care about.

The proposed fix. The first lever is cheap: instruct the agent to ask clarification questions before acting. Simular says that improves reliability across Claude and its own Agent S with GPT-5. The second is structural: Simulan, an open-source scripting language, records successful agent trajectories as deterministic code. On the second run, the code does the work without model calls, which cuts token cost sharply and makes repeatable workflows economically plausible.

The trade-off. This is the stability-plasticity dilemma. Deterministic code is reliable but brittle; a VLM agent is adaptive but probabilistic. Simular's neurosymbolic loop tries to get both: the agent explores, writes Simulan code, verifies each step, and when the environment breaks the script, repairs it and saves a new skill. The live demo showed exactly that recovery, with visible hiccups. It is promising, but still early and not production-proven.

The product push. The talk is also a vendor pitch. Simular is moving agents to cloud Windows and Android virtual machines via S1 and S1 API, and the roadmap implies hundreds of AI workers per organization. A Microsoft Windows 365 demo fills out a traditional ERP form, and a Berkeley hackathon showed remote Mac control over FaceTime. Closed beta, invitation codes, and credit giveaways make the near-term product claims hard to verify independently.

The signal. Ignore the autonomous-computer rhetoric. The durable insight is that reliability, not model capability, is the bottleneck. Teams should measure pass@k on their own workflows, use code replay before model calls, and add verification around irreversible actions. That is a concrete plan for piloting computer-use agents, even if the vendor vision is still hype.