Why most AI agent benchmarks are lying about 'long-horizon' capability

Theta Software argues that most current AI agent benchmarks fail to measure what actually matters for long-horizon tasks. Metrics like 'average human hours' are noisy and often misleading, since agent capabilities differ fundamentally from human workflows. Models can efficiently complete tasks humans find tedious (e.g.,

formatting Excel files), yet struggle with tasks that require sequential reasoning and state management. Real long-horizon work involves tool coordination across multiple systems, cascading state changes where early decisions affect later ones, and high ambiguity in initial instructions. These dimensions—complexity, sequential dependency, ambiguity—are largely absent

from popular benchmarks like GAIA, SWE-bench, or finance-specific evaluations. Most benchmarks measure only narrow, parallelizable tasks, leading to inflated claims about agent autonomy. The hardest problem is verification: as environments grow complex, deterministic verifiers become impractical. Judge models must become agents themselves, inspecting environment

state and trajectories while avoiding reward hacking. This introduces new failure modes—over-constraining exploration paths, inconsistent rubric application, and brittle reward signals. Teams building agents should focus less on benchmark numbers and more on designing environments that test real-world decision chains and recovery from mistakes.