The AI Progress Chart Everyone Is Misreading — Beth Barnes & David Rein

Meter reframes AI capability with a single axis: how long a task takes a competent human. They fit success rates vs human-time across diverse tasks in an agentic terminal harness to compare models from GPT‑2 to current SOTA. The curve looks logistic: models nail very short tasks and mostly fail longer ones. Crucially, they emphasize huge uncertainty: slope-regularization choices moved recent estimates ~35%, and real error bars are closer to 2x.

The biggest risk isn’t statistical noise; it’s distributional shift and task selection. Meter tries to avoid adversarial benchmarks that current models fail by design (which later regress-to-the-mean as labs train on them). They include weird, constrainted tasks to reduce regurgitation, but admit baselines are noisy (lognormal human times, varied expertise) and that any single “time horizon” number shouldn’t be conflated with real job performance.

On agent design, simple, general scaffolds often perform as well as fancy ones across varied tasks. Returns to inference compute are high; you may need hundreds to thousands of dollars of runs per model-task to know it truly plateaus. Small harness changes (e.g., making agents aware of token/time budgets) materially shift outcomes—raising credit-assignment and reproducibility issues.

Operationally, teams should not read “50% time horizon at 12 hours” as “12-hour jobs are done.” Models are more like week-one contractors: broad knowledge, little org context or tacit process. Reliability is mostly binary per task—either it always works or always fails—so production use needs task-level gating, strong success criteria, and escalation paths, not blanket autonomy.

What to watch: build an internal, workload-aligned eval suite with task durations, track binary success per task, measure sensitivity to scaffolding and compute budgets, and resist over-indexing on headline scores. Expect faster gains on short, well-bounded, toolable tasks; long, tacit, cross-system work remains brittle.