tldw.news logoTLDWToo Long; Didn't Watch
All daily briefs

Agent reliability is a maintenance problem no model update will fix

Today's vendor talks agree on one thing: the model is the easy part, and the loop around it is where value leaks.

Build memory, measure pass@k, budget for upkeep, and don't trust a vendor number until you've got your own.

  • The reliability gap. Single-run demos are hiding a reliability gap.

    Simular's pass@k data shows success drops about 10 points by the third run, and Browserbase's pitch is built on agents that forget what worked last time.

    Don't scale anything until you've measured repeatability on your own workflow.

    AI EngineerAI Engineer

  • Lock-in keeps moving. Every 'escape the vendor' pitch just moves the lock-in to another layer.

    Fireworks replaces API spend with its own fine-tuning stack, and Modular wants you to swap PyTorch plus vLLM for Mojo and MAX.

    Neither shows neutral production data, so don't commit until you've tested speedups against your own traffic.

    AI EngineerAI Engineer

  • Wrong design targets. Containment and prompts are the wrong design targets.

    Druid AI's 96% containment agent can score worse than an 80% one when escalation routes a mortgage question to a banker, and Ondrej's software-factory argument is about scheduled, event-triggered agents.

    Don't reward containment; build your scorecard around business outcomes and who owns each handoff.

    AI EngineerDavid Ondrej

My take

The vendors are right that the model is the easy part, and wrong that their platform is the answer.

The same operating problem hides behind every demo, whatever they call it.

The durable move is to build your own pass@k baseline and a human review loop before you trust any demo number.

I'd treat every benchmark as a hypothesis until it survives your own traffic.

And the teams that win will own the measurement and the upkeep, and rent the models.

The videos behind this brief

We watched these so you don't have to.

Computer-use agents beat benchmarks but fail the reliability test video thumbnailComputer-use agents beat benchmarks but fail the reliability testAI EngineerStop Swapping Models: Give Your Agents Memory of What Works video thumbnailStop Swapping Models: Give Your Agents Memory of What WorksAI EngineerThe real cost of the open-source fine-tuning loop is platform lock-in video thumbnailThe real cost of the open-source fine-tuning loop is platform lock-inAI EngineerThe AI hardware boom makes inference software the real lock-in video thumbnailThe AI hardware boom makes inference software the real lock-inAI EngineerContainment is a vanity metric: agent escalations can be the real work. video thumbnailContainment is a vanity metric: agent escalations can be the real work.AI EngineerSoftware factories are the signal, not the AGI hype video thumbnailSoftware factories are the signal, not the AGI hypeDavid Ondrej

Get the next brief in your inbox

Video and thumbnails remain the property of their respective creators. tldw.news provides editorial analysis, commentary, and discovery links to original content.