Perception Agents: A New Bet on Reliability for Unverifiable Work

While agents can now click, type, and navigate interfaces, they still cannot handle cross-application workflows like employee onboarding. The real work happens in the seams between systems, where context is lost and verification is impossible. Coding was solved first because code is testable; knowledge work has no equivalent.

Reliability—not capability—is the bottleneck. Without trust built on verification, agents remain fragile. Perception agents perceive the rendered screen, not the underlying code, and can confirm their own output or receive direct human feedback by marking elements. This creates a tighter, real-time collaboration loop.

The open-source annotation and verification tools are early steps. They require upfront design specs and definition of rules. While they bypass API dependencies, they introduce challenges around defining correctness for messy, subjective tasks. The approach is promising but unproven at scale.

Engineering leaders should watch this shift. It may change how we invest in automation: focusing on workflow integration and shared context tooling rather than just model upgrades. The real transformation may come when agents can watch what you do, not just execute commands.