Your Agent Can Lie While Monitoring Stays Green. Build Eval Gates.

The failure mode. The strongest signal in the workshop is the refund example: a customer asks for a refund, the agent says it issued one, and the tool call never happens. Traditional monitoring sees no latency or cost problem, so the system looks green. This is the real problem with high-stakes agents: they can fail while every system metric stays healthy, and catching it requires evaluation, not more logs.

The proposed loop. Langfuse's recommended practice is a continuous loop: trace every agent step, monitor for quality signals, build golden datasets from production traces, run experiments, and evaluate with a mix of LLM-as-judge and deterministic code checks. The demo shows a banking assistant with PII masking, guardrail scores for investment advice, and a prompt version promoted to production that triggers a CI job with pass/fail eval gates. The workflow is practical and repeatable.

The hidden cost. The most valuable part is the human annotation step. Domain experts review a curated sample of traces, not all of them, and score them with simple yes/no or comments. An LLM then groups the comments into failure-mode categories. That is a sensible division of labor, but it makes the real cost explicit: golden datasets and annotation queues need ongoing expert time. Maintaining them is a staffing and process decision, not a tooling purchase.

The trade-off. The speakers warn against relying entirely on LLM-as-judge, because the judge can be wrong. They recommend blending it with deterministic evaluators, and show a simple exact-match code evaluator. Teams need to decide whether a score is a guardrail or a signal, set thresholds, and accept false positives and false negatives. Early on, they suggest tolerating extra false positives, then tightening over time. That is honest but still leaves calibration work to the team.

What's missing. The evidence is a clean demo with fictional banking data, not production results. There are no numbers on how often LLM judges agree with human raters, no comparison with other eval platforms, and no discussion of dataset drift. The Langfuse claims, like 90 billion observations and Fortune 50 adoption, are vendor talking points, not proof this workflow reduces incidents. The underlying practice is sound, but it is being sold, not independently proven.

The bottom line. Engineering leaders should treat agent evaluation as a delivery gate, like QA, and fund the human review time that makes it work. The tooling matters less than the discipline of defining expected behavior before every model or prompt change. Teams that skip the golden dataset and rely on observation alone will keep discovering agent failures after customers do.