Agent automation: the real lesson is evaluation, not model selection

Niels Rogge automated his community science role at Hugging Face using LLM agents to open GitHub issues and follow up. Initially a deterministic workflow, later a fully autonomous agent with Claude SDK and GLM 5.2. The key insight: not disclosing the agent to researchers improves response rates.

Tradeoffs are clear. Deterministic workflows offer control but lack flexibility; autonomous agents are powerful but require robust evaluation. The speaker warns against slop and recommends Hamel Husain's LLM Evals FAQ. Without evaluation, agents risk spamming.

The model choice is secondary. Open models like GLM 5.2 now match closed models, but the real bottleneck is designing the agent loop and managing outcomes. The speaker's Twitter bot (Daily Papers) grew to 90k followers autonomously, showing scale.

Engineering leaders should note the staffing implications: one engineer replaced a team's manual work. However, governance and evaluation processes are non-negotiable. The disclosure choice (not revealing bot) is pragmatic but raises ethical questions.