The RLHF Trap: Why AI Is Great at Chat but Terrible at

Diogo Almeida, former OpenAI researcher behind GPT-4 and RLHF, argues that today's AI is fundamentally built for assistance, not automation. The core problem isn't model intelligence but the optimization objective itself: RLHF trains models to please humans in every interaction, making them unreliable for autonomous decisions. This explains the puzzling gap where LLMs can crush

math benchmarks but fail at basic customer service. Almeida's central insight is that RLHF's reward structure creates an inherent asymmetry: models are rewarded for sounding confident and agreeable rather than for being correct in a calibrated way. This is by design for chatbots but catastrophic for any task where a business needs to trust the

output without human oversight. Every team trying to deploy AI for backend automation is fighting against this architectural reality. The talk claims Claude Code and ChatGPT belong to the same 'assistance era' because both are RLHF-trained. The true next wave, Almeida argues, is rethinking the entire optimization stack for reliability and calibrated decision-making. His company

TypeSafe is building a fundamentally different post-training approach, though details remain vague and proprietary. While the diagnosis is compelling and grounded in real experience, the solution is speculative and self-serving as a company pitch. Engineering leaders should take the problem framing seriously, but treat the proposed solution with healthy skepticism until benchmarks and production evidence