OpenAI's finance lead: MCP error rates hit 40% — bundle data yourself

The announcement. Joseph Kim, OpenAI's Head of Financial Services, introduces ChatGPT for Financial Services, an enterprise product blending ChatGPT's consumer tools with firm-specific templates and data sources like S&P and Dilupa. Design partners include Morgan Stanley and Evercore. Kim positions it as a productivity tool for knowledge workers, always assuming a human in the loop.

The surprising humility. Kim states that 95% of what his team built is something any startup could replicate. The product sits on top of ChatGPT's plugin architecture using the same primitives available to external developers. The secret sauce is data bundling and UI customization — not foundational model changes.

The MCP reality check. Kim tested major MCP connections and found some had 40%+ error rates. Rather than rely on these, OpenAI chose to bring financial data in-house, controlling the end-to-end experience. Customers now cite data quality as the top feature. The assumption that MCPs make proprietary data bundling redundant is empirically false.

Safety as a people problem. Current safety design assumes human oversight, similar to how junior analysts' work is reviewed by associates and VPs. Kim argues models should be treated like people — requiring readability permissions, hierarchical review, and the same error-checking systems banks already have for humans.

The eval trap. Kim warns that building good evals is extremely hard. Financial evals are saturated or broken. GPT-6 showed no improvement on some financial benchmarks because the evals themselves were flawed — wrong answer keys, poorly specified prompts, or saturation. His team is building a new eval that predicts ability to automate specific professions.

The taste gap. Models today fail on human judgment, not factual accuracy. Hallucination complaints have dropped, but aesthetic and taste-related failures remain — ugly slides, poor formatting. Kim expects models to improve on these softer objective functions by 2027, using human preference training similar to coding taste benchmarks like Frontier Code.