Why a Positive A/B Test Is Not Enough for AI Features

The core problem. Supriya Subudhi, a product manager at Bumble, argues that a positive A/B test result for an AI feature is only the starting point, not the finish line. Drawing from Bumble's launch of an AI photo feedback tool on a platform with over 20 million monthly active users, she outlines five practical traps that can turn a statistically significant win into an operational failure. The talk targets product and engineering leaders building GenAI into consumer products, but the lessons apply broadly.

The survey gap. Bumble's research showed 78% of users wanted AI photo feedback. When shipped, only 10-15% actually used it. Subudhi's team had built a standalone feature users had to discover on their own. The fix was to embed the AI into existing workflows—offering feedback when users added photos or hit a matching wall. This dramatically increased adoption. The lesson: stated interest in AI does not predict real usage. Teams must design for the full user journey, not just the feature.

The average trap. The top-line A/B results looked great: more users added photos and made changes. But digging into the long tail revealed nonsense—the AI told people who were already smiling to smile more. Subudhi contrasts Spotify's AI DJ, where two bad songs out of four is acceptable, with a banking app that gives bad budgeting advice. The scope of error matters. Bumble had to move from aggregate green-versus-red metrics to segment-level eval reviews, checking for harmful or irrelevant outputs across demographics and languages.

The cost conundrum. Bumble started with the best LLM for highest accuracy. It was 10x more expensive than a simpler approach, and success drove costs higher—the opposite of traditional ML where inference is cheap after training. Switching to a cheaper model cut costs but also cut accuracy, meaning the original A/B win was model-dependent, not concept-dependent. The team had to iterate through models and prompts using eval tools to find a cost-effective combination.

The regulatory wall. Bumble launched the feature only in the US due to varying AI regulations globally. Subudhi argues that compliance can no longer be a late-stage concern. Teams must decide early whether to build a globally compliant but potentially weaker feature, or a best-in-class feature for a single market. This tradeoff forces engineering leaders to budget for multiple code paths and legal reviews, not just model experiments.