Your Model Rankings Are Wrong: Fix with IRT

Using a single accuracy number assumes all benchmark items are equally informative, which is false. IRT models each item’s difficulty and discrimination, estimating a model’s ability (theta) with confidence intervals. This can flip rankings: two models with identical accuracy differed by a full standard deviation in theta.

For teams that rely on private benchmarks to select models, IRT-based auditing can flag mislabeled answers, useless questions, and even items that are negatively correlated with ability—meaning good models get them wrong. This prevents building a product on a model that looked good in a flawed benchmark.

Implementing IRT requires statistical know-how and retooling eval pipelines. The output is not a single digestible number, but a richer view that demands interpretation. However, the cost of mis-selection is far higher; the talk also shows how to shrink benchmark size dramatically while preserving rank correlation, making it practical.

The most overlooked application: using residual analysis to detect benchmark leakage. By fingerprinting different organizations with unique item sets, you can identify which vendor leaked your proprietary eval data, a governance risk that is invisible with simple accuracy. This mirrors concerns about model distillation and similarity detection.