Relative Scoring and In-Loop Eval Fix AI Video Quality

AI video generation has raced ahead, but evaluation still leans on frame-level metrics that ignore storytelling, physics, and pacing. Character.ai found LLM-as-judge pipelines too slow and costly for production, while human review doesn't scale.

Their fix: distill a committee of metrics and human-calibrated judges into a small vision model that scores axes—story, physics, audio sync—via relative A/B comparisons. Inference takes 3 seconds per 15-second video. V1 failed by learning vibe over axes; curating a dataset of real vs AI footage, avoiding AI-detector bias, corrected it.

By embedding this fast evaluator inside the generation loop, they shifted quality control from post-hoc to real-time. Agentic workflows now self-correct using the scores, cutting expensive re-generation and improving output consistency at scale.

The core tradeoff was speed vs accuracy. A larger model scored better but was too slow; the marginal gain didn't justify the latency. The lesson: define quality axes, compare relatively, and move eval as close to generation as possible. This pattern applies to any generative output where quality is multi-dimensional and subjective.