Engineering brief
AI's messiest test: insurance claims buried in faxes and 8GB PDFs
At a glance
The brief is enough
- Relevance
- Practical value
- Radar
- Time horizon
- This year
- Warnings
- Vendor pitch
Claims files arrive as faxes, PDFs, and duplicate records, not clean data. AI can extract insights only after real digitization and human-in-loop verification.
Messy enterprise data, not model capability, decides whether AI document processing delivers value.
Watch if you're evaluating AI document processing for regulated, document-heavy workloads like claims.
Summary
The core argument. Denys Linkov, who leads the AI program at Wisedocs, makes a familiar point with an unusually concrete backdrop: AI in insurance is not a model problem. Claims arrive as paper, faxes, and enormous PDFs, often before any data pipeline touches them. He cites a fax market above 7 billion dollars and EHR adoption that only reached 80-90% during COVID. The consequence is that claims processing still depends on manual extraction and human review.
The mess. A single workers' compensation claim can include intake forms from Texas, New York, and California that ask for different fields, doctor notes that contradict each other, missing X-ray reports, and duplicate records that run as high as 50% in some jurisdictions. Linkov describes multi-gigabyte PDFs, one 8GB file in production, and adjusters juggling 200 claims. Each handoff adds delay; the best claims close in weeks, the worst drag for years. That is the loss-ratio problem.
The benchmark. Wisedocs built MLCR with Artificial Analysis to test models on medical claims. Initial accuracy was around 20%; months later frontier models reached about 75% on harder reasoning tiers. The benchmark also measures conciseness and completeness separately, which matters because Linkov reports GPT models score well on accuracy but omit information, while Anthropic models are more complete with slightly lower accuracy. The lesson for teams: do not evaluate on one metric, and do not trust a vendor's single number.
The workflow. Linkov's proposed architecture is layered: OCR and preprocessing, classifiers and extraction models, summarization, human verification, then cross-claim analysis. Human-in-the-loop review is non-negotiable, both for regulatory comfort and because bad upstream data invalidates downstream insights. The more interesting claims are portfolio-level: provider networks, co-occurring diagnoses, treatment cost variation, and year-over-year price increases. None of that is possible until one claim is cleanly extracted.
The trade-offs. The vendor position is visible. Wisedocs sells exactly this pipeline, and the benchmark is self-reported. Linkov is honest about limitations: no industry-wide proof of reduced claim time, and case studies instead of hard numbers. The counterintuitive signal is that plaintiff lawyers adopted AI faster than insurers because their incentive is speed and settlement size. That asymmetry, combined with fraud and duplicate submissions, means slow digital transformation makes insurers more vulnerable, not just less efficient.
My take
The most useful part of this talk is the reminder that clean data and evals determine whether AI adds value. I'd be careful treating the benchmark as neutral: it is a vendor's own test, and the jump from 20% to 75% accuracy may say as much about question design as model progress. Still, the operational picture matches what I see in enterprise AI: most failures come from duplicate records, missing attachments, and undefined answers, not from model capability. The practical move is to start with data quality and human verification, then ask portfolio questions once extraction is trustworthy.
What to do with this
- Audit one claim pipeline for duplicate records, missing attachments, and nonstandard forms before choosing an AI vendor.
- Run a small benchmark on your own claims with human-verified answers; measure accuracy and completeness separately.
- Ask vendors how they handle 8GB PDFs and conflicting diagnoses, and confirm human-in-loop review sits before decisions.
Watch the video
This video is blocked due to your privacy settings. To watch this video, please accept YouTube marketing cookies.
Related breakdowns
Cheap models beat frontier models at writing like a human
Tone-Bench tests whether AI models can write like one specific human. Its most practical lesson: cheaper models with long prompts may beat frontier models on…
Why a faster model can make your users wait 8x longer
MiniMax's M3.1 flash is faster per token, but doubling concurrency lifted p95 wait from 1.5s to 11.3s. Evaluate the whole job, not the model.
Computer-use agents beat benchmarks but fail the reliability test
Computer-use agents are hitting human-level benchmark scores, but fail the repeatability test. Here is why reliability, not capability, is the real adoption…
Get TL;DW
Too Long; Didn't Watch.
A concise breakdowns of the AI and devtools videos that actually matter for engineering leaders.
Free. Weekly. No hype.
Video and thumbnails remain the property of their respective creators. tldw.news provides editorial analysis, commentary, and discovery links to original content.