Your LLM Bill Is a Model Selection Problem

Most teams default to large foundation models for every AI task, incurring unnecessary costs, latency, and security risks. Smaller language models (SLMs) can handle many features at a fraction of the cost, often on-device. This shift saves significant inference spend, especially as agentic workflows consume tokens faster than prices drop.

The real bottleneck isn't model capability but evaluation discipline. Rachel's 'prototype big, deploy small' framework: prove feasibility with a large model, create a golden dataset, then evaluate smaller models like Llama 3.2 vs. Claude using Phoenix. The smallest model meeting accuracy and latency thresholds—the 'SAGE' model—often performs comparably after prompt engineering.

On-device inference eliminates per-call costs and reduces latency but shifts energy consumption to users' devices. Model selection isn't one-time; evals must be integrated into CI/CD to prevent regressions. The approach works best for bounded tasks like summarization where accuracy is not mission-critical.

The video reveals small models respond better to few-shot examples than to explicit constraints. Also, using Claude as a judge biased results; human evaluation or A/B testing is essential. Engineering leaders should audit LLM calls to offload tasks to local models, potentially slashing inference costs.