Model Swapping Only Works When Evals, Not Benchmarks, Decide

The core signal. The workshop compares four models, GPT-5.5, Kimi K2.6, DeepSeek V4 Flash, and Mistral Large 3, on realistic tasks. Asked to count E's in a sentence, GPT-5.5 and Kimi return 13, DeepSeek returns 6, Mistral 8. Prompt engineering, count per word with reasoning, fixes Mistral. The lesson is that model behavior is task-specific and improvable, not that one model is best.

The RAG grounding gap. A retrieval test asks each model to answer from a carpenter-bee dataset that contains no honeybee information. GPT-5.5, Kimi, and DeepSeek all say the sources lack honeybee nest details. Mistral invents an answer from its weights. Forcing it to attach every claim to a source gets closer, but it still leaks unsupported facts. For teams running RAG, the key metric is not answer quality alone; it is whether the model will admit ignorance.

Multimodal limitations. DeepSeek is text-only, so it drops out. On image tasks, GPT-5.5 identifies crocodiles and reads a small-font table to pick the cheapest plant; Kimi matches. Mistral calls the crocodiles alligators and selects a $2.30 plant when the cheapest is $0.58. Switching to Spanish makes even capable models stumble. Image understanding cannot be prompt-engineered away. If your app accepts images, each model needs a visual test set tied to your actual content and languages.

Agent efficiency. A travel-planning agent searches flights, hotels, budget, and activities. GPT-5.5 uses parallel tool calls and finishes in three calls. Kimi does not parallelize and takes extra turns. DeepSeek needs four calls. Mistral searches flights and hotels, then stops, skipping budget and activities. Each extra call is network latency and cost, so model choice changes both user experience and operating expense. Agent frameworks also shape behavior, which is why Fox warns that agents are the hardest workload to evaluate.

The evaluation stack. Fox moves from string checks for structured tool calls to LLM-as-judge. She recommends 200 to 800 RAG test cases, warns against built-in coherence evals, and stresses calibrating judges with human-labeled examples. She also introduces Assert, a Microsoft Research tool released a week before the talk, which generates test scenarios and rubrics from a YAML spec. The demo is promising, but the evidence is new, so treat it as a signal, not a standard.