Common mistakes

1 min read · Chapter 9 of 10

What usually goes wrong

Symptom	Likely cause	First thing to check
Evals green, users complaining	The test set has no case for this failure	Recent complaints against the test set; add the cases
Quality dropped with no release on your side	An alias moved, the provider changed something, or a fallback model answered	Pinned versions; the provider's status and change notes
Scores jumped or fell overnight with no product change	The judge model or judge prompt changed	Re-score the anchor set
A/B winner turns out worse	The metric rewards pleasing, not helping	Guardrail metrics: escalations, refunds, corrections
Rollback didn't fix it	A cache or queued jobs, or actions already taken	Caches and the systems the agent acted on
Grounded answer is still wrong	The source document is stale	Freshness of the retrieved sources
Tracing bill grew tenfold	Full logging of large agent traces	Sampling rate and payload size
History lost after a vendor change	Datasets and results lived only in a console	Exports in storage you own