How to choose

How much evaluation does a change need? Five questions

1. Money or harm at stake? Refunds, policy statements and actions on customer data need the full suite, a human-reviewed sample and a slow rollout; a greeting tweak doesn't.
2. What changed? A prompt edit needs the regression set plus cases for the failure it targets; a model swap, the whole suite on the pinned new version and a look at cost per task; a new tool, cases for when to call it, when not to, and what happens when it fails.
3. Can you take it back? If the agent's actions can't be undone, or a cache will keep serving old answers, test harder before release.
4. Who set the deadline? A retirement notice gives you 60 days to six months. If your suite takes weeks to run and review, the deadline sets your quality bar.
5. Who checks the checker? If a model judge decides the gate, someone should have compared it with expert labels recently, and if a customer or regulator may ask for evidence, keep the report with every version recorded.

My defaults: code assertions on every commit; a smoke set of 50-100 cases with judges on the two to four failure modes that matter, on every pull request touching prompts, tools or model settings; the full set nightly and before release, with at least three runs per agent task, compared case by case against production. Must-not-break cases should pass close to 100% of the time, and the rest should show no meaningful drop against the current version. Cost and latency budgets are separate hard limits, and model and tool changes go to a 5% slice for a few days. A domain expert reads 100 fresh traces every two to four weeks, every incident becomes a test case before the fix ships, and every run records the full bundle and judge version, because a score without them can't be compared with anything.