Key concepts

The unit that gets evaluated is a release bundle: the dated model version, sampling settings, prompt, tool definitions, retrieval settings and guardrail versions, treated as one thing. Changing any part makes a new bundle. A judge has an identity too: model version, judge prompt and the example answers in it. "GPT-x judge with prompt v3" and "GPT-x" are different instruments, and a score means little unless the report names both.

"The model" is several things: the named model, a dated snapshot, an alias that points wherever the provider chooses, the serving platform and the product wrapped around it. Anthropic's 2025 bugs hit the same model at different rates on its own API, Amazon Bedrock and Google's Vertex AI, so pinning a version wouldn't have helped. Its 2026 problems sat in the Claude Code product layer, and the company said the API and inference layer weren't affected, so teams that built on the API were spared.

Below the bundle sit the eval case, the versioned dataset, the run, the trial (one attempt) and the trace; OpenTelemetry gives traces standard IDs, and nothing gives a dataset row or a grader one. Vendor names shift too: Galileo became Splunk Agent Observability within weeks of its sale.

A change moves through proposed, evaluated, rolled out, released and monitored, with blocked and rolled back as exits. Test cases have states of their own: candidate, active, quarantined for flakiness, retired when stale or saturated. In Anthropic's guidance, capability evals start with low pass rates and give the team something to climb, then "graduate" into regression evals, which should sit near 100%, once they pass reliably.

Models carry states set by the provider (active, legacy, deprecated, retired), with slightly different meanings at each lab and cloud. Judges move between calibrated, drifting and recalibrated, and vendors between free, paid, acquired and sunset. OpenAI's guide says its hosted Evals product goes read-only on October 31 and shuts down on November 30, 2026.

There's no single record of quality. Four stores hold pieces of it:

None of these is a ledger. I'd keep each eval result with its bundle, model version, judge version, dataset version and date in storage you control, in files git can diff. The EU AI Act makes deployers of high-risk systems keep logs for at least six months, longer than some model versions live.

The rules that bite are internal: the rubric (what "good" means for each failure mode), pass thresholds, which checks block a merge, who signs off, and who may override or lower a threshold. I'd add who may change the judge, since that changes every score after it.

Practice varies on strictness. Anthropic wants regression evals near 100%; Hamel Husain, a consultant who writes widely on evals, says the acceptable rate is a product decision. Will Larson found the tests for an internal agent useful as "directional context" rather than a blocking gate, because runs were flaky; one fix is to block only when a case fails three tries out of three.

After incidents, labs wrote stricter rules for themselves. OpenAI said it would treat model behaviour issues "as launch-blocking like we do other safety risks". Anthropic added a per-model eval suite for every Claude Code system-prompt change, soak periods and gradual rollouts. xAI said it would publish Grok's system prompts and require review after an unauthorised prompt edit in May 2025.

Contracts set the outer rule. Lab terms disclaim the accuracy of outputs: Anthropic's commercial terms, for example, provide outputs "as is" and make the customer responsible "to evaluate whether Outputs are appropriate". Testing quality is contractually the builder's job.

An eval result holds for one bundle, one judge and one dataset version on one date. An alias in production is an unannounced model change, and a judge called by alias is an unannounced change to your ruler. Datasets age too: a test set built before a policy change passes the agent confidently on the old policy.

Dates I'd keep on a calendar as of October 2026:

Notice periods set the rhythm: at least 60 days at Anthropic, at least six months for OpenAI's generally available models, as little as two weeks for some Google previews, and Microsoft's Foundry upgrades standard deployments automatically when a model retires. A team on several providers inherits the shortest clock in its stack. Changes can break the call itself: Anthropic's models from Claude 4.7 onward return an error if temperature or top_p is set to a non-default value. Benchmarks age as well; SWE-bench Verified lasted about 18 months from launch in August 2024 to OpenAI dropping it.

Tracing is closer to portable than evals. OpenTelemetry's AI conventions define spans for model calls, agent runs and tool calls, plus an event to carry an evaluator's score; they're still in "Development", so vendors fill gaps with their own SDKs or Arize's related spec, OpenInference.

Evals have no shared format: Promptfoo's YAML files, Inspect's Python tasks, Google ADK's evalset JSON and Braintrust's experiments don't move cleanly between tools, so lock-in sits in the eval history more than in the traces. Gates plug in through CI: Braintrust's GitHub Action can block a merge when a score falls below a threshold, and Promptfoo's has a fail-on-threshold setting. Inspect, from the UK AI Security Institute, is the closest thing to a neutral runner, and batch APIs at OpenAI, Anthropic and Google halve token prices for runs that can wait, which covers most nightly suites.

Labs disagree on how to grade agents. Anthropic grades the outcome (did the refund land, did the tests pass) so a valid path nobody anticipated isn't punished. Google's ADK scores tool calls against an expected sequence, with an exact-match default, which suits regression tests and penalises creativity. OpenAI's trace grading sits between, scoring whole traces against criteria.

The domain expert is the counterparty I'd underestimate least. Husain and Shankar recommend one domain expert who sets the standard, and that person's labels are the most valuable asset in the stack. Their method comes from practitioners who also sell a paid course on it, so I treat it as well-tested advice rather than settled research.

Users are also, in practice, the monitor of last resort, and their feedback is biased: OpenAI said its GPT-4o update over-weighted short-term thumbs-up and thumbs-down signals, which is how flattery looked better in the numbers.

No law tells a PM how to build a test set. The rules arrive in layers, and which ones apply depends on what the product decides:

For the help-desk company the binding layers are general law and its customers' contracts. A team whose agent screens job applicants or prices insurance gets the AI-specific and sector layers too, and bias testing becomes part of the eval suite. That side is covered in AI governance and model risk.

Reversal is easy for prompts and model pins and hard for everything else. A retired model can't be rolled back to, and agent actions stay done: in July 2025 Replit's coding agent ran destructive commands on a customer's database during a code freeze, and the fix afterwards was separating development from production. Compensation is close to nonexistent. After its April 2026 postmortem Anthropic reset subscribers' usage limits; I found no lab credits or refunds to builders for degraded quality after any incident.

The deployer owns what its AI says and does. Providers disclaim accuracy, cap liability at roughly a year of fees and write SLAs for uptime (Bedrock's credits apply when a region is down), so when a lab's change causes the regression, the builder still faces its customers. Inside the company, the PM who owns the quality bar owns the gate decision, so I'd record each sign-off with the report behind it.