How it works

How a change reaches users, step by step

Here's one change to the help-desk agent, from idea to production. Offline evals gate the release, then a slice of real traffic tests it before everyone gets it. A change can pass every eval and still fail in production, because evals only cover what the test set covers, so production traces feed new cases back in.

1. Proposed. Someone wants to change a prompt, swap the model or add a tool. The best proposals start from a failure found by reading traces ("the agent gets dates wrong on returns" is testable); others are forced, like a retirement notice. Write down which cases should improve, which must not get worse, and the cost and latency budget. What can go wrong: the criteria get written after the results. A 2024 study by Shreya Shankar and colleagues found people discover what "good" means partly by grading outputs, so the bar moves while you grade unless it's fixed in advance.
2. Evaluated. The change runs against the test set. Code assertions (format, tool arguments, banned phrases) and a smoke set of 50-100 cases run on every pull request. The full set runs before release, several times per case for agents, because an agent can pass a task once and fail it the next time. Model judges score the failure modes that matter, the new version is compared with production on the same cases, and someone reads the cases that flipped from pass to fail. This step carries the diagram's note: it's only as good as the test set. The exception on the diagram, Blocked, happens here when a regression shows up. What can go wrong: the suite misses the failure. OpenAI's offline evals for its April 2025 GPT-4o update "generally looked good", and it had no eval for sycophancy (telling users what they want to hear), which is what went wrong.
3. Rolled out. The change goes to a slice of real traffic, often a few percent of users, behind a feature flag with sticky assignment. Some teams first replay recent traffic against the new version without showing users the answers (a "shadow" run). Compare quality scores, escalations, refunds, complaints, cost and latency against the current version. What can go wrong: the metric you watch improves while behaviour gets worse. The GPT-4o A/B test favoured the new version, because users rewarded the flattery that was the problem.
4. Released. Everyone gets it. The previous bundle (model version, prompt, tool definitions, retrieval settings) stays recorded so you can go back. What can go wrong: problems that only show at scale. Google red-teamed AI Overviews, evaluated samples of typical queries and tested on live traffic before its May 2024 launch, and it still returned satire and forum jokes as answers on rare queries; Google wrote afterwards that its testing couldn't match millions of people searching for novel things.
5. Monitored. Traces are on, online judges score a sample of production (5% is a common start), and alerts watch judge scores, guardrail triggers, cost per task and loops. A domain expert reads fresh traces every few weeks, and new failures become test cases. What can go wrong: the monitor doesn't notice. In Anthropic's September 2025 postmortem, three serving bugs degraded answers for weeks, misrouting 16% of one model's requests at the worst hour, and the company said its evaluations "didn't capture" what users reported.

Rolled back is the other exception, when an incident forces you back to the previous version: flip the flag or re-pin the old bundle, which works only while the old model still exists. A rollback doesn't undo what the agent already did (refunds issued, emails sent) or clear answers a cache keeps serving. Write the incident up as a test case before the fix, and add a check for that class of failure to the gate. OpenAI did this after GPT-4o, saying it would treat behaviour issues as launch-blocking; Anthropic did it after an April 2026 postmortem about changes to its Claude Code product, adding an eval suite on every system-prompt change and gradual rollouts.

Blocked needs a decision, since a red result has four common causes: a real regression, a bug in the eval, noise or a changed judge, or a trade-off the team accepts. Eval bugs are commoner than people expect: Anthropic reported a model scoring 42% on an agent benchmark before grading bugs were fixed and about 95% after. The options are fixing the change, fixing the eval (with someone checking the bar didn't drop), quarantining a flaky case with an expiry date, or overriding with a named approver and a written reason.

Cheat sheet: three kinds of change

A prompt edit has the smallest blast radius and the easiest path around the gate: tools such as Langfuse let non-engineers change prompts in production "without needing to involve engineering", a release that skips review unless someone gates it. A model upgrade has the widest effect, because the old prompts were tuned to the old model. A new tool needs evals in both directions: Anthropic's web-search evals checked that the model searched when it should and didn't when it shouldn't.

Reading an eval report and a trace

An eval report I'd trust shows, for each failure mode, the pass rate of the new and current versions on the same cases, with an interval around the difference. It lists the cases that flipped, cost and latency per task, and the version of the dataset, the judge model and prompt, and the model under test. For agents it reports both "passed at least once in k tries" and "passed every time in k tries". Users feel the latter: in Anthropic's example, 75% success per attempt means about 42% success on three attempts in a row.

A trace is a tree for one request: the agent's run at the top, each model call, tool call and retrieval underneath, with inputs, outputs, tokens, cost and timing. It serves for debugging, for building test cases and for carrying online judge scores. OpenTelemetry, the open standard most tracing tools speak, has conventions for AI calls and agent steps, still all marked "Development" as of October 2026. Recording prompts and answers is off by default in them, so the team has to decide who may read real conversations; Anthropic named privacy controls as one reason its 2025 bugs were slow to reproduce.

Public benchmarks and your own evals

Public benchmarks are for shortlisting models, and they're less solid than their fame suggests. OpenAI stopped reporting SWE-bench Verified, the best-known coding benchmark, in February 2026 after finding flawed tests and signs that models had memorised answers (contamination); AI coding agents covers what came next. Experts found errors in about 6.5% of MMLU's questions, and a 2025 study found Meta tested 27 private variants on Chatbot Arena before Llama 4's launch.