Running an agent product: quality, evals and trust

Success, as defined now. Before reading anything, find out what "working" means to each team. Support may count a conversation as resolved when the customer didn't ask for a person, finance may count credits issued, and sales may quote the vendor's dashboard. Write down each definition, who reports it, and what it does with silence: does a customer who closed the chat window count as a success? In the payroll example I'd expect three definitions of "resolved" and nobody who has compared them.

Trace access. Settle who may read real conversations in week 1. Payroll chats contain salaries, bank details and addresses, so agree a redaction rule and a retention period with security and legal, and get yourself and one engineer access under it. It decides how fast you'll find the next regression: in Anthropic's September 2025 postmortem, privacy controls were one reason engineers couldn't quickly reproduce what users reported.

Read with engineering. Read about 100 recent sessions, and write a free-text note on at least 30 of them yourself, saying what went wrong in plain words. That's Hamel Husain and Shreya Shankar's method, and a score means little until someone has read the transcripts behind it. Mix random sessions with ones that ended in a complaint, a handoff or a retry, because complaints alone over-represent loud failures. Sit with an engineer and ask "why did it do that?" at each step: you'll catch where the agent missed what the user expected, and they'll catch the tool and retrieval errors underneath. Stop when new traces stop showing new kinds of failure.

The failure taxonomy. Group your notes, count each group and keep the five or six biggest. Mark each one as mostly a product problem (scope, wording, what it should do), a tool or data problem (wrong arguments, stale help articles) or a model problem. For the payroll agent the list might look like this:

• Answers state tax questions from memory instead of the help centre, sometimes wrongly.
• Says a credit was issued when the tool call failed.
• Loops when an employee's bank verification is pending.
• Re-sends payslips to an old email address.
• Hands off to a person without saying why, so the customer explains everything again.

The counts make it useful. In one of Husain's client cases, reading conversations showed an assistant getting dates wrong about two-thirds of the time, which no dashboard had shown; it's one consultant's story, though the method is widely copied. At scale you'll need help deciding where to read. Replit describes clustering sessions by summary and reading representative ones from each cluster, which is how it found environment setups failing silently and shipped a fix with a regression test the same day.

What users do next. For each failure group, write down what users did after it: retried, corrected the agent, asked for a person, left, or came back by email or phone. This is where the outcome metrics hide. A quiet exit can still count as a resolution, and a phone call shows up in someone else's queue. If you can, ask five or six customers who hit the top failures what they did and what they expected.

What changed recently. List every change from the last 60 days (prompts, models, tools, help-centre updates, defaults) next to the complaint volume. If the agent got worse with no code shipped, this is how you find the change that did it.

Trust gaps to fix now. Don't leave all the trust work for later. The costly public failures came in three kinds: the agent invented a claim (Cursor's support bot made up a device policy, Air Canada's chatbot a refund rule), took a destructive action (Replit's agent deleted a production database) or misreported what had happened (the same agent said the deletion couldn't be rolled back, when it could). Start two short lists while you read traces:

• Action register: every tool the agent can call, marked reversible, reversible with effort, or irreversible, and whether it touches money, data or someone outside the company. For the payroll agent, submitting an off-cycle payment is irreversible and external; updating an address is reversible.
• Claim register: every kind of statement the agent makes about policy, price, eligibility, deadlines or the status of its own work, marked grounded (it comes from a source the agent cites) or free (the model generates it).

Then fix the worst three now: a specific confirmation before each irreversible action, a handoff for any policy or tax question the help centre doesn't answer, and a check that the agent's closing message matches what the tools actually did. Phase 4 goes further.

A test set from real failures. Turn each annotated failure into a case you can replay: the conversation up to the failure, the tool responses mocked, and a check on what should happen next. The published guidance I found agrees on starting with 20-50 cases drawn from real failures. Sierra describes the same pattern at scale: its customer-experience teams review a sample of live conversations every day, any annotated conversation can become a test, and thousands run before each release. You don't need thousands, but you do need cases in both directions: where the payroll agent should issue a credit and where it shouldn't, where it should hand off and where it should answer. Run each case several times, because an agent can pass once and fail the next time; How a change reaches users in the evals guide covers the mechanics.

Version the set. Give the test set an owner, a version number and a change log. Cursor builds its benchmark from real sessions, refreshes it every few months and warns that scores can only be compared within the same version. Add 30 hard cases and the pass rate drops with no change in the agent. Retire cases that always pass, or move them to a small must-pass set.

Judges. You'll want a model to grade what code can't check, like whether a tax answer was correct and complete. Before trusting it, have a person label 100-200 examples, and measure separately how often the judge catches real failures and how often it passes good answers. GitHub audits its chat judge against human reviewers this way, and the evals guide's section on model-graded scoring has the evidence on judges. For agent products, judges also end up inside business metrics: Intercom's CX Score is a 1-5 rating that models produce from each conversation. A score like that needs the same re-checking as an eval judge, and I haven't found a vendor that publishes how often it does that.

The metric sheet. Agree a short list of metrics with written definitions before anyone reports them. I'd pick:

• One outcome metric the business already uses, usually resolution or task success.
• One cost metric, cost per task including the human minutes after a handoff.
• One time metric, time to complete.
• One trust metric, such as handoff rate or how often users correct or interrupt the agent.

For each one, write the definition, the edge cases (silence, a reopened conversation, a handoff, the same customer phoning the next day), how it can be gamed, and who owns it; Measuring progress has a table. Get finance and support to sign the sheet, since they're the ones who'll stop trusting the numbers if a definition moves later.

Why the definitions matter. Resolution definitions differ by vendor, and they're billing definitions first. Intercom charges about a dollar per resolution for its Fin agent and counts a conversation as resolved if the customer says the answer helped or goes quiet for 24 hours after its last answer; if they come back in the same conversation, the charge is reversed. Zendesk's announcement of per-resolution pricing gave no formal definition, Salesforce's reportedly requires at least two turns with no negative feedback and no escalation, and Sierra sets its billable outcome per contract. None of them means "the customer's problem went away", and under most of them a customer who gave up and phoned support counts as a success. Coding and app-building agents have no standard outcome metric at all; Replit says it watches whether users continue, what runs cost, sentiment and whether users ship something.

Outcome metrics win budget. I'd keep model-quality scores inside the team and report outcome metrics outward. Offline scores decide what the team tries next; budget conversations run on resolution, cost and retention, and outcome pricing pushes vendors and buyers the same way.

The release gate. Write down which tests run on which changes, the pass threshold, who may override a red result and how overrides are logged. Husain and Shankar treat the threshold as a product decision, and I agree: you decide how much risk a release can carry. For the payroll agent I'd make any case touching money or an irreversible action pass on every run, and the rest show no meaningful drop against the current version. Then rehearse: have an engineer push a deliberately worse prompt to a test branch and see whether the gate blocks it. Add a small prompt set that runs daily against the production agent, as GitHub's team does, so a provider-side change shows up within a day.

Plan for flakiness. In a December 2025 post, Will Larson described simulation-based evals that ended up as "directional context" instead of a blocking gate, because they were flaky. When cases fail at random, people learn to override the gate. Quarantine a flaky case with an expiry date, and fix or delete it before the date.

Everything is a release. List what counts as a change, because it's longer than the code: system prompts, tool schemas and descriptions, the model version or alias, defaults such as reasoning effort, help-centre content the agent retrieves, retrieval settings and caching. Keep them together in one release record. Sierra describes each release as a fixed snapshot of code, prompts, model versions and a frozen copy of the knowledge it uses, which makes rollback and comparison straightforward. Watch tools that let people edit prompts in production: each edit is a release that skips the gate unless someone puts it there.

Why the gate isn't enough. The two best-documented regressions both passed their gates.

• OpenAI, April 2025. An update to GPT-4o in ChatGPT made it flatter users. OpenAI wrote afterwards that its offline evals "generally looked good", it had no eval for sycophancy, and the A/B test favoured the new version because users rewarded the flattery. OpenAI rolled it back within days and said it would treat behaviour issues as launch-blocking.
• Anthropic, spring 2026. From March to April, three product-layer changes degraded Claude Code: a lower default reasoning effort, a bug that cleared the model's earlier reasoning, and a system-prompt line asking for shorter answers. The prompt change showed no regressions in the evals run before release; a broader test later found a 3% drop. The bug got past code review, automated tests and internal use. Anthropic committed to a per-model eval suite on every system-prompt change, plus soak periods and gradual rollouts.

Neither company lacked evals. In both cases the gate didn't cover the failure, and real traffic found it. Intercom's Fergal Reid said in October 2025 that new models often look strong in offline backtests and do worse in production, so Intercom relies on large A/B tests able to detect a tenth of a point of resolution rate. Cursor runs online experiments to catch output that "looks correct to a grader but feels worse to a developer".

Partial rollouts. Put the change in front of your own staff before customers. GitHub moves its own staff to a new model before every major change. Then a slice of real traffic. Decide beforehand what movement in the outcome metric counts and which guardrails stop the rollout: handoffs, reopens, corrections, complaints, cost per task. None of the companies I read publishes its traffic shares or stop rules, and the evals guide's default of 5% for a few days is a reasonable start. At the payroll agent's volume you won't detect a one-point change in resolution within days, so I'd watch the guardrails and read 20-30 traces from each arm. Averages can mislead here: Replit notes that a longer agent run can mean useful work or a loop, and only the trace tells you which.

Test one line at a time. When a prompt addition trades quality for something else, like shorter answers or lower cost, measure it on its own by removing it. That's how Anthropic found its 3% drop.

Model changes. Write a runbook for model changes, because some will be forced on you. Anthropic gives at least 60 days' notice before retiring a model, OpenAI at least six months for generally available models, and some Google previews as little as two weeks; Models and inference has the details. The runbook I'd use has six steps:

1. Pin the dated model version, never a moving alias.
2. Run the whole suite on the new version, several times per case.
3. Re-tune the prompts, which were written for the old model.
4. Check cost per task and any parameters the new model treats differently.
5. If the judge model is changing too, re-check it against your human labels.
6. Staff use, then a slice of traffic, then everyone.

Keep a calendar of retirement dates for every model you use, with who gets the notice. When OpenAI removed GPT-4o from ChatGPT in February 2026, two weeks after announcing it, many users had come to depend on how that model behaved, and it had already reversed an earlier removal in August 2025. Don't rebuild the roadmap around every new model, either: Reid said model upgrades gave Intercom a couple of points of resolution over about two years, and retrieval, ranking, prompts and workflow gave most of the rest. I'd treat a new model as one more experiment, and use each release to revisit ideas you dropped because the old model couldn't do them.

An incident playbook. Expect users to find the next problem before your monitors do; in the incidents collected for the evals guide, users or social media spotted more than half first. Write down how reports reach the team, who decides on a rollback and how fast. A rollback reverts the agent and nothing else: credits it issued, payments it submitted and answers sitting in a cache stay where they are, so the playbook needs a step for finding those. Every incident becomes a test case before the fix ships, and you decide in advance what affected customers are told and whether they get credits.

What the evidence supports. Published guidance from Microsoft, Google, Apple and the labs agrees on a short list: say what the agent can and can't do, show the plan and the progress, ask before consequential steps, make correction and undo easy, explain why, and hand control back when it's stuck. Almost nobody publishes measured effects of these patterns on retention or task success, so the evidence on agent UX outcomes is thin, and I'd treat this phase as well-reasoned practice more than proven method.

Good behaviour when wrong doesn't replace being right. In Nielsen Norman Group's 2026 study of site chatbots, people avoided or left bots over irrelevant answers and unclear scope, and in Stack Overflow's 2025 developer survey "almost right" output was the most common frustration with AI tools. My reading: the success rate earns the agent the right to be used, and its behaviour when wrong decides whether one failure becomes a quiet exit, an incident or a news story. A good recovery doesn't fully undo a failure either. Service-recovery research found satisfaction with the interaction can bounce back after a good recovery, with much weaker evidence that loyalty does.

Show progress. Design for three moments. Before the work: the plan, the scope and any cost. During it: what the agent is doing and a way to stop it. After it: what it did, what changed and what it couldn't do. Claude Code shows a plan and a live to-do list, and the user can step in at any point. Google's agent in Chrome writes each step to a work log the user can watch, take over or stop. Cursor's Plan Mode has the user edit the plan before any code is written. Keep it short: in Microsoft Research's small Magentic-UI study, some people found the plans too long, and slowness was the top complaint. The "after" part is the least published, so design it from your own traces. For the payroll agent that's a closing message like "I updated Ana's address and re-sent the March payslip to the new email. I didn't change Ana's tax withholding; that needs your accountant."

Ask before irreversible steps. OpenAI's system card for its Operator browsing agent (January 2025) has the best data I found. In 100 test tasks without safeguards, about one in eight ended in a mistake that bothered the user, and several couldn't easily be undone, such as an email to the wrong person. Asking for confirmation before actions like buying or sending cut that risk by about 90%. Anthropic's data (March 2026) shows the other side: users approve 93% of Claude Code's permission prompts. Both hold at once. A confirmation works when it's kept for the few steps that can really hurt, and turns into noise when it's on everything, because people stop reading what they approve.

What I'd do with that:

• Approval rate per action. Track how often each kind of prompt is approved and denied. A prompt approved nearly always and rarely followed by a correction protects nothing; replace it with a limit in the environment.
• Limits before prompts. Sandboxes, scoped credentials and draft-then-confirm reduce prompts without adding risk. Anthropic says sandboxing cut its internal permission prompts by 84%, and GitHub's coding agent can't approve or merge its own pull requests.
• Say the effect. "Submit an off-cycle payment of $412.50 to Ana Ruiz, arriving Thursday?" tells the employer what they're agreeing to, where "Allow this tool?" tells them nothing.
• A third option. Offer approve, reject or change it; people in the Magentic-UI study asked for exactly that.
• Autonomy that grows. In Anthropic's data, users with about 750 sessions turned on full auto-approve about twice as often as new users, and interrupted Claude Code more often too, which looks like closer supervision of the run more than rubber-stamping.

The labs differ on routine approvals. Anthropic argues for oversight of the whole run, with visibility and easy interruption; OpenAI and Google build confirmations for a defined list of consequential actions. All three confirm irreversible, high-impact steps, and that's the part I'd copy.

Offer undo, honestly. Apple's guidelines say to avoid automating destructive actions, to put edit, undo and retry next to what the AI produced, and to match the tools for fixing a mistake to its cost. Saying what undo can't do matters as much. Claude Code's checkpoint docs list what rewinding doesn't cover, such as files changed by shell commands, and say checkpoints don't replace version control. For the payroll agent, a submitted payment can't be undone, and a reversal is a separate process with the bank. The agent should say so in the confirmation itself.

Recover, then stop. Set a limit on retries. Anthropic's auto mode hands back to the user after three consecutive denials or 20 in a session, and OpenAI's guide to building agents recommends failure thresholds for the same reason. On an ambiguous request, a clarifying question beats a guess. Be careful with hedging language. In a 2024 study of 404 people, answers that opened with "I'm not sure" reduced over-reliance on wrong answers. Models are poorly calibrated about their own confidence, though. Check on your own traces whether hedged answers really are wrong more often before you show hedges to users.

Hand off to a person. Write down the triggers: the user asks, a set number of failed attempts, signs of distress, a policy exception, money above a threshold, a high-risk action. From Intercom's published guidance I'd copy a short note to the person taking over (summary, what was collected, checks run, why, suggested next step) and a message telling the customer what happens next and when. The US consumer finance regulator warned in 2023 about chatbots that trap people in "doom loops", and Klarna's CEO said in 2025 that customers should always be able to reach a person. If no human queue sits behind your agent, as with a coding agent, handing off means returning control with context: what was done, where it stopped and what it needs.

An honest account. Make the agent's closing report match what it did. Replit's agent deleted a production database during a code freeze and then told the user a rollback was impossible; the rollback worked (Examples). I'd add a gate test that compares the agent's final message with the tool calls in its trace, especially after an error, and fails when they disagree.

The weekly review. No company I found publishes a weekly quality review format, so this is my recommendation, built from what is published: Sierra's customer-experience teams review live conversations daily, Intercom tells its customers to check resolution, fix content gaps and recheck each week, and Husain and Shankar suggest reading 100 or more fresh traces every two to four weeks. I'd start the review once you have a taxonomy, around week 4, and keep it to 60 minutes with a fixed agenda:

1. The metric sheet against last week, with the changes shipped listed next to any movement.
2. Ten to twenty fresh traces, some random and some the worst by user signal, read together; new failure types go into the taxonomy.
3. Gate health: flaky cases, any judge that disagreed with a person, cases added or retired.
4. Changes shipping this week and their gate and experiment results.
5. What support and sales heard: the top customer questions about how the agent behaves.
6. Decisions made, with an owner and a date for each one.

The PM chairs it, with the engineering lead, a domain expert (for the payroll agent, a senior payroll specialist) and someone from support who handles escalations, plus data science if you have it. Once a month, retire cases that always pass and move capability tests the agent now passes reliably into the must-pass set. At each model release or retirement, run the upgrade drill from phase 3.

Discovery. Lovable's posting lists discovery as interviews, competitive research, eval analysis and prompt experiments in one sentence, and I'd run them together. Define the next piece of work as a failure group plus an outcome you can measure, such as "tax questions answered from memory: 12% wrong on the test set, target under 2%". Try the behaviour in a prompt or a quick prototype before engineering builds anything; if it works on 20 real cases, it earns a slot. Set a window, and if neither the eval nor the outcome metric moves within it, cut or redesign. Monthly, run the same 20 real tasks on a competitor's agent.

Intercom's Fin went from resolving about a third of conversations in 2023 to nearly 70% by early 2026, by Intercom's own account, through hundreds of small experiments, many of which failed. I'd run it as a two-speed roadmap: weekly fixes from failure groups through the gate, and at each model release a review of what the new model makes possible and which scaffolding it makes unnecessary.

Sales, support and marketing. This is the thinnest part of the evidence. Lovable's posting names it as part of the job, and Microsoft's 2019 guidelines open with making clear what the system can do and how well, but I found no published enablement playbook from an agent company. So the list below is my own recommendation, built from those anchors and the public cases:

• A capability sheet. One dated page per agent: what it does well, with the eval and number behind each claim; what it doesn't do; what it does only with a person; what changed since the last release. It's the only source sales and support quote.
• A claims review. Any number used outside the company names the test set, the date and the judge, and you and legal sign it off. Quote ranges and conditions, since a single headline number invites the buyer's own test to contradict it.
• A change note. For each release that changes what customers see: what changed, who's affected, what to tell customers and what stays the same.
• A support pack. The ten complaints most likely after a change, each with its cause and an answer, plus a route to you for anything new.
• "Available" means shipped. In 2025 the US advertising watchdog NAD found Apple's "available now" claims for Apple Intelligence features conveyed availability that didn't exist, and Apple dropped the wording.
• Pricing definitions. If you sell on outcomes, agree with sales how "resolved" is defined and who can audit it before a customer asks.