AI evals and observability: how teams measure AI quality

Picture a company that sells help-desk software to mid-size online retailers. Its AI agent answers shoppers' tickets: it reads the retailer's policies, looks up the order and can issue a refund of up to $100 without a person, across about a million conversations a month. A new PM has just inherited the agent and the question nobody on the team can answer cleanly: is it getting better or worse, and how would we know before a retailer tells us?

The AI product team, here the help-desk company, builds on models that their providers keep changing. Anthropic, Google and OpenAI ship new versions, change defaults and retire old ones on published clocks, so the product changes even in a week when the team ships nothing. It tests each change on an eval platform such as Braintrust, LangSmith or Langfuse, which runs the agent against a fixed set of test cases and scores the answers with code checks and with other models acting as judges. Human graders check the hardest cases: in-house experts such as senior support leads, or contractors from vendors such as Scale AI and Surge AI, though those vendors sell mainly to the labs. Then the team releases to users. Production traces (the step-by-step record of what the agent did on each request) and user feedback flow back into the evals, which is how the test set learns about failures nobody wrote down.

That loop is weaker than it looks. In most public incidents so far, users noticed the problem before any monitor did. Of 15 dated public incidents I collected, users or social media flagged eight before anyone else, and not one names an automated production monitor as what caught it. Several of those teams had evals, and the evals passed.

"Evals" here means repeatable tests of an AI product's behaviour, with a score you can compare across versions; "observability" means the traces, dashboards and online scoring that show what's happening in production. They've become one market, because the same traces serve for debugging and as raw material for the next test case.

If you only remember a few things:

1. Test set: build your first test set from real failures in your own traces, before you pick a tool. A few dozen cases from real conversations are a better start than hundreds of generic ones, and the tool is the cheap part of the bill. How to start is in How a change reaches users, step by step.
2. Coverage: treat a passing eval as evidence about the cases in the test set and nothing more. Put every change in front of a slice of real traffic before everyone gets it, and score a sample of production continuously, because in the public incidents so far evals passed and users found the problem.
3. Judges: use model judges for volume, and check each one against 100-200 cases a human expert labelled before trusting its scores. Re-check whenever the judge model, its prompt or your product changes. In expert domains, judges have agreed with experts only about two-thirds of the time; more in Model-graded scoring: what's real.
4. Model changes: pin dated model versions and keep a regression suite you can re-run in a day. Providers retire models with as little as 60 days' notice, and every retirement is a release you didn't choose. The dates are in Effective dating.
5. Budget: plan for people's time first. A 500-case run scored by a model judge costs a few dollars in tokens and the same run reviewed by experts hundreds to thousands, while over a year the people who read traces and maintain the suite cost more than either. What that adds up to at each size is in How companies of different sizes use it.
6. Ownership: keep datasets, scores and judge prompts in storage you control, and assume a regression is yours to pay for. No provider contract I found covers output quality, and eval tools keep changing owners; see Liability allocation.

This guide is the quality layer across the other AI guides, and I don't repeat them. How agents run, call tools and get secured, with their own evals, is in AI agents: orchestration platforms; validation in the bank and regulatory sense in AI governance and model risk; tracing for ordinary software and AI SRE in Observability; model versions and retirements in Models and inference; and coding benchmarks such as SWE-bench in AI coding agents.

The main players

The companies behind the diagram's parties, layer by layer, in no particular order.

How they make money:

The tools are cheap and getting bought. Between January and August 2026 ClickHouse bought Langfuse, Promptfoo joined OpenAI, ServiceNow bought Traceloop, Cisco closed its purchase of Galileo, and Dynatrace announced on August 13 that it had agreed to buy Arize for $915 million, a deal still awaiting regulatory reviews. The biggest independents left are LangChain and Braintrust. Human data is a much bigger, separate market: Mercor reportedly passed a $2 billion gross annual run rate in June 2026, almost all of it from labs buying training data.

As of October 2026. Most private-company figures are self-reported or from press coverage, and this market changes owners every few months, so treat the list as a map to check.

Back to the help-desk company. Its model provider announces that the model under the agent retires in 60 days. The team pins the recommended replacement's dated version and re-runs its 500-case suite, which fails before any judge sees an answer: the newer model rejects a custom temperature setting the old code still sends. Once that's fixed, a code check confirms every refund call has the right fields, three model judges score tone, policy accuracy and escalation, and two senior support leads read the 60 cases where the judges and the old version disagree. The PM signs off, a feature flag in LaunchDarkly sends 5% of tickets to the new version, and online judges score a sample. A week later a retailer reports the agent quoting last season's returns window. The answer was faithful to a policy document nobody had updated, so the grounding check passed it, and that trace becomes test case 501 (the vendors in the story are illustrative).

How a change reaches users, step by step

Here's one change to the help-desk agent, from idea to production. Offline evals gate the release, then a slice of real traffic tests it before everyone gets it. A change can pass every eval and still fail in production, because evals only cover what the test set covers, so production traces feed new cases back in.

1. Proposed. Someone wants to change a prompt, swap the model or add a tool. The best proposals start from a failure found by reading traces ("the agent gets dates wrong on returns" is testable); others are forced, like a retirement notice. Write down which cases should improve, which must not get worse, and the cost and latency budget. What can go wrong: the criteria get written after the results. A 2024 study by Shreya Shankar and colleagues found people discover what "good" means partly by grading outputs, so the bar moves while you grade unless it's fixed in advance.
2. Evaluated. The change runs against the test set. Code assertions (format, tool arguments, banned phrases) and a smoke set of 50-100 cases run on every pull request. The full set runs before release, several times per case for agents, because an agent can pass a task once and fail it the next time. Model judges score the failure modes that matter, the new version is compared with production on the same cases, and someone reads the cases that flipped from pass to fail. This step carries the diagram's note: it's only as good as the test set. The exception on the diagram, Blocked, happens here when a regression shows up. What can go wrong: the suite misses the failure. OpenAI's offline evals for its April 2025 GPT-4o update "generally looked good", and it had no eval for sycophancy (telling users what they want to hear), which is what went wrong.
3. Rolled out. The change goes to a slice of real traffic, often a few percent of users, behind a feature flag with sticky assignment. Some teams first replay recent traffic against the new version without showing users the answers (a "shadow" run). Compare quality scores, escalations, refunds, complaints, cost and latency against the current version. What can go wrong: the metric you watch improves while behaviour gets worse. The GPT-4o A/B test favoured the new version, because users rewarded the flattery that was the problem.
4. Released. Everyone gets it. The previous bundle (model version, prompt, tool definitions, retrieval settings) stays recorded so you can go back. What can go wrong: problems that only show at scale. Google red-teamed AI Overviews, evaluated samples of typical queries and tested on live traffic before its May 2024 launch, and it still returned satire and forum jokes as answers on rare queries; Google wrote afterwards that its testing couldn't match millions of people searching for novel things.
5. Monitored. Traces are on, online judges score a sample of production (5% is a common start), and alerts watch judge scores, guardrail triggers, cost per task and loops. A domain expert reads fresh traces every few weeks, and new failures become test cases. What can go wrong: the monitor doesn't notice. In Anthropic's September 2025 postmortem, three serving bugs degraded answers for weeks, misrouting 16% of one model's requests at the worst hour, and the company said its evaluations "didn't capture" what users reported.

Rolled back is the other exception, when an incident forces you back to the previous version: flip the flag or re-pin the old bundle, which works only while the old model still exists. A rollback doesn't undo what the agent already did (refunds issued, emails sent) or clear answers a cache keeps serving. Write the incident up as a test case before the fix, and add a check for that class of failure to the gate. OpenAI did this after GPT-4o, saying it would treat behaviour issues as launch-blocking; Anthropic did it after an April 2026 postmortem about changes to its Claude Code product, adding an eval suite on every system-prompt change and gradual rollouts.

Blocked needs a decision, since a red result has four common causes: a real regression, a bug in the eval, noise or a changed judge, or a trade-off the team accepts. Eval bugs are commoner than people expect: Anthropic reported a model scoring 42% on an agent benchmark before grading bugs were fixed and about 95% after. The options are fixing the change, fixing the eval (with someone checking the bar didn't drop), quarantining a flaky case with an expiry date, or overriding with a named approver and a written reason.

Cheat sheet: three kinds of change

A prompt edit has the smallest blast radius and the easiest path around the gate: tools such as Langfuse let non-engineers change prompts in production "without needing to involve engineering", a release that skips review unless someone gates it. A model upgrade has the widest effect, because the old prompts were tuned to the old model. A new tool needs evals in both directions: Anthropic's web-search evals checked that the model searched when it should and didn't when it shouldn't.

Reading an eval report and a trace

An eval report I'd trust shows, for each failure mode, the pass rate of the new and current versions on the same cases, with an interval around the difference. It lists the cases that flipped, cost and latency per task, and the version of the dataset, the judge model and prompt, and the model under test. For agents it reports both "passed at least once in k tries" and "passed every time in k tries". Users feel the latter: in Anthropic's example, 75% success per attempt means about 42% success on three attempts in a row.

A trace is a tree for one request: the agent's run at the top, each model call, tool call and retrieval underneath, with inputs, outputs, tokens, cost and timing. It serves for debugging, for building test cases and for carrying online judge scores. OpenTelemetry, the open standard most tracing tools speak, has conventions for AI calls and agent steps, still all marked "Development" as of October 2026. Recording prompts and answers is off by default in them, so the team has to decide who may read real conversations; Anthropic named privacy controls as one reason its 2025 bugs were slow to reproduce.

Public benchmarks and your own evals

Public benchmarks are for shortlisting models, and they're less solid than their fame suggests. OpenAI stopped reporting SWE-bench Verified, the best-known coding benchmark, in February 2026 after finding flawed tests and signs that models had memorised answers (contamination); AI coding agents covers what came next. Experts found errors in about 6.5% of MMLU's questions, and a 2025 study found Meta tested 27 private variants on Chatbot Arena before Llama 4's launch.

How much evaluation does a change need? Five questions

1. Money or harm at stake? Refunds, policy statements and actions on customer data need the full suite, a human-reviewed sample and a slow rollout; a greeting tweak doesn't.
2. What changed? A prompt edit needs the regression set plus cases for the failure it targets; a model swap, the whole suite on the pinned new version and a look at cost per task; a new tool, cases for when to call it, when not to, and what happens when it fails.
3. Can you take it back? If the agent's actions can't be undone, or a cache will keep serving old answers, test harder before release.
4. Who set the deadline? A retirement notice gives you 60 days to six months. If your suite takes weeks to run and review, the deadline sets your quality bar.
5. Who checks the checker? If a model judge decides the gate, someone should have compared it with expert labels recently, and if a customer or regulator may ask for evidence, keep the report with every version recorded.

My defaults: code assertions on every commit; a smoke set of 50-100 cases with judges on the two to four failure modes that matter, on every pull request touching prompts, tools or model settings; the full set nightly and before release, with at least three runs per agent task, compared case by case against production. Must-not-break cases should pass close to 100% of the time, and the rest should show no meaningful drop against the current version. Cost and latency budgets are separate hard limits, and model and tool changes go to a 5% slice for a few days. A domain expert reads 100 fresh traces every two to four weeks, every incident becomes a test case before the fix ships, and every run records the full bundle and judge version, because a score without them can't be compared with anything.

What's different here

How the money moves

The product team pays almost everyone, and the biggest bill is its own people's time.

One buyer-data platform puts the median LangSmith contract at about $40,000 a year. By my rough arithmetic, bills for the same traffic vary about tenfold between vendors, because each counts a different unit (traces, spans, gigabytes, scores), so I'd compare quotes at my own trace size and retention, and sample production traffic before haggling over plans.

Here's what one eval run of 500 cases costs, scored by a model judge or by people, from list prices as of October 2026. A short chat case is about 1,500 tokens for the judge to read, an agent trace about 12,000.

Measuring agreement needs two independent reviewers, which doubles the human rows. A judge costs between a tenth and a thousandth of human review per case, so the usual design is code checks, then a model judge for volume, then people for a calibration set of 100-200 cases per failure mode. The platform's share of one run is effectively nothing.

The run is the cheap part. By my rough estimate, building a suite (reading about 100 traces, writing 20-50 cases and a rubric per failure mode, wiring it into CI) takes 40-120 engineer-hours, then a few hours per release. For one product running the suite twice a week, tokens come to hundreds or a few thousand dollars a year, and people to between a quarter of a person and two people.

Most of the market's money never touches a product team: human-expert data sold to labs runs to billions a year, while eval tooling is hundreds of millions by my rough estimate. Analyst sizings disagree by several times, and one firm's whole data-labelling forecast for 2030 is smaller than Mercor's reported run rate now, so I wouldn't quote any of them.

How companies of different sizes use it

Nobody publishes eval spend by company size, so the bands and figures below are my rough estimates from list prices, vendor case studies and LangChain's survey. Spend means cash on tools, tokens and outside review; staff time comes on top and is usually bigger.

A startup (under 50 people) runs on one lab's API, keeps a handful of test cases in its repo with an open-source runner such as Promptfoo or DeepEval, and uses a free tracing tier. A model judge scores, the founders read traces, and the engineer who owns the agent owns quality. Startup programs make year one nearly free: up to $10,000 in LangSmith credits for venture-backed startups, six to twelve months at Braintrust. My rough estimate is under $10,000 a year. A buyer asking for test evidence, an embarrassing regression or a retirement notice pushes it on.

A scale-up (about 50 to 500 people) has one or two AI features or agents with a regression set of 100-1,000 cases gating releases in CI, online judges on sampled traffic and regular error analysis. It buys a specialist platform such as Braintrust, LangSmith or Langfuse. The help-desk company is this size: an AI engineering lead owns the framework, a dedicated evals engineer often builds it, and the PM owns the definition of good and the gate. Notion, in a case study from its eval vendor, says its automated evals let it ship a new frontier model in under 24 hours. My rough estimate is $20,000-200,000 a year, plus one to four people's time. Security reviews, shared datasets across teams and customer audits push it on.

A mid-market company (about 500 to 2,000 people) is often not a software company at heart, and its AI features sit inside existing support, search or document workflows. It uses what it already pays for (Datadog, its cloud's evaluators, suites such as ServiceNow), often without systematic offline evals, and subject experts review answers in spreadsheets under a head of engineering or data. My rough estimate is $60,000-550,000 a year, with human review the most variable line. Another product line, a regulated use in banking or health, or a CFO asking for an AI cost line pushes it on.

An enterprise (about 2,000 to 10,000 people) runs a shared eval and tracing service from a central platform team, several model providers behind a gateway, and a model risk or AI governance function for the high-risk uses. It buys its observability vendor's AI module, self-hosts Langfuse or Phoenix where data control matters and runs red-team tools such as Promptfoo or Microsoft's PyRIT, while each business unit owns its test sets. My rough estimate is $250,000-2.6 million a year, mostly human review and expert panels, plus an evals and quality group of perhaps five to 20 people. Internal audit, the EU's high-risk timetable or an incident push it on.

A Fortune 500 company (revenue above about $7.5 billion) has many teams on many tools under a central AI policy. A few product lines have mature suites, the rest rely on evals built into the suites they buy, and external expert panels review clinical, legal or financial output. Langfuse says 63 of the Fortune 500 use it and Promptfoo claims more than a quarter, both self-reported and overlapping. A chief AI or data officer sets policy, business units own outcomes, and model risk and internal audit challenge them. My rough estimate is $1.7-18 million a year, with expert review the largest line. There's no next size up; the pressure comes from regulators and incidents.

What changes as a company grows (spend is my rough estimate):

Who holds the power

• Model providers hold the clock: model versions, aliases, retirement dates, parameters and terms. Their own tooling is priced near cost, which pressures everyone selling evals, and they're unpredictable platform partners: Promptfoo joined OpenAI in March 2026 and OpenAI announced the shutdown of its own hosted Evals that June, while Anthropic's guidance names outside tools instead of selling a platform.
• Observability incumbents hold the budget, the security review and the agent already on every host. Datadog's free tier makes it the default for anyone already paying it, and Dynatrace and Cisco bought their way in.
• Independent platforms hold the eval workflow, review screens, dataset tools and neutrality across providers. Their leverage is weaker than it looks, because traces are becoming portable and eval files are small.
• Domain experts hold the definition of good, and they're scarce; whoever controls their time controls how fast the test set improves.
• Product teams hold the gate and the rollout, and users hold the real ground truth and, so far, the job of finding regressions.
• Labs as buyers hold the power in human data. Scale AI reportedly lost Google and OpenAI work after Meta took a 49% stake in 2025, and Meta reportedly paused work with Mercor after a breach there in March 2026.

Tailwinds and headwinds

These are the forces I'd watch over the next couple of years. Each helps some players and hurts others, and the companies that already hold the traffic, the cloud account or the model come out ahead on most of them.

Tailwinds:

Headwinds:

The one thing I'd watch is whether the clouds start bundling the human review screens and dataset tools, because that's the part where the independents still lead.

How the rules work

Most AI products face no rule about how to test them. Testing duties exist for EU high-risk systems and the most capable general-purpose models, medical devices, bank models (generative AI now excluded in the US) and through contracts; everything else is general law applied after something goes wrong.

EU AI Act. High-risk systems must be tested before going on the market against metrics and thresholds set in advance, with declared accuracy and post-market monitoring over the system's life. The Digital Omnibus moved those duties for Annex III uses (credit, hiring, insurance pricing and others) to December 2, 2027. A support chatbot like the help-desk agent sits in the limited-risk tier: it must tell people they're talking to AI, and nothing in the Act tells it how to test. Separately, providers of the most capable general-purpose models have owed model evaluations and adversarial testing since August 2025, enforced from August 2, 2026. As of early October 2026 no AI Act fine had been found.

NIST and ISO. NIST's Generative AI Profile (AI 600-1, July 2024) is voluntary, and it says plainly that current pre-deployment tests "may be inadequate" and are often mismatched to how a system is used. Its suggested actions read like an eval checklist, from checking that each metric measures what it claims to evaluating in real-world conditions. ISO/IEC 42001 certifies a management system, so its auditor checks that you define verification criteria, keep evidence and monitor in operation, and never looks at your pass rate.

FDA and banks. The closest regulatory model of a good release gate is the FDA's predetermined change control plan for AI devices: the maker agrees in advance which changes it may make, how it will test them and what thresholds they must meet, and changes inside that envelope ship without a new submission. One study counted only 26 devices with authorised plans by May 2025. US bank regulators' April 2026 guidance excluded generative and agentic AI, so its ideas (independent challenge, outcome analysis, ongoing monitoring) describe a good eval program that no US bank is required to apply to an LLM.

Claims about quality. Eval numbers end up in marketing, and that's regulated. Texas settled with Pieces Technologies in 2024 over a claimed "critical hallucination rate" below 0.001%, and the SEC fined two investment advisers $400,000 combined in March 2024 for overstating their AI. A Stanford study found legal research tools marketed as "hallucination-free" made things up on roughly 17-33% of queries.

Customers. For a B2B seller, procurement bites before any law. AI addenda to security questionnaires ask whether the customer can pin a version, how much notice it gets of model changes, whether it can run its own evals beforehand, and how fast the vendor can roll back, while the vendor's own model provider may give it only 60 days' notice.

What mistakes cost

Let's say the help-desk company ships a prompt-and-model change that makes 5% of its million monthly conversations worse, and it takes three weeks before support tickets and satisfaction scores expose it. Rough arithmetic on stated assumptions:

• The damage: about 30,000-40,000 bad conversations. If each one means a person has to step in, at $2-6 a contact, that's $60,000-240,000, mostly paid first by the retailers' support teams and then by the help-desk company in credits and churn.
• What it gets back: nothing from its model provider. No SLA I found covers quality, accuracy is disclaimed, and any claim would be capped at a year of fees, about $54,000-162,000 here. General liability insurance increasingly excludes generative AI.
• What prevention costs: a regression suite of 500-2,000 cases run three times on a mid-tier judge costs about $20-300 per run, a human review of 100-200 judged cases $100-1,000, and keeping the suite current perhaps $30,000-90,000 a year in engineering time. One avoided three-week incident pays for the suite for roughly one to five years, depending on how often incidents happen.

The record so far:

The cash is small and falls on whoever ran the product; the large numbers are reputation and market value. Users, journalists or customers found almost all of these. In Anthropic's 2026 case, user reports started in early March, initially looked like normal variation, and the last fix landed on April 20, after the changes had passed internal evals and staff use. OpenAI's sycophancy update passed offline evals and an A/B test, and Google's AI Overviews passed red-teaming and live tests. Each of these teams had evals and lacked the right test case.

Model-graded scoring: what's real

My view as of October 2026: model judges are good enough to be the scaling layer of an eval program and too unreliable to be its source of truth. On general questions they agree with people about as well as people agree with each other, noticeably less with experts in specialist fields, and their biases are known and can be designed around. The weakest evidence is on what happens when the judge model itself changes, which every team will face.

How far judges agree with people

• General chat: the study that made judges popular (Zheng and colleagues, 2023) found GPT-4 agreed with human preferences about 85% of the time on non-tied votes, slightly above the 81% at which humans agreed with each other.
• Expert domains: in a 2024 study, dietitians and mental-health experts agreed with model judges only 68% and 64% of the time. A 2026 review of 134 healthcare studies called alignment "moderate to strong" and said it varies a lot by task. Experts disagree among themselves too, so the fair benchmark is expert-to-expert agreement.
• Hard correctness: on JudgeBench, pairs of answers where one is subtly wrong, strong judges of the GPT-4o era did only slightly better than chance.
• Measuring it properly: a large 2026 preprint (21 judges, about 541,000 judgments) found simple percent agreement overstates quality: a chance-corrected measure came out 33-41 points lower on one standard benchmark, and a judge's ranking moved up to 14 places between benchmarks. It's one paper and hasn't been replicated.
• Narrow, tuned judges do better: Husain reports a judge for one company's query assistant reaching over 90% agreement with its expert after three rounds of prompt changes, a single consultant's case that fits what labs advise.

I'd measure a judge by how often it catches real failures and how often it passes real successes, separately, because overall agreement looks fine on a test set where 95% of cases pass.

Known biases

• Position: judges favour an answer for where it sits. A study of 15 judges on 22 tasks found the effect varies by judge and task and grows when the answers are close in quality; in the 2023 study GPT-4 kept its verdict after a swap only 65% of the time with the default prompt. Run both orders and count a flip as a tie.
• Length: judges like longer answers. AlpacaEval, a leaderboard scored by a model judge, built a length-controlled version that raised its correlation with human rankings on Chatbot Arena from 0.94 to 0.98.
• Self-preference: models rate their own outputs higher, more so the better they recognise their own writing; GPT-4 recognised its own output 73.5% of the time in one 2024 study. Anthropic calls grading with a different model "generally best practice", which matters most when the product and the judge move to the same new family at once.
• The rest: a 2024 audit, CALM, measured 12 kinds of judge bias and found significant ones persist even in strong models.

The labs' advice converges: ask for reasoning before the score, prefer pass or fail (or a pairwise choice) over a 1-5 scale, give the judge an "unknown" option, use one judge per dimension, control for length, and validate against human labels before scaling up.

When the judge changes

Hosted models change behind a fixed name: in a 2023 study the same GPT-4 service went from 84% to 51% on identifying prime numbers between March and June. Critics said part of that was a change in answer format; either way the name stayed and the behaviour moved, and a judge called through an alias is exposed in the same way.

Direct evidence on judges is thin. A single-author preprint from June 2026 showed in simulation that a silent judge update looks exactly like a change in your product unless you re-score a fixed set of human-labelled anchor cases; that check caught the update in all 60 simulated runs, while a common statistical alarm went off on 75% of streams where nothing had changed. I found no study measuring real score shifts when a GPT, Claude or Gemini judge version changes, and no lab document telling teams to pin their judge.

What I'd do anyway: pin the judge's dated version, keep 100-200 human-labelled anchor cases per failure mode, re-score them on any judge or judge-prompt change, and mark old and new scores as not comparable if agreement moves.

When you still need people

• Defining good: criteria change as people grade (the Shankar study again), so a person has to look at real outputs before any rubric exists. Husain and Shankar's method starts here: read about 100 traces, note what's wrong with each, group the notes into failure modes and count them. In one of Husain's client cases, this showed an assistant failing on date handling 66% of the time; it's one consultant's story, though the method is widely copied.
• Calibration and spot checks: every judge needs a human-labelled set to be checked against after each change, and someone should read a sample of its verdicts, especially disagreements.
• Expert domains and subjective quality: law, medicine, finance and anything where taste matters, since that's where agreement drops.
• Signals that reward the wrong thing: thumbs-up rates can improve while behaviour worsens, as with GPT-4o, whose expert testers had said it "felt" slightly off before launch.

Human review is still more common than model judges. In LangChain's own survey of 1,340 developers, a vendor's sample, 59.8% used human review and 53.3% model judges; 89% had some observability, and only 52.4% ran offline evals. Simulated users, where a model plays the customer, need the same caution: two 2026 studies found simulators more cooperative than real people, inflating agent success, with agent scores moving by up to 9 points depending on which model played the user. They help with regression and coverage and say little about how real people will fare.

Grader types side by side

Questions to ask about any judge score

1. Which judge version and prompt produced it, and when was it last checked against human labels?
2. How often does it catch real failures, and how often does it fail good answers?
3. Is the judge from the same model family as the product it grades?
4. If the judge changed tomorrow, which human-labelled cases would tell us?

What usually goes wrong

Words that mean something else here

What surprised me

Placeholders in your voice, drafted from the research and the earlier guides. Rewrite each with your own moment.

"Teams without evals ship regressions." In payments, a failing test stopped a release. Here the best-known regressions came from labs that had evals, and the evals passed.

"Scoring is the expensive part." A model judge scores 500 cases for a few dollars. The time went into a support lead deciding what a right answer looks like.

"Monitoring catches what testing misses." In observability an alert usually fired before customers noticed. In the AI incidents I read, outsiders noticed first almost every time.

"Human data is the eval budget." The billion-dollar human-data companies sell mostly to labs, and a product team's graders are usually its own staff.

Sources

Undated entries were read on October 12, 2026; "search result" means seen only as a search snippet or summary. Company figures are self-reported unless they come from a filing or a regulator, vendor surveys cover the vendor's own respondents, and list prices change often.

Model providers' guidance, docs and terms

• Anthropic: Demystifying evals for AI agents (Jan 2026); define success criteria and build evaluations; model deprecations (Oct 2026); Commercial Terms (effective Jun 2025); pricing
• OpenAI: evaluation best practices, including the hosted Evals deprecation notice; trace grading; pricing
• Google: ADK evaluation; Gen AI evaluation service overview and evaluation metrics; agent and model evaluations generally available (Jul 2026); Gemini API pricing
• AWS: AgentCore pricing; Bedrock pricing
• Microsoft: Foundry observability pricing; Foundry observability concepts (Aug 2026)
• Databricks, MLflow 3 (search result)

Postmortems and incidents

• Anthropic: a postmortem of three recent issues (Sep 17, 2025); April 23 postmortem (Apr 2026)
• OpenAI: expanding on what we missed with sycophancy (May 2025, search result), read via Simon Willison's summary; VentureBeat, sycophancy rollback; TechCrunch, why ChatGPT became sycophantic (Apr 2025, search result); VentureBeat, GPT-5 rollout (Aug 2025, search result)
• Google: AI Overviews: about last week (May 2024); Gemini image generation (Feb 2024)
• Others: AI Incident Database, Cursor support bot (Apr 2025, search result); Manatt, Air Canada ruling (2024, search result); NBC News, Grok prompt edit (May 2025, search result); OECD.AI, Replit database deletion (Jul 2025, search result); CTV News, Deloitte partial refund (Oct 2025, search result); Silicon Republic, Apple pauses news summaries (Jan 2025, search result); Nation's Restaurant News, McDonald's ends IBM drive-thru test (Jun 2024, search result); The Markup, NYC chatbot (Jan 2026, search result)

Research on judges, evals and benchmarks

• Zheng and colleagues, Judging LLM-as-a-judge (2023); position bias in LLM judges (2024, rev. 2025); Panickssery and colleagues, self-recognition and self-preference (2024); length-controlled AlpacaEval (2024); CALM (2024); Szymanski and colleagues, LLM judges in expert knowledge tasks (2024); LLM-as-a-judge in healthcare, a scoping review (May 2026); JudgeBench (2024); Norman, Rivera and Hughes, Reliability without Validity (Jun 2026); Who Drifted: the System or the Judge? (Jun 2026, preprint)
• Chen, Zaharia and Zou, How is ChatGPT's behavior changing over time? (2023); Shankar and colleagues, Who Validates the Validators? (2024)
• Simulated users: Lost in Simulation (Jan 2026); Mind the Sim2Real Gap (Mar 2026, rev. Jul 2026)
• Benchmarks: OpenAI, why we no longer evaluate SWE-bench Verified (Feb 2026, search result); Singh and colleagues, The Leaderboard Illusion (2025); TechCrunch, Llama 4 Maverick ranks below rivals (Apr 2025, search result); MMLU-Redux (2024)
• Magesh and colleagues, hallucination in legal research tools (2024, search result); RAGAS, faithfulness metric

Practitioners and surveys

• Hamel Husain: Your AI Product Needs Evals (Mar 2024); creating an LLM judge; a field guide to improving AI products; with Shreya Shankar, evals FAQ (Sep 2026); their paid course listing (search result)
• Will Larson, evals to validate workflows (Dec 2025)
• LangChain, State of Agent Engineering (n=1,340, fielded Nov-Dec 2025; vendor survey)

Tracing, tools and standards

• OpenTelemetry: generative AI conventions (moved notice), new repository and agent spans (search result)
• Langfuse, prompt management and LLM evaluation guide (Nov 2025, updated Jul 2026); LaunchDarkly, AI Configs; Braintrust, run evals in CI (search result); Promptfoo, CI/CD (search result)
• UK AI Security Institute, Inspect Evals (Nov 2024)

Rules and regulators

• EU AI Act: Article 9, Article 15, Article 72
• NIST, AI 600-1, Generative AI Profile (Jul 2024)
• FDA, predetermined change control plan webinar (Jan 2025) and AI-enabled device list
• ISMS.online, ISO/IEC 42001 A.6.2.4 summary (search result)
• The Globe and Mail, SEC fines two advisers over AI claims (Mar 2024, search result)
• Delve, what security questionnaires should ask AI vendors (vendor, search result)

Market, companies and pricing

• Deals: ClickHouse, welcomes Langfuse (Jan 2026); Orrick, Langfuse acquired (Jan 2026); Dynatrace, to acquire Arize (Aug 2026); Cisco, intent to acquire Galileo (Apr 2026, updated May 2026); Splunk, Galileo; Security Boulevard, OpenAI acquires Promptfoo (Mar 2026, search result); Calcalist, ServiceNow and Traceloop (Mar 2026, search result); CoreWeave, completes Weights & Biases (May 2025, search result); Snowflake, acquires TruEra (May 2024, search result)
• Funding: Fortune, LangChain at $1.25 billion (Oct 2025, search result); Axios Pro, Braintrust Series B (Feb 2026, search result); TechCrunch, Arena at $3.1 billion (Oct 2026); The AI Insider, Vals AI Series A (Aug 2026); The Next Web, Raindrop Series A (Sep 2026); Judgment Labs, $32 million round (May 2026, search result)
• Braintrust, Notion case study (vendor)
• Pricing: LangChain, pricing and LangSmith pricing; a third-party tracker, LangSmith price change (Jul 2026); Braintrust pricing; Langfuse pricing; Datadog, LLM Observability and a pricing tracker (Q2 2026); Vendr, LangChain purchase data

Human data

• Dealroom, Mercor at a $2 billion gross run rate (Jul 2026); Computing, Meta pauses work after Mercor incident (Apr 2026, search result)
• Sacra, Surge AI and Handshake (estimates); AI Weekly, micro1 at $4 billion (Sep 2026)
• Scale AI: NBC Bay Area, after Meta's investment (Jun 2025, search result)
• Prolific, pricing; AI Weekly, Amazon closes Mechanical Turk (Aug 2026, search result)

Reused from earlier guides on this site

• Model retirement notice periods by provider, the Digital Omnibus dates, US bank model risk guidance, lab liability caps and SLAs, the Air Canada ruling, Texas v. Pieces and insurance exclusions: Models and inference, AI agents: orchestration platforms and AI governance and model risk