Field Guide
AI evals and observability: how teams measure AI quality
06
Market and trends
3 min read · Chapter 6 of 10
Who holds the power
- Model providers hold the clock: model versions, aliases, retirement dates, parameters and terms. Their own tooling is priced near cost, which pressures everyone selling evals, and they're unpredictable platform partners: Promptfoo joined OpenAI in March 2026 and OpenAI announced the shutdown of its own hosted Evals that June, while Anthropic's guidance names outside tools instead of selling a platform.
- Observability incumbents hold the budget, the security review and the agent already on every host. Datadog's free tier makes it the default for anyone already paying it, and Dynatrace and Cisco bought their way in.
- Independent platforms hold the eval workflow, review screens, dataset tools and neutrality across providers. Their leverage is weaker than it looks, because traces are becoming portable and eval files are small.
- Domain experts hold the definition of good, and they're scarce; whoever controls their time controls how fast the test set improves.
- Product teams hold the gate and the rollout, and users hold the real ground truth and, so far, the job of finding regressions.
- Labs as buyers hold the power in human data. Scale AI reportedly lost Google and OpenAI work after Meta took a 49% stake in 2025, and Meta reportedly paused work with Mercor after a breach there in March 2026.
Tailwinds and headwinds
These are the forces I'd watch over the next couple of years. Each helps some players and hurts others, and the companies that already hold the traffic, the cloud account or the model come out ahead on most of them.
Tailwinds:
| What's changing | Who gains | Who loses |
|---|---|---|
| Agents are moving into production, and in LangChain's late-2025 survey quality was the top barrier. | Datadog, Dynatrace, Splunk and the clouds; LangChain and Braintrust; sellers of judge tokens | Teams with no owner for quality |
| Providers retire models on clocks of 60 days to six months, so products re-test several times a year. | Eval platforms, since the suite is what gets re-run; clouds selling many models | Teams hard-coded to one model alias |
| Labs buy expert human data at scale; Mercor's reported run rate doubled in about four months to June 2026. | Mercor, Surge AI, Handshake, micro1; domain experts | Crowd platforms (Amazon closed Mechanical Turk on September 30, 2026); single-lab vendors |
| Public benchmarks are losing credibility, so buyers test models on their own data. | Product eval tools; independent evaluators such as Arena and Vals AI | Buying a model on its leaderboard rank |
| Judges got about ten times cheaper in a model generation, and coding agents now write eval cases. | Labs and clouds selling tokens; teams building their own suites | Platforms charging per score or per trace |
Headwinds:
| What's changing | Who gains | Who loses |
|---|---|---|
| Clouds bundle evaluation at near cost: Microsoft bills quality evaluations as model tokens "with no surcharge", and Google's went generally available on July 31, 2026 at standard rates. | AWS, Microsoft, Google; Databricks and Snowflake | Standalone eval platforms |
| Independent platforms get bought by whoever owns the traffic or the data: five in eight months of 2026. | Dynatrace, Cisco, ServiceNow, ClickHouse, OpenAI | Buyers of small tools, who face migrations; the remaining independents |
| Human data depends on a few lab buyers, and a security incident can stop work overnight. | Rival vendors; labs building in-house | Single-lab vendors and their contractors |
| Open source and free tiers cap what tools can charge. | Buyers; open-core vendors selling hosting | Tools priced per seat or per trace |
| Evals stay something teams do more than buy: 22.8% of LangChain's respondents with agents in production ran no evals. | Suites and incumbents folding evals into existing spend | Vendors that need a dedicated eval budget |
The one thing I'd watch is whether the clouds start bundling the human review screens and dataset tools, because that's the part where the independents still lead.