Costs and pricing

How the money moves

The product team pays almost everyone, and the biggest bill is its own people's time.

One buyer-data platform puts the median LangSmith contract at about $40,000 a year. By my rough arithmetic, bills for the same traffic vary about tenfold between vendors, because each counts a different unit (traces, spans, gigabytes, scores), so I'd compare quotes at my own trace size and retention, and sample production traffic before haggling over plans.

Here's what one eval run of 500 cases costs, scored by a model judge or by people, from list prices as of October 2026. A short chat case is about 1,500 tokens for the judge to read, an agent trace about 12,000.

Measuring agreement needs two independent reviewers, which doubles the human rows. A judge costs between a tenth and a thousandth of human review per case, so the usual design is code checks, then a model judge for volume, then people for a calibration set of 100-200 cases per failure mode. The platform's share of one run is effectively nothing.

The run is the cheap part. By my rough estimate, building a suite (reading about 100 traces, writing 20-50 cases and a rubric per failure mode, wiring it into CI) takes 40-120 engineer-hours, then a few hours per release. For one product running the suite twice a week, tokens come to hundreds or a few thousand dollars a year, and people to between a quarter of a person and two people.

Most of the market's money never touches a product team: human-expert data sold to labs runs to billions a year, while eval tooling is hundreds of millions by my rough estimate. Analyst sizings disagree by several times, and one firm's whole data-labelling forecast for 2030 is smaller than Mercor's reported run rate now, so I wouldn't quote any of them.

How companies of different sizes use it

Nobody publishes eval spend by company size, so the bands and figures below are my rough estimates from list prices, vendor case studies and LangChain's survey. Spend means cash on tools, tokens and outside review; staff time comes on top and is usually bigger.

A startup (under 50 people) runs on one lab's API, keeps a handful of test cases in its repo with an open-source runner such as Promptfoo or DeepEval, and uses a free tracing tier. A model judge scores, the founders read traces, and the engineer who owns the agent owns quality. Startup programs make year one nearly free: up to $10,000 in LangSmith credits for venture-backed startups, six to twelve months at Braintrust. My rough estimate is under $10,000 a year. A buyer asking for test evidence, an embarrassing regression or a retirement notice pushes it on.

A scale-up (about 50 to 500 people) has one or two AI features or agents with a regression set of 100-1,000 cases gating releases in CI, online judges on sampled traffic and regular error analysis. It buys a specialist platform such as Braintrust, LangSmith or Langfuse. The help-desk company is this size: an AI engineering lead owns the framework, a dedicated evals engineer often builds it, and the PM owns the definition of good and the gate. Notion, in a case study from its eval vendor, says its automated evals let it ship a new frontier model in under 24 hours. My rough estimate is $20,000-200,000 a year, plus one to four people's time. Security reviews, shared datasets across teams and customer audits push it on.

A mid-market company (about 500 to 2,000 people) is often not a software company at heart, and its AI features sit inside existing support, search or document workflows. It uses what it already pays for (Datadog, its cloud's evaluators, suites such as ServiceNow), often without systematic offline evals, and subject experts review answers in spreadsheets under a head of engineering or data. My rough estimate is $60,000-550,000 a year, with human review the most variable line. Another product line, a regulated use in banking or health, or a CFO asking for an AI cost line pushes it on.

An enterprise (about 2,000 to 10,000 people) runs a shared eval and tracing service from a central platform team, several model providers behind a gateway, and a model risk or AI governance function for the high-risk uses. It buys its observability vendor's AI module, self-hosts Langfuse or Phoenix where data control matters and runs red-team tools such as Promptfoo or Microsoft's PyRIT, while each business unit owns its test sets. My rough estimate is $250,000-2.6 million a year, mostly human review and expert panels, plus an evals and quality group of perhaps five to 20 people. Internal audit, the EU's high-risk timetable or an incident push it on.

A Fortune 500 company (revenue above about $7.5 billion) has many teams on many tools under a central AI policy. A few product lines have mature suites, the rest rely on evals built into the suites they buy, and external expert panels review clinical, legal or financial output. Langfuse says 63 of the Fortune 500 use it and Promptfoo claims more than a quarter, both self-reported and overlapping. A chief AI or data officer sets policy, business units own outcomes, and model risk and internal audit challenge them. My rough estimate is $1.7-18 million a year, with expert review the largest line. There's no next size up; the pressure comes from regulators and incidents.

What changes as a company grows (spend is my rough estimate):