Deep dive: model-graded scoring

Model-graded scoring: what's real

My view as of October 2026: model judges are good enough to be the scaling layer of an eval program and too unreliable to be its source of truth. On general questions they agree with people about as well as people agree with each other, noticeably less with experts in specialist fields, and their biases are known and can be designed around. The weakest evidence is on what happens when the judge model itself changes, which every team will face.

How far judges agree with people

• General chat: the study that made judges popular (Zheng and colleagues, 2023) found GPT-4 agreed with human preferences about 85% of the time on non-tied votes, slightly above the 81% at which humans agreed with each other.
• Expert domains: in a 2024 study, dietitians and mental-health experts agreed with model judges only 68% and 64% of the time. A 2026 review of 134 healthcare studies called alignment "moderate to strong" and said it varies a lot by task. Experts disagree among themselves too, so the fair benchmark is expert-to-expert agreement.
• Hard correctness: on JudgeBench, pairs of answers where one is subtly wrong, strong judges of the GPT-4o era did only slightly better than chance.
• Measuring it properly: a large 2026 preprint (21 judges, about 541,000 judgments) found simple percent agreement overstates quality: a chance-corrected measure came out 33-41 points lower on one standard benchmark, and a judge's ranking moved up to 14 places between benchmarks. It's one paper and hasn't been replicated.
• Narrow, tuned judges do better: Husain reports a judge for one company's query assistant reaching over 90% agreement with its expert after three rounds of prompt changes, a single consultant's case that fits what labs advise.

I'd measure a judge by how often it catches real failures and how often it passes real successes, separately, because overall agreement looks fine on a test set where 95% of cases pass.

Known biases

• Position: judges favour an answer for where it sits. A study of 15 judges on 22 tasks found the effect varies by judge and task and grows when the answers are close in quality; in the 2023 study GPT-4 kept its verdict after a swap only 65% of the time with the default prompt. Run both orders and count a flip as a tie.
• Length: judges like longer answers. AlpacaEval, a leaderboard scored by a model judge, built a length-controlled version that raised its correlation with human rankings on Chatbot Arena from 0.94 to 0.98.
• Self-preference: models rate their own outputs higher, more so the better they recognise their own writing; GPT-4 recognised its own output 73.5% of the time in one 2024 study. Anthropic calls grading with a different model "generally best practice", which matters most when the product and the judge move to the same new family at once.
• The rest: a 2024 audit, CALM, measured 12 kinds of judge bias and found significant ones persist even in strong models.

The labs' advice converges: ask for reasoning before the score, prefer pass or fail (or a pairwise choice) over a 1-5 scale, give the judge an "unknown" option, use one judge per dimension, control for length, and validate against human labels before scaling up.

When the judge changes

Hosted models change behind a fixed name: in a 2023 study the same GPT-4 service went from 84% to 51% on identifying prime numbers between March and June. Critics said part of that was a change in answer format; either way the name stayed and the behaviour moved, and a judge called through an alias is exposed in the same way.

Direct evidence on judges is thin. A single-author preprint from June 2026 showed in simulation that a silent judge update looks exactly like a change in your product unless you re-score a fixed set of human-labelled anchor cases; that check caught the update in all 60 simulated runs, while a common statistical alarm went off on 75% of streams where nothing had changed. I found no study measuring real score shifts when a GPT, Claude or Gemini judge version changes, and no lab document telling teams to pin their judge.

What I'd do anyway: pin the judge's dated version, keep 100-200 human-labelled anchor cases per failure mode, re-score them on any judge or judge-prompt change, and mark old and new scores as not comparable if agreement moves.

When you still need people

• Defining good: criteria change as people grade (the Shankar study again), so a person has to look at real outputs before any rubric exists. Husain and Shankar's method starts here: read about 100 traces, note what's wrong with each, group the notes into failure modes and count them. In one of Husain's client cases, this showed an assistant failing on date handling 66% of the time; it's one consultant's story, though the method is widely copied.
• Calibration and spot checks: every judge needs a human-labelled set to be checked against after each change, and someone should read a sample of its verdicts, especially disagreements.
• Expert domains and subjective quality: law, medicine, finance and anything where taste matters, since that's where agreement drops.
• Signals that reward the wrong thing: thumbs-up rates can improve while behaviour worsens, as with GPT-4o, whose expert testers had said it "felt" slightly off before launch.

Human review is still more common than model judges. In LangChain's own survey of 1,340 developers, a vendor's sample, 59.8% used human review and 53.3% model judges; 89% had some observability, and only 52.4% ran offline evals. Simulated users, where a model plays the customer, need the same caution: two 2026 studies found simulators more cooperative than real people, inflating agent success, with agent scores moving by up to 9 points depending on which model played the user. They help with regression and coverage and say little about how real people will fare.

Grader types side by side

Questions to ask about any judge score

1. Which judge version and prompt produced it, and when was it last checked against human labels?
2. How often does it catch real failures, and how often does it fail good answers?
3. Is the judge from the same model family as the product it grades?
4. If the judge changed tomorrow, which human-labelled cases would tell us?