How evals work, and when to trust the scores

Step 01 of 07

A case from a real failure

This starts from How agents fail. A customer (the user) writes in a support chat, and the agent answers: the app (the chat window and the code behind it) and the model, working in a loop with tools. The team that runs it reads traces, the step-by-step record of each conversation, and keeps a failure log whose rows end in “test added”. New here: the support lead, who decides what a right answer looks like; the model provider, the lab that makes the model and retires old versions on its own dates; the eval runner, a tool that replays test cases against the agent and records the scores; and the judge model, a model the runner asks to grade a reply. An eval is a repeatable test with a score you can compare from one version to the next. Each test case is a conversation the team can replay, with the tool responses recorded and played back (mocked), an expected outcome and a grader that decides pass or fail. Together they're the test set.

Make a guess to continue.