Trust the output, sheet 01 of 13
Know what an agent did, score it against a golden set, and find out what breaks when the model underneath it is retired.
Put the golden set in version control and re-run it on every model swap. promptfoo in CI does this today, for free.
← → move to the next sheet over, Esc zooms out.
When an agent does a job for you, two questions matter. What did it actually do, step by step? And was it any good? Observability answers the first. Evals answer the second.
Think of a restaurant kitchen. Observability is the camera over the line: who cooked what, in what order, with which ingredients. Evals are the tasting spoon: the same dishes, tasted the same way, every time something changes.
Something always changes, because vendors retire models on a published schedule. That's your flour supplier switching mills. The bread still looks like bread. You only find out it's different if somebody tastes it the same way every time, before the customers do.
Every link opened on 19 Nov 2026. The numbers match the squares on the map.
Open source, free
A YAML matrix of prompts by models, with assertions, run from the command line or in CI. The cheapest way to swap a model and diff the golden set.
The catch. Now part of OpenAI and still MIT licensed. A referee owned by one of the teams is worth noticing.
Open source
Tracing, prompt management, LLM judges, datasets and annotation queues, built on OpenTelemetry. The self-host default.
The catch. Part of ClickHouse since January 2026. The cloud and the open-source project both continue.
Source-available
Agent tracing and evals you can run yourself, from the team behind Arize AX.
The catch. Licensed under the Elastic License 2.0, which is not an open-source license. Read it before you build on it.
Product, free tier
Tracing, datasets, experiments and scorers in one hosted product, with a free starter tier.
The catch. The autoevals scorer library is open source. The platform is not, and the BAA is Enterprise-only.
Open source
Pytest-style eval gates for LLM apps, Apache-2.0, with Confident AI as the hosted layer on top.
The catch. The rubric is still yours to write. The framework only makes it run on every commit.
Standard
The vendor-neutral names for model, agent, tool and MCP spans. The portability layer under every tracing vendor here.
The catch. It now lives in its own repository and is still changing. Check a convention's status before you build on it.
Reference, Anthropic
Code graders first, model graders where you need them, people to calibrate both, and an "Unknown" exit for the judge.
The catch. A model vendor on grading models. Still the clearest short statement of the method.
Reference, OpenAI
Shutdown dates and replacements. GPT-4o, GPT-4o mini, GPT-4.1 and GPT-4.1 mini leave the API on January 14, 2027.
The catch. The models endpoint returns a shutdown_date field. Nothing polls it for you.
Practice
Measure how often two people agree first. Then score the judge against the people with a chance-corrected number such as Cohen's kappa.
The catch. With a lopsided pass rate, 90% raw agreement can be a kappa near zero. Recalibrate when the judge model, the prompt or the task changes.