THEDE TECHNOLOGIES
Revised 19 Nov 2026

Trust the output, sheet 01 of 13

Observability and evals

Know what an agent did, score it against a golden set, and find out what breaks when the model underneath it is retired.

9Tools I checked
4Open gaps
Crowded and consolidatingWhere the market is
19 Nov 2026Last checked
If you do one thing

Put the golden set in version control and re-run it on every model swap. promptfoo in CI does this today, for free.

The toolkit as a map: 4 layers, 13 sheets, 112 tools drawn as filled squares and 46 open gaps drawn as hatched squares INDEX TO THE THIRTEEN SHEETS 112 tools, 46 open gaps, revised 19 Nov 2026 LAYER 1 TRUST THE OUTPUT Layer total 3 sheets 22 tools on the map 11 open gaps 01 Observability and evals promptfoo 1 promptfoo Langfuse 2 Langfuse Arize Phoenix 3 Arize Phoenix Braintrust 4 Braintrust DeepEval 5 DeepEval OpenTelemetry GenAI conventions 6 OTel GenAI Demystifying evals for AI agents 7 Evals guide OpenAI deprecations 8 OpenAI dates Judge calibration 9 Judge kappa Open gap a a Open gap b b Open gap c c Open gap d d 02 The door before the merge GitHub branch protection and CODEOWNERS 1 Branch rules GitHub Copilot code review 2 Copilot review Claude Code Review 3 Claude Review Greptile 4 Greptile CodeRabbit 5 CodeRabbit Socket Firewall 6 Socket Firewall GitHub Agentic Workflows 7 Agentic Workflows Open gap a a Open gap b b Open gap c c Open gap d d 03 Measuring the org DORA, State of AI-assisted Software Development 2025 1 DORA 2025 METR, early-2025 AI and experienced developers 2 METR study DX AI Measurement Framework 3 DX GitClear code quality research 4 GitClear Faros 5 Faros Multitudes 6 Multitudes Open gap a a Open gap b b Open gap c c LAYER 2 CONTAIN THE AGENT 4 sheets, 35 tools, 13 gaps 04 MCP gateways and registries MCP specification, 2026-07-28 1 MCP spec Official MCP Registry 2 MCP Registry Docker MCP Gateway 3 Docker Gateway Obot 4 Obot Microsoft MCP Gateway 5 Microsoft gateway Mintive 6 Mintive Kong AI Gateway 7 Kong Cloudflare MCP server portals 8 Cloudflare portals Amazon Bedrock AgentCore Gateway 9 AgentCore Gateway Open gap a a Open gap b b Open gap c c Open gap d d Open gap e e 05 Identity and secrets Okta Cross App Access 1 Cross App Access Auth0 Token Vault 2 Token Vault Microsoft Entra Agent ID 3 Entra Agent ID Aembit 4 Aembit Teleport 5 Teleport SPIFFE 6 SPIFFE Open gap a a Open gap b b 06 Shadow AI and ghost jobs Nudge Security 1 Nudge Harmonic Security 2 Harmonic Reco 3 Reco Microsoft Purview for AI 4 Purview Healthchecks.io 5 Health checks Cronitor 6 Cronitor Sentry Crons 7 Sentry Crons Open gap a a Open gap b b Open gap c c 07 Sandboxing and containment Claude Code sandbox 1 Claude Code sandbox Codex sandboxing 2 Codex nono 3 nono Docker Sandboxes 4 Docker Sandboxes E2B 5 E2B Hosted sandboxes 6 Hosted sandboxes Apple container 7 Apple container Anthropic Sandbox Runtime 8 Sandbox Runtime Agent hooks 9 Agent hooks Claude Code managed settings 10 Managed settings How we built Claude Code auto mode 11 Auto mode The lethal trifecta 12 Lethal trifecta OWASP LLM06: Excessive Agency 13 OWASP LLM06 Open gap a a Open gap b b Open gap c c LAYER 3 PAY FOR IT Layer total 2 sheets 13 tools on the map 7 open gaps 08 Spend governance Claude plans and pricing 1 Claude plans Claude Code cost controls 2 Claude Code costs GitHub budgets and alerts 3 GitHub budgets LiteLLM 4 LiteLLM Cloudflare AI Gateway 5 Cloudflare AI Gateway Vantage for Anthropic 6 Vantage Open gap a a Open gap b b Open gap c c Open gap d d 09 Routing and model lifecycle Anthropic model deprecations 1 Claude dates OpenRouter Auto Router 2 OpenRouter Auto Microsoft Foundry model router 3 Foundry router Amazon Bedrock intelligent prompt routing 4 Bedrock routing Not Diamond 5 Not Diamond Requesty 6 Requesty RouteLLM 7 RouteLLM Open gap a a Open gap b b Open gap c c LAYER 4 FEED IT, STAFF IT, SELL IT 4 sheets, 42 tools, 15 gaps 10 Context layer and memory Glean 1 Glean Onyx 2 Onyx Credal 3 Credal Dust 4 Dust msgvault 5 msgvault AGENTS.md 6 AGENTS .md Mem0 7 Mem0 Open gap a a Open gap b b Open gap c c Open gap d d Open gap e e 11 Hiring and verification Greenhouse Real Talent 1 Greenhouse Real Talent Pindrop 2 Pindrop Karat 3 Karat CoderPad 4 CoderPad Open gap a a Open gap b b 12 Discovery and selling first User Interviews 1 User Interviews Respondent 2 Respondent Boardy 3 Boardy Happenstance 4 Happen stance Clay 5 Clay Listen Labs 6 Listen Labs Synthetic Users 7 Synthetic Users Synthetic Users: If, When, and How to Use AI-Generated Research 8 NN/g article Open gap a a Open gap b b Open gap c c 13 Agent harnesses Claude Code 1 Claude Code OpenAI Codex 2 Codex Gemini CLI 3 Gemini CLI GitHub Copilot agents 4 Copilot agents Cursor 5 Cursor OpenCode 6 OpenCode Goose 7 Goose Cline 8 Cline Pi, and its fork omp 9 Pi Claude Agent SDK 10 Claude Agent SDK OpenAI Agents SDK 11 OpenAI Agents SDK Google Agent Development Kit 12 Google ADK Microsoft Agent Framework 13 Agent Framework Strands Agents 14 Strands LangGraph 15 LangGraph Effective harnesses for long-running agents 16 Harness guide Paradev, formerly Superset 17 Paradev Orca 18 Orca Herdr 19 Herdr Paseo 20 Paseo Symphony 21 Symphony Stripe Minions 22 Stripe Minions StrongDM's software factory 23 StrongDM factory Open gap a a Open gap b b Open gap c c Open gap d d Open gap e e The toolkit as a map: 4 layers, 13 sheets, 112 tools drawn as filled squares and 46 open gaps drawn as hatched squares INDEX TO THE THIRTEEN SHEETS 112 tools, 46 open gaps, revised 19 Nov 2026 LAYER 1 TRUST THE OUTPUT Layer total 3 sheets 22 tools on the map 11 open gaps 01 Observability and evals promptfoo 1 promptfoo Langfuse 2 Langfuse Arize Phoenix 3 Arize Phoenix Braintrust 4 Braintrust DeepEval 5 DeepEval OpenTelemetry GenAI conventions 6 OTel GenAI Demystifying evals for AI agents 7 Evals guide OpenAI deprecations 8 OpenAI dates Judge calibration 9 Judge kappa Open gap a a Open gap b b Open gap c c Open gap d d 02 The door before the merge GitHub branch protection and CODEOWNERS 1 Branch rules GitHub Copilot code review 2 Copilot review Claude Code Review 3 Claude Review Greptile 4 Greptile CodeRabbit 5 CodeRabbit Socket Firewall 6 Socket Firewall GitHub Agentic Workflows 7 Agentic Workflows Open gap a a Open gap b b Open gap c c Open gap d d 03 Measuring the org DORA, State of AI-assisted Software Development 2025 1 DORA 2025 METR, early-2025 AI and experienced developers 2 METR study DX AI Measurement Framework 3 DX GitClear code quality research 4 GitClear Faros 5 Faros Multitudes 6 Multitudes Open gap a a Open gap b b Open gap c c LAYER 2 CONTAIN THE AGENT 4 sheets, 35 tools, 13 gaps 04 MCP gateways and registries MCP specification, 2026-07-28 1 MCP spec Official MCP Registry 2 MCP Registry Docker MCP Gateway 3 Docker Gateway Obot 4 Obot Microsoft MCP Gateway 5 Microsoft gateway Mintive 6 Mintive Kong AI Gateway 7 Kong Cloudflare MCP server portals 8 Cloudflare portals Amazon Bedrock AgentCore Gateway 9 AgentCore Gateway Open gap a a Open gap b b Open gap c c Open gap d d Open gap e e 05 Identity and secrets Okta Cross App Access 1 Cross App Access Auth0 Token Vault 2 Token Vault Microsoft Entra Agent ID 3 Entra Agent ID Aembit 4 Aembit Teleport 5 Teleport SPIFFE 6 SPIFFE Open gap a a Open gap b b 06 Shadow AI and ghost jobs Nudge Security 1 Nudge Harmonic Security 2 Harmonic Reco 3 Reco Microsoft Purview for AI 4 Purview Healthchecks.io 5 Health checks Cronitor 6 Cronitor Sentry Crons 7 Sentry Crons Open gap a a Open gap b b Open gap c c 07 Sandboxing and containment Claude Code sandbox 1 Claude Code sandbox Codex sandboxing 2 Codex nono 3 nono Docker Sandboxes 4 Docker Sandboxes E2B 5 E2B Hosted sandboxes 6 Hosted sandboxes Apple container 7 Apple container Anthropic Sandbox Runtime 8 Sandbox Runtime Agent hooks 9 Agent hooks Claude Code managed settings 10 Managed settings How we built Claude Code auto mode 11 Auto mode The lethal trifecta 12 Lethal trifecta OWASP LLM06: Excessive Agency 13 OWASP LLM06 Open gap a a Open gap b b Open gap c c LAYER 3 PAY FOR IT Layer total 2 sheets 13 tools on the map 7 open gaps 08 Spend governance Claude plans and pricing 1 Claude plans Claude Code cost controls 2 Claude Code costs GitHub budgets and alerts 3 GitHub budgets LiteLLM 4 LiteLLM Cloudflare AI Gateway 5 Cloudflare AI Gateway Vantage for Anthropic 6 Vantage Open gap a a Open gap b b Open gap c c Open gap d d 09 Routing and model lifecycle Anthropic model deprecations 1 Claude dates OpenRouter Auto Router 2 OpenRouter Auto Microsoft Foundry model router 3 Foundry router Amazon Bedrock intelligent prompt routing 4 Bedrock routing Not Diamond 5 Not Diamond Requesty 6 Requesty RouteLLM 7 RouteLLM Open gap a a Open gap b b Open gap c c LAYER 4 FEED IT, STAFF IT, SELL IT 4 sheets, 42 tools, 15 gaps 10 Context layer and memory Glean 1 Glean Onyx 2 Onyx Credal 3 Credal Dust 4 Dust msgvault 5 msgvault AGENTS.md 6 AGENTS .md Mem0 7 Mem0 Open gap a a Open gap b b Open gap c c Open gap d d Open gap e e 11 Hiring and verification Greenhouse Real Talent 1 Greenhouse Real Talent Pindrop 2 Pindrop Karat 3 Karat CoderPad 4 CoderPad Open gap a a Open gap b b 12 Discovery and selling first User Interviews 1 User Interviews Respondent 2 Respondent Boardy 3 Boardy Happenstance 4 Happen stance Clay 5 Clay Listen Labs 6 Listen Labs Synthetic Users 7 Synthetic Users Synthetic Users: If, When, and How to Use AI-Generated Research 8 NN/g article Open gap a a Open gap b b Open gap c c 13 Agent harnesses Claude Code 1 Claude Code OpenAI Codex 2 Codex Gemini CLI 3 Gemini CLI GitHub Copilot agents 4 Copilot agents Cursor 5 Cursor OpenCode 6 OpenCode Goose 7 Goose Cline 8 Cline Pi, and its fork omp 9 Pi Claude Agent SDK 10 Claude Agent SDK OpenAI Agents SDK 11 OpenAI Agents SDK Google Agent Development Kit 12 Google ADK Microsoft Agent Framework 13 Agent Framework Strands Agents 14 Strands LangGraph 15 LangGraph Effective harnesses for long-running agents 16 Harness guide Paradev, formerly Superset 17 Paradev Orca 18 Orca Herdr 19 Herdr Paseo 20 Paseo Symphony 21 Symphony Stripe Minions 22 Stripe Minions StrongDM's software factory 23 StrongDM factory Open gap a a Open gap b b Open gap c c Open gap d d Open gap e e
Sheet 01: numbers are tools, letters are gapsOne sheet to the view5 tools

← → move to the next sheet over, Esc zooms out.

The problem, in plain words

When an agent does a job for you, two questions matter. What did it actually do, step by step? And was it any good? Observability answers the first. Evals answer the second.

Think of a restaurant kitchen. Observability is the camera over the line: who cooked what, in what order, with which ingredients. Evals are the tasting spoon: the same dishes, tasted the same way, every time something changes.

Something always changes, because vendors retire models on a published schedule. That's your flour supplier switching mills. The bread still looks like bread. You only find out it's different if somebody tastes it the same way every time, before the customers do.

Words you'll hear

Trace
The step-by-step record of one agent run: every model call, tool call and result, in order.
Golden set
A fixed collection of real tasks with known good answers, re-run after every change.
Eval
A repeatable test that scores an output, or a whole run, against a standard.
LLM judge
A model that grades another model's work against a rubric. Fast and cheap, and it needs checking itself.
Calibration
Measuring how often the judge agrees with careful humans, and fixing it when it drifts.
pass^k
The share of tasks an agent gets right on every one of k tries. The honest number for anything you'll run a thousand times.
Shutdown date
The day a vendor turns a model off. Any prompt pinned to it breaks or gets rerouted.

What to understand, and what to decide

Understand

Decide

Who's working on it

Every link opened on 19 Nov 2026. The numbers match the squares on the map.

  1. promptfoo

    Open source, free

    A YAML matrix of prompts by models, with assertions, run from the command line or in CI. The cheapest way to swap a model and diff the golden set.

    The catch. Now part of OpenAI and still MIT licensed. A referee owned by one of the teams is worth noticing.

  2. Langfuse

    Open source

    Tracing, prompt management, LLM judges, datasets and annotation queues, built on OpenTelemetry. The self-host default.

    The catch. Part of ClickHouse since January 2026. The cloud and the open-source project both continue.

  3. Arize Phoenix

    Source-available

    Agent tracing and evals you can run yourself, from the team behind Arize AX.

    The catch. Licensed under the Elastic License 2.0, which is not an open-source license. Read it before you build on it.

  4. Braintrust

    Product, free tier

    Tracing, datasets, experiments and scorers in one hosted product, with a free starter tier.

    The catch. The autoevals scorer library is open source. The platform is not, and the BAA is Enterprise-only.

  5. DeepEval

    Open source

    Pytest-style eval gates for LLM apps, Apache-2.0, with Confident AI as the hosted layer on top.

    The catch. The rubric is still yours to write. The framework only makes it run on every commit.

  6. OpenTelemetry GenAI conventions

    Standard

    The vendor-neutral names for model, agent, tool and MCP spans. The portability layer under every tracing vendor here.

    The catch. It now lives in its own repository and is still changing. Check a convention's status before you build on it.

  7. Demystifying evals for AI agents

    Reference, Anthropic

    Code graders first, model graders where you need them, people to calibrate both, and an "Unknown" exit for the judge.

    The catch. A model vendor on grading models. Still the clearest short statement of the method.

  8. OpenAI deprecations

    Reference, OpenAI

    Shutdown dates and replacements. GPT-4o, GPT-4o mini, GPT-4.1 and GPT-4.1 mini leave the API on January 14, 2027.

    The catch. The models endpoint returns a shutdown_date field. Nothing polls it for you.

  9. Judge calibration

    Practice

    Measure how often two people agree first. Then score the judge against the people with a chance-corrected number such as Cohen's kappa.

    The catch. With a lopsided pass rate, 90% raw agreement can be a kappa near zero. Recalibrate when the judge model, the prompt or the task changes.

Still nobody sells

  1. Deprecation-driven regression: watch the shutdown dates, find the prompts pinned to the dying model, run the golden set on the replacement, and open a pull request with the scored difference. Today it's promptfoo plus a cron job.
  2. Judge calibration as an enforced artifact, with a budget and an alert when the judge drifts away from the people.
  3. A clean handoff from a rubric a product manager owns to a CI gate an engineer owns.
  4. Holdout tests the agent that wrote the code can't read or rewrite. A test that lives beside the code can be rewritten to pass, which is why StrongDM often keeps its scenarios outside the codebase.