THEDE TECHNOLOGIES
Revised 19 Nov 2026

Trust the output, sheet 03 of 13

Measuring the org

Metrics for an org where agents author a growing share of the code, and an honest counter to a top-down mandate.

6Tools I checked
3Open gaps
Measures volume, misses the vetoesWhere the market is
11 Nov 2026Last checked
If you do one thing

Pair cost per merged PR with incidents per merged PR. Nobody sells a pushback metric yet, and your GitHub event stream already has the data.

The toolkit as a map: 4 layers, 13 sheets, 112 tools drawn as filled squares and 46 open gaps drawn as hatched squares INDEX TO THE THIRTEEN SHEETS 112 tools, 46 open gaps, revised 19 Nov 2026 LAYER 1 TRUST THE OUTPUT Layer total 3 sheets 22 tools on the map 11 open gaps 01 Observability and evals promptfoo 1 promptfoo Langfuse 2 Langfuse Arize Phoenix 3 Arize Phoenix Braintrust 4 Braintrust DeepEval 5 DeepEval OpenTelemetry GenAI conventions 6 OTel GenAI Demystifying evals for AI agents 7 Evals guide OpenAI deprecations 8 OpenAI dates Judge calibration 9 Judge kappa Open gap a a Open gap b b Open gap c c Open gap d d 02 The door before the merge GitHub branch protection and CODEOWNERS 1 Branch rules GitHub Copilot code review 2 Copilot review Claude Code Review 3 Claude Review Greptile 4 Greptile CodeRabbit 5 CodeRabbit Socket Firewall 6 Socket Firewall GitHub Agentic Workflows 7 Agentic Workflows Open gap a a Open gap b b Open gap c c Open gap d d 03 Measuring the org DORA, State of AI-assisted Software Development 2025 1 DORA 2025 METR, early-2025 AI and experienced developers 2 METR study DX AI Measurement Framework 3 DX GitClear code quality research 4 GitClear Faros 5 Faros Multitudes 6 Multitudes Open gap a a Open gap b b Open gap c c LAYER 2 CONTAIN THE AGENT 4 sheets, 35 tools, 13 gaps 04 MCP gateways and registries MCP specification, 2026-07-28 1 MCP spec Official MCP Registry 2 MCP Registry Docker MCP Gateway 3 Docker Gateway Obot 4 Obot Microsoft MCP Gateway 5 Microsoft gateway Mintive 6 Mintive Kong AI Gateway 7 Kong Cloudflare MCP server portals 8 Cloudflare portals Amazon Bedrock AgentCore Gateway 9 AgentCore Gateway Open gap a a Open gap b b Open gap c c Open gap d d Open gap e e 05 Identity and secrets Okta Cross App Access 1 Cross App Access Auth0 Token Vault 2 Token Vault Microsoft Entra Agent ID 3 Entra Agent ID Aembit 4 Aembit Teleport 5 Teleport SPIFFE 6 SPIFFE Open gap a a Open gap b b 06 Shadow AI and ghost jobs Nudge Security 1 Nudge Harmonic Security 2 Harmonic Reco 3 Reco Microsoft Purview for AI 4 Purview Healthchecks.io 5 Health checks Cronitor 6 Cronitor Sentry Crons 7 Sentry Crons Open gap a a Open gap b b Open gap c c 07 Sandboxing and containment Claude Code sandbox 1 Claude Code sandbox Codex sandboxing 2 Codex nono 3 nono Docker Sandboxes 4 Docker Sandboxes E2B 5 E2B Hosted sandboxes 6 Hosted sandboxes Apple container 7 Apple container Anthropic Sandbox Runtime 8 Sandbox Runtime Agent hooks 9 Agent hooks Claude Code managed settings 10 Managed settings How we built Claude Code auto mode 11 Auto mode The lethal trifecta 12 Lethal trifecta OWASP LLM06: Excessive Agency 13 OWASP LLM06 Open gap a a Open gap b b Open gap c c LAYER 3 PAY FOR IT Layer total 2 sheets 13 tools on the map 7 open gaps 08 Spend governance Claude plans and pricing 1 Claude plans Claude Code cost controls 2 Claude Code costs GitHub budgets and alerts 3 GitHub budgets LiteLLM 4 LiteLLM Cloudflare AI Gateway 5 Cloudflare AI Gateway Vantage for Anthropic 6 Vantage Open gap a a Open gap b b Open gap c c Open gap d d 09 Routing and model lifecycle Anthropic model deprecations 1 Claude dates OpenRouter Auto Router 2 OpenRouter Auto Microsoft Foundry model router 3 Foundry router Amazon Bedrock intelligent prompt routing 4 Bedrock routing Not Diamond 5 Not Diamond Requesty 6 Requesty RouteLLM 7 RouteLLM Open gap a a Open gap b b Open gap c c LAYER 4 FEED IT, STAFF IT, SELL IT 4 sheets, 42 tools, 15 gaps 10 Context layer and memory Glean 1 Glean Onyx 2 Onyx Credal 3 Credal Dust 4 Dust msgvault 5 msgvault AGENTS.md 6 AGENTS .md Mem0 7 Mem0 Open gap a a Open gap b b Open gap c c Open gap d d Open gap e e 11 Hiring and verification Greenhouse Real Talent 1 Greenhouse Real Talent Pindrop 2 Pindrop Karat 3 Karat CoderPad 4 CoderPad Open gap a a Open gap b b 12 Discovery and selling first User Interviews 1 User Interviews Respondent 2 Respondent Boardy 3 Boardy Happenstance 4 Happen stance Clay 5 Clay Listen Labs 6 Listen Labs Synthetic Users 7 Synthetic Users Synthetic Users: If, When, and How to Use AI-Generated Research 8 NN/g article Open gap a a Open gap b b Open gap c c 13 Agent harnesses Claude Code 1 Claude Code OpenAI Codex 2 Codex Gemini CLI 3 Gemini CLI GitHub Copilot agents 4 Copilot agents Cursor 5 Cursor OpenCode 6 OpenCode Goose 7 Goose Cline 8 Cline Pi, and its fork omp 9 Pi Claude Agent SDK 10 Claude Agent SDK OpenAI Agents SDK 11 OpenAI Agents SDK Google Agent Development Kit 12 Google ADK Microsoft Agent Framework 13 Agent Framework Strands Agents 14 Strands LangGraph 15 LangGraph Effective harnesses for long-running agents 16 Harness guide Paradev, formerly Superset 17 Paradev Orca 18 Orca Herdr 19 Herdr Paseo 20 Paseo Symphony 21 Symphony Stripe Minions 22 Stripe Minions StrongDM's software factory 23 StrongDM factory Open gap a a Open gap b b Open gap c c Open gap d d Open gap e e The toolkit as a map: 4 layers, 13 sheets, 112 tools drawn as filled squares and 46 open gaps drawn as hatched squares INDEX TO THE THIRTEEN SHEETS 112 tools, 46 open gaps, revised 19 Nov 2026 LAYER 1 TRUST THE OUTPUT Layer total 3 sheets 22 tools on the map 11 open gaps 01 Observability and evals promptfoo 1 promptfoo Langfuse 2 Langfuse Arize Phoenix 3 Arize Phoenix Braintrust 4 Braintrust DeepEval 5 DeepEval OpenTelemetry GenAI conventions 6 OTel GenAI Demystifying evals for AI agents 7 Evals guide OpenAI deprecations 8 OpenAI dates Judge calibration 9 Judge kappa Open gap a a Open gap b b Open gap c c Open gap d d 02 The door before the merge GitHub branch protection and CODEOWNERS 1 Branch rules GitHub Copilot code review 2 Copilot review Claude Code Review 3 Claude Review Greptile 4 Greptile CodeRabbit 5 CodeRabbit Socket Firewall 6 Socket Firewall GitHub Agentic Workflows 7 Agentic Workflows Open gap a a Open gap b b Open gap c c Open gap d d 03 Measuring the org DORA, State of AI-assisted Software Development 2025 1 DORA 2025 METR, early-2025 AI and experienced developers 2 METR study DX AI Measurement Framework 3 DX GitClear code quality research 4 GitClear Faros 5 Faros Multitudes 6 Multitudes Open gap a a Open gap b b Open gap c c LAYER 2 CONTAIN THE AGENT 4 sheets, 35 tools, 13 gaps 04 MCP gateways and registries MCP specification, 2026-07-28 1 MCP spec Official MCP Registry 2 MCP Registry Docker MCP Gateway 3 Docker Gateway Obot 4 Obot Microsoft MCP Gateway 5 Microsoft gateway Mintive 6 Mintive Kong AI Gateway 7 Kong Cloudflare MCP server portals 8 Cloudflare portals Amazon Bedrock AgentCore Gateway 9 AgentCore Gateway Open gap a a Open gap b b Open gap c c Open gap d d Open gap e e 05 Identity and secrets Okta Cross App Access 1 Cross App Access Auth0 Token Vault 2 Token Vault Microsoft Entra Agent ID 3 Entra Agent ID Aembit 4 Aembit Teleport 5 Teleport SPIFFE 6 SPIFFE Open gap a a Open gap b b 06 Shadow AI and ghost jobs Nudge Security 1 Nudge Harmonic Security 2 Harmonic Reco 3 Reco Microsoft Purview for AI 4 Purview Healthchecks.io 5 Health checks Cronitor 6 Cronitor Sentry Crons 7 Sentry Crons Open gap a a Open gap b b Open gap c c 07 Sandboxing and containment Claude Code sandbox 1 Claude Code sandbox Codex sandboxing 2 Codex nono 3 nono Docker Sandboxes 4 Docker Sandboxes E2B 5 E2B Hosted sandboxes 6 Hosted sandboxes Apple container 7 Apple container Anthropic Sandbox Runtime 8 Sandbox Runtime Agent hooks 9 Agent hooks Claude Code managed settings 10 Managed settings How we built Claude Code auto mode 11 Auto mode The lethal trifecta 12 Lethal trifecta OWASP LLM06: Excessive Agency 13 OWASP LLM06 Open gap a a Open gap b b Open gap c c LAYER 3 PAY FOR IT Layer total 2 sheets 13 tools on the map 7 open gaps 08 Spend governance Claude plans and pricing 1 Claude plans Claude Code cost controls 2 Claude Code costs GitHub budgets and alerts 3 GitHub budgets LiteLLM 4 LiteLLM Cloudflare AI Gateway 5 Cloudflare AI Gateway Vantage for Anthropic 6 Vantage Open gap a a Open gap b b Open gap c c Open gap d d 09 Routing and model lifecycle Anthropic model deprecations 1 Claude dates OpenRouter Auto Router 2 OpenRouter Auto Microsoft Foundry model router 3 Foundry router Amazon Bedrock intelligent prompt routing 4 Bedrock routing Not Diamond 5 Not Diamond Requesty 6 Requesty RouteLLM 7 RouteLLM Open gap a a Open gap b b Open gap c c LAYER 4 FEED IT, STAFF IT, SELL IT 4 sheets, 42 tools, 15 gaps 10 Context layer and memory Glean 1 Glean Onyx 2 Onyx Credal 3 Credal Dust 4 Dust msgvault 5 msgvault AGENTS.md 6 AGENTS .md Mem0 7 Mem0 Open gap a a Open gap b b Open gap c c Open gap d d Open gap e e 11 Hiring and verification Greenhouse Real Talent 1 Greenhouse Real Talent Pindrop 2 Pindrop Karat 3 Karat CoderPad 4 CoderPad Open gap a a Open gap b b 12 Discovery and selling first User Interviews 1 User Interviews Respondent 2 Respondent Boardy 3 Boardy Happenstance 4 Happen stance Clay 5 Clay Listen Labs 6 Listen Labs Synthetic Users 7 Synthetic Users Synthetic Users: If, When, and How to Use AI-Generated Research 8 NN/g article Open gap a a Open gap b b Open gap c c 13 Agent harnesses Claude Code 1 Claude Code OpenAI Codex 2 Codex Gemini CLI 3 Gemini CLI GitHub Copilot agents 4 Copilot agents Cursor 5 Cursor OpenCode 6 OpenCode Goose 7 Goose Cline 8 Cline Pi, and its fork omp 9 Pi Claude Agent SDK 10 Claude Agent SDK OpenAI Agents SDK 11 OpenAI Agents SDK Google Agent Development Kit 12 Google ADK Microsoft Agent Framework 13 Agent Framework Strands Agents 14 Strands LangGraph 15 LangGraph Effective harnesses for long-running agents 16 Harness guide Paradev, formerly Superset 17 Paradev Orca 18 Orca Herdr 19 Herdr Paseo 20 Paseo Symphony 21 Symphony Stripe Minions 22 Stripe Minions StrongDM's software factory 23 StrongDM factory Open gap a a Open gap b b Open gap c c Open gap d d Open gap e e
Sheet 03: numbers are tools, letters are gapsOne sheet to the view5 tools

← → move to the next sheet over, Esc zooms out.

The problem, in plain words

Once agents write a real share of the code, the old dashboards keep counting and stop meaning much. More pull requests, more lines, more tokens. None of it says whether the work got better, or how often a person caught a bad change and stopped it.

It's like judging a kitchen by how many plates left the pass. Useful, until you notice nobody's counting how many came back.

The honest instruments come in pairs: what it cost and what it broke, what shipped and what got rejected. Most of the data already sits in your Git history. Nobody packages the uncomfortable half.

Words you'll hear

Throughput
How much work gets merged and shipped in a period.
Change failure rate
The share of changes that cause an incident, a rollback or a hotfix.
Cost per merged PR
Tokens and seats spent, divided by the pull requests that actually landed.
Pushback
Evidence that someone said no: agent PRs closed unmerged with a reason, suggestions rejected, human edits before merge.
Acceptance rate
How often people keep what the assistant suggested. Easy to measure and easy to game.

What to understand, and what to decide

Understand

Decide

Who's working on it

Every link opened on 11 Nov 2026. The numbers match the squares on the map.

  1. DORA, State of AI-assisted Software Development 2025

    Reference, DORA

    The long-running research program on software delivery performance, with its 2025 report on what AI-assisted development changes.

    The catch. Survey research. It tells you what correlates across thousands of teams, not what happened in yours.

  2. METR, early-2025 AI and experienced developers

    Reference, METR

    A randomized controlled trial: experienced open-source developers took 19% longer to finish issues when they could use AI tools.

    The catch. A snapshot of early-2025 tools. METR's 2026 follow-up found returning developers about 18% faster, with a confidence interval from 38% faster to 9% slower, and METR calls the new data an unreliable signal. Read it for the gap between perception and reality.

  3. DX AI Measurement Framework

    Framework, DX

    Research-based metrics for measuring the impact of AI code assistants and agents on an engineering org.

    The catch. A framework from a vendor that sells the measuring. Use the questions even if you don't buy the platform.

  4. GitClear code quality research

    Reference, GitClear

    A multi-year look at changed lines: refactoring down, copy-paste up, and short-term churn rising as AI assistance spread.

    The catch. GitClear sells the analysis. The trend is useful, and your own repo is the better evidence.

  5. Faros

    Product

    Traces coding-agent usage and token spend to shipped features and incidents, so cost and outcome sit in one place.

    The catch. The figures in its reports are Faros's own. Check them against your incident log.

  6. Multitudes

    Product

    Team-level engineering analytics that flag early signs of AI slop, such as falling code quality or human review missing issues in AI changes.

    The catch. Leading indicators are prompts for a conversation, not grades for a person.

Still nobody sells

  1. A pushback metric: agent PRs closed with a reason, human edits before merge, reviewer suggestions declined, and reverts traced to agent changes.
  2. A measure of spec quality: whether the work an agent was handed was clear enough to finish.
  3. Executive instruments that show cost and breakage on one page.