Source files: sources/21-benchmarks/llm-evaluation-compendium-2026-raw.md · sources/21-benchmarks/llm-evaluation-compendium-enterprise-evals-2026-ledger.md · sources/21-benchmarks/enterprise-custom-eval-tooling-research-2026-raw.md · sources/21-benchmarks/llmevaluation-cited-sources-2026.md · sources/21-benchmarks/llmevaluation-cited-sources-2026.jsonl · research/21-benchmarks/raw/llmevaluation-deck-2024.chandra.txt
See also: Benchmark Foundations · Enterprise Eval for RAG and Fine-Tuned SLMs · Agent Observability Format Standard · Enterprise Desktop Agent Knowledge Graphs
Source status: This report ingests Andrei Lopatenko’s
LLMEvaluationcompendium as a discovery index and taxonomy source. The compendium itself is a curated map, not a measurement result. Claims about tools, benchmarks, and papers should be verified against primary sources before being cited as current facts. The useful contribution here is the operating model: how the compendium’s 796 resource bullets, 860 parsed citation records, and Chandra-extracted evaluation deck map into an enterprise evaluation program.
Executive Summary
Enterprise AI evaluation has split into three different jobs that should not share one score.
Model evals ask whether a model has a general capability. They use public benchmarks such as MMLU, SWE-bench, LiveCodeBench, MTEB, BEIR, and medical/legal/finance domain sets. They are useful for shortlisting, but weak for procurement because the tasks are public, often saturated, and rarely match internal work.
System evals ask whether an application works when retrieval, tools, permissions, prompts, memory, and human review are attached. RAGAS, DeepEval, Promptfoo, TruLens, Phoenix, Langfuse, MLflow Evaluate, OpenEvals, and Braintrust-style workflows live here. This is the right layer for enterprise knowledge bases and agentic workflows.
Work evals ask whether a real persona can complete a real job safely and economically. This is the layer that matters for Claude Code, Codex, Microsoft Copilot, Cursor, customer-support agents, legal research agents, finance copilots, and executive briefing agents. It needs private golden tasks, trace review, cost per successful outcome, latency, source quality, escalation behavior, and auditability.
The core recommendation: build an internal evaluation stack around golden work packets. Each packet should contain a persona, task, available sources, permission boundary, expected output contract, grading rubric, deterministic checks where possible, judge rubric where necessary, and trace/audit requirements. Public benchmarks are calibration; golden work packets are deployment evidence.
The second recommendation: classify failures by eval domain, not only by output defect. A retrieval miss, an unsafe code edit, a stale policy answer, a bad tool call, and a judge disagreement are different operational failures. They need different owners, harness metadata, remediation paths, and release gates.
What The LLMEvaluation Compendium Adds
The compendium is valuable because it shows that evaluation has become an ecosystem, not a leaderboard. The raw mirror has 861 URLs and 796 resource bullets across model benchmarks, RAG evaluation, LLM-as-judge, HITL, coding, agents, document understanding, search/recommendation, task utility, and verticals.
The companion complete-citation inventory parses 860 citation records, 740 unique URLs, and 115 domains. Resource mix:
| Resource type | Count |
|---|---|
| paper_preprint | 564 |
| web_resource | 95 |
| paper_published_or_conference | 77 |
| code_repo | 42 |
| model_dataset_or_leaderboard | 30 |
| blog_or_vendor_article | 29 |
| leaderboard_or_arena | 20 |
| talk_or_video | 3 |
That inventory is the ingestion queue. It does not mean every linked page has already been independently summarized; it means every cited resource is now addressable by section, line, title/context, URL, domain, and type for deeper verification.
The Chandra-extracted deck adds the operating principle that enterprise evaluation is lifecycle-specific. Pretraining, fine-tuning, retrieval-augmented systems, and deployed products need different eval sets. The deck also reinforces that no metric is final or comprehensive; RAG metrics such as RAGAS/ARES need task-specific customization; and coding/copilot evaluation should cover documentation generation, bug fixing, code generation, test generation, and workspace understanding rather than only final answer quality.
The sections with highest enterprise value are:
| Compendium section | What to take from it | How to use it |
|---|---|---|
| Evaluation Software | Harnesses and app-level eval tools: lm-eval-harness, OpenAI Evals, OpenEvals, RAGAS, DeepEval, MLflow Evaluate, Promptfoo, TruLens, Phoenix, Langfuse, BigCode harness, AutoGenBench, Copilot Arena | Choose a small number of formats for internal eval definitions, CI gates, trace capture, and dashboards |
| Evaluation of evaluation | Construct validity, leaderboard illusion, error bars, robustness, contamination, reproducibility | Require confidence intervals, held-out tasks, benchmark provenance, and score-delta thresholds |
| LLM as Judge | Rubric scoring, pairwise ranking, evaluator bias, multi-agent judging, code judging | Use judges for subjective dimensions only; calibrate against human labels and route borderline cases |
| HITL | Human rating, review workflows, task sampling | Make human review part of eval design, not a cleanup step after deployment |
| Question Answering / RAG Evaluation | Faithfulness, citation quality, multi-hop retrieval, multi-turn RAG, answer-engine citation failure | Score retrieval, grounding, citation, abstention, and answer utility separately |
| Code Generating LLMs / Copilots | SWE-bench, LiveCodeBench, Copilot Evaluation Harness, Copilot Arena, secure-code evals | Evaluate code agents by executable outcomes, trace discipline, security, and repo convention adherence |
| Agent LLM Architectures / Agentic Search | Tool use, long-horizon tasks, browser/web tasks, MCP/tool benchmarks, agent-as-judge | Score task completion plus trajectory: tool selection, recovery, permissions, and side effects |
| Document Understanding / Verticals | Long documents, PDFs, multimodal docs, medical/legal/finance QA | Build persona-specific tasks on the real document forms each function uses |
| Task Utility | Human-agent alignment and work utility | Elevate outcome usefulness above answer accuracy when tasks affect a business decision |
Evaluation Domains
Users should classify evals into domains before assigning metrics. One prompt can touch several domains, but one domain should own the release gate.
| Domain | What it evaluates | Common harnesses / tools | Primary owner |
|---|---|---|---|
| Model capability | Raw model performance on standard tasks | lm-eval-harness, OpenAI Evals, BigCode harness, MTEB, HELM-style suites | AI platform / model risk |
| Retrieval quality | Whether the right evidence is found and ranked | BEIR/MTEB-style retrieval, RAGAS context metrics, custom source-hit checks | Knowledge platform / search |
| Grounded generation | Whether claims are supported by retrieved evidence | RAGAS, TruLens, DeepEval faithfulness, source-span validators | Product owner / QA |
| Citation and provenance | Whether answer citations point to exact evidence | citation span tests, answer-engine evals, LongCite-style checks | Knowledge governance |
| Knowledge freshness | Whether stale or superseded sources are avoided | canonical-source registry, stale-fact golden sets, recency tests | Knowledge owner / records owner |
| Tool and function calling | Whether the right tool is selected with valid args | BFCL/Gorilla-style tests, tool schema validators, MCP/tool-use traces | Agent platform |
| Agent trajectory | Whether planning, retries, recovery, and side effects are acceptable | AutoGenBench, AgentBench, trace reviewers, OpenTelemetry/OpenInference spans | Agent runtime owner |
| Coding-agent work | Whether a repo moves from failing to verified with acceptable diff | SWE-bench-style harnesses, unit/integration tests, static analysis, code review rubric | Engineering |
| Browser / UI operation | Whether a web or desktop task is completed safely | WebArena/Mind2Web/OSWorld-style environments, DOM assertions, approval logs | Automation owner |
| Document understanding | Whether PDFs, tables, charts, and long docs are parsed correctly | Chandra extraction checks, DocVQA/OmniDocBench-style tests, table validators | Data / document pipeline |
| Safety and policy | Whether the system refuses or escalates appropriately | jailbreak/adversarial suites, policy rubrics, HITL sampling | Risk / security / legal |
| Persona utility | Whether the output helps the target role complete work | human rubric, task completion, time saved, escalation quality | Business function |
| Cost and latency | Whether success is economically deployable | token/cost logs, p95 latency, cache hit rate, cost per correct outcome | FinOps / platform |
| Evaluator reliability | Whether the judge/human rubric is trustworthy | inter-rater reliability, judge calibration, disagreement audits, error bars | Evaluation owner |
This classification prevents the common failure where a team treats a RAG faithfulness score as a coding-agent score, or a public model benchmark as evidence that a persona workflow is ready for a controlled deployment gate.
The Enterprise Evaluation Stack
Layer 1: Public Benchmark Calibration
Use public benchmarks to identify obvious mismatches:
- Coding: SWE-bench, SWE-bench Verified/Pro, LiveCodeBench, HumanEval, MBPP, BigCode, secure-code evals.
- Retrieval and embeddings: MTEB, MMTEB, BEIR, LoTTE, BIRCO, STARK, LongEmbed, LoCo.
- RAG: RAGAS, RAGChecker, RAGTruth, MTRAG, FRAMES, MultiHop-RAG, LiveRAG, Answer Engine Eval.
- Agentic tasks: AgentBench, AutoGenBench, WebArena, Mind2Web, MCP/tool-use benchmarks, deep-research benchmarks.
- Documents: OmniDocBench, DocBench, LongDocURL, MMLongBench-Doc, FinanceBench, engineering-document QA.
- Verticals: medical QA, legal summarization, financial document QA, scientific-document representation.
Do not use this layer as the buying decision. It answers “is this candidate plausible?” It does not answer “will this work in our estate?”
Layer 2: Enterprise System Evals
This is the right place for application deployment gates.
| System type | Required eval dimensions |
|---|---|
| Knowledge-base assistant | retrieval recall, source authority, answer faithfulness, citation accuracy, freshness, abstention, permission fidelity |
| GraphRAG / enterprise memory | entity resolution, edge provenance, multi-hop reasoning, stale-fact handling, correction workflow, permission inheritance |
| Coding agent | issue resolution, test pass rate, diff minimality, repository convention adherence, secret handling, command safety, rollback quality |
| Browser / workflow agent | task completion, DOM/tool accuracy, side-effect control, approval checkpoints, recovery from changed UI |
| Deep-research agent | source discovery, source ranking, citation fidelity, contradiction handling, synthesis quality, uncertainty labeling |
| Support / service agent | policy adherence, resolution accuracy, escalation timing, tone, multi-turn memory, handoff quality |
Use deterministic checks whenever possible. Code should compile and tests should run. JSON should validate. Citations should point to source spans. Tool calls should be schema-valid. The LLM judge should not be the first evaluator when a machine-checkable contract exists.
Layer 3: Persona-Specific Golden Work Packets
The enterprise gap is persona. Generic tasks hide the real failure modes.
Each golden packet should include:
| Field | Requirement |
|---|---|
| Persona | The role performing the work: engineer, support lead, finance analyst, legal counsel, HR business partner, executive assistant, security analyst |
| Task | A real work request, phrased as the persona would receive it |
| Source bundle | Documents, repo files, tickets, dashboards, policies, messages, or transcripts the agent is allowed to use |
| Permission boundary | What the agent can see and what it must refuse |
| Output contract | Memo, code diff, SQL, JSON, ticket response, action plan, spreadsheet, citation table |
| Deterministic checks | Tests, schema validation, exact extraction, source-span match, calculation recomputation |
| Rubric checks | Usefulness, risk framing, tone, concision, legal/medical/professional appropriateness |
| Trace requirements | Tool calls, retrieved chunks, prompts, costs, approvals, errors, retries, final evidence |
| Failure taxonomy | Hallucination, stale source, wrong persona, over-action, under-action, permission leak, bad citation, unsafe code, poor escalation |
| Harness context | Harness name/version, model name/version, prompt template version, retrieval index version, tool catalog version, judge model/version, dataset snapshot |
The golden packet is the enterprise eval format. It is portable across vendors and models because the task, evidence, and score contract are independent of any one platform.
Harness Context Record
Every eval result should carry enough context to reproduce and explain it later.
harness:
name: deepeval
version: 0.0.0
runner_commit: git_sha
suite_id: enterprise_kb_policy_qa_v1
task_id: kb_policy_017
model_under_test:
provider: anthropic
model: claude-sonnet
model_version: vendor_reported_version_or_snapshot
temperature: 0
max_output_tokens: 1200
application_context:
prompt_template_id: policy_qa_v4
retrieval_index_id: confluence_hr_policy_2026_06_20
embedding_model: text_embedding_model_and_version
reranker_model: reranker_and_version
tool_catalog_version: mcp_tools_2026_06_20
persona: hr_business_partner
judge_context:
judge_type: rubric_llm_plus_human_sample
judge_model: gpt-4o_or_local_prometheus_version
rubric_version: policy_grounding_rubric_v3
calibration_set: judge_calibration_2026_06
run_context:
started_at: 2026-06-21T00:00:00Z
latency_ms: 0
input_tokens: 0
output_tokens: 0
cost_usd: 0
trace_id: otel_trace_id
Without this record, a score is not operational evidence. It is an anecdote.
Evaluating Enterprise Knowledge Bases
Knowledge-base evaluation must separate retrieval quality from answer quality.
Minimum Metric Set
| Dimension | Question | Example metric |
|---|---|---|
| Retrieval recall | Did the system retrieve the source needed to answer? | source hit rate at k; context recall |
| Retrieval precision | Were the top chunks actually useful? | context precision; nDCG@k |
| Source authority | Did it prefer canonical documents over stale duplicates? | canonical-source rank; stale-source penalty |
| Grounding | Are answer claims supported by retrieved context? | claim-level faithfulness |
| Citation fidelity | Do citations point to the exact supporting span? | citation span accuracy |
| Abstention | Did it refuse when sources were insufficient? | false-answer rate; false-abstention rate |
| Permission fidelity | Did retrieval obey ACLs and persona scope? | forbidden-source access count |
| Freshness | Did it detect superseded policies and outdated runbooks? | stale fact error rate |
| Utility | Did the answer help the persona complete the task? | human rubric or task-completion score |
Why RAGAS Is Necessary But Insufficient
RAGAS-style metrics are useful because they score faithfulness, answer relevance, context precision, and context recall. The limitation is structural: faithfulness to retrieved context is not truth. If the retrieved document is stale, wrong, or non-canonical, a high faithfulness score can certify a bad answer.
Enterprise RAG evals need a parallel knowledge-quality layer:
- canonical-source registry;
- document freshness and supersession metadata;
- owner/reviewer fields;
- source-system permissions;
- confidence decay;
- correction events;
- stale-fact golden questions.
For internal knowledge bases, the most important failure is often not hallucination. It is a confidently grounded answer from the wrong internal source.
Evaluating Coding Agents Like Claude Code
Coding agents have a stronger verification surface than general assistants because the work can be executed. That advantage should be used fully.
What To Score
| Dimension | Why it matters |
|---|---|
| Issue resolution | Did the requested behavior change occur? |
| Test discipline | Did the agent run the right tests and interpret failures correctly? |
| Diff quality | Is the patch minimal, maintainable, and consistent with local patterns? |
| Repo navigation | Did it find the right files without broad, noisy changes? |
| Build-system handling | Did it work with real dependency, cache, and environment constraints? |
| Security | Did it avoid secrets, destructive commands, unsafe shell behavior, and exfiltration? |
| Tool trace quality | Are file reads, edits, commands, and approvals auditable? |
| Recovery | Did it recover from failing tests or wrong assumptions? |
| Human handoff | Does the final report state what changed and what was verified? |
Recommended Coding-Agent Eval Format
id: coder_agent_repo_task_001
persona: senior_backend_engineer
repo_snapshot: git_sha_or_archive_id
task_prompt: "Fix the failing renewal reminder calculation for annual contracts."
allowed_tools:
- read_files
- edit_files
- run_tests
- inspect_git_diff
disallowed_actions:
- network_download
- destructive_git_reset
- secret_file_access
expected_artifacts:
- source_patch
- test_output
- final_summary
deterministic_checks:
- command: "npm test -- renewal"
must_pass: true
- command: "npm run lint"
must_pass: true
rubric:
maintainability: "Patch follows existing date utilities and avoids one-off parsing."
blast_radius: "Only renewal calculation and focused tests changed."
explanation: "Final answer names changed files and verification."
trace_requirements:
- files_read
- files_modified
- commands_run
- test_failures_observed
- approvals_requested
This format maps naturally to Claude Code, Codex, Cursor, Copilot Workspace, and other repo agents. It also creates training and regression material: successful traces become procedural memory; failed traces become benchmark negatives.
Evaluation Across Personas
The same knowledge base should score differently for different users because the right answer changes by role.
| Persona | Task class | Special eval requirements |
|---|---|---|
| CIO / CTO | Vendor comparison, architecture decision, incident response | evidence ranking, risk/benefit framing, actionability, cost and governance implications |
| Engineer | Code change, debug, architecture review, repo question answering | tests, static checks, diff quality, local convention adherence, secure command behavior |
| Support leader | Customer answer, policy exception, escalation | policy grounding, tone, resolution accuracy, escalation timing, CRM/ticket trace |
| Legal counsel | Clause review, risk memo, litigation summary | jurisdictional citation, privilege boundaries, abstention, exact source span support |
| Finance analyst | KPI explanation, filing QA, spreadsheet/workbook analysis | numeric recomputation, table extraction, formula provenance, stale-period detection |
| HR / People ops | Policy interpretation, employee-response drafting | sensitive-data handling, policy currency, consistent application, escalation triggers |
| Security analyst | Alert triage, incident writeup, control mapping | evidence chain, ATT&CK/control mapping, false-positive handling, no autonomous containment without approval |
| Executive assistant | Scheduling, briefing, follow-up | preference accuracy, calendar/email permissions, action confirmation, no silent external sends |
This is where “domain-specific knowledge” becomes testable. Do not ask whether a model knows the domain in general. Ask whether a persona can use it to complete the work under the right evidence and permission constraints.
Standards And Format Strategy
No single standard covers the full enterprise evaluation surface. The practical approach is a small format stack:
| Need | Format / tool family |
|---|---|
| Model-level benchmark tasks | lm-evaluation-harness, OpenAI Evals-style YAML/Python, BigCode harness |
| RAG and knowledge-base scoring | RAGAS-style metric records plus source-span/citation tables |
| Prompt and model regression | Promptfoo/OpenEvals-style config suites |
| CI/CD gates | DeepEval pytest-style assertions or custom GitHub Actions |
| Trace capture | OpenTelemetry + OpenInference-style spans, native Langfuse/Phoenix/Weave/Braintrust traces |
| Human review | Rubric rows with reviewer identity, score, rationale, and calibration set |
| Golden work packet | Repo-native YAML/JSON plus source bundle, expected artifact, checks, rubric, and trace contract |
The common denominator should be a warehouse table, not a vendor UI:
| Table | Grain |
|---|---|
eval_runs |
one run of a suite against a model/system version |
eval_tasks |
one golden packet or benchmark item |
eval_attempts |
one model/system attempt at one task |
eval_scores |
one metric result for one attempt |
eval_traces |
one trajectory or trace root |
eval_spans |
tool/retrieval/model/action spans |
eval_evidence |
retrieved chunks, citation spans, source docs, code files |
eval_human_reviews |
reviewer rubric scores and comments |
eval_failures |
normalized failure taxonomy |
That warehouse shape lets an enterprise compare Claude Code, Codex, Copilot, internal RAG, a support agent, and a legal assistant without forcing all teams into one vendor’s dashboard.
Corporate Eval Workbench Research Task
The eval pillar needs a practical research task in addition to the compendium index: design the private corporate eval workbench that teams can actually run. The current primary-source shortlist is captured in sources/21-benchmarks/enterprise-custom-eval-tooling-research-2026-raw.md and covers LangSmith, DeepEval, RAGAS, OpenAI Evals, Anthropic’s evaluation workflow, Microsoft Foundry, and Phoenix.
The research question is:
How should an enterprise create, version, run, observe, and govern custom evals and benchmark packs for its own knowledge bases, agents, personas, tools, and domain workflows?
Tool Roles
| Tool | Best role in the workbench | What it should not own alone |
|---|---|---|
| LangSmith | Offline and online app eval workflow, trace-to-dataset loop, annotation queues, evaluator comparison, production feedback loop | The only copy of eval evidence or the only canonical format |
| DeepEval | Repo-native eval tests, CI/CD gates, custom G-Eval/DAG metrics, RAG metrics, agent metrics, multi-turn metrics, safety metrics, synthetic goldens | Business acceptance without SME calibration |
| RAGAS | RAG and agentic-RAG component scoring: context precision/recall, faithfulness, response relevance, tool-call accuracy, agent-goal accuracy, custom RAG metrics | Proof that an answer is current, canonical, or permission-safe |
| OpenAI Evals | JSONL-style task cases, graders/testing criteria, model/prompt iteration, portable eval run records | Full trace observability or enterprise source governance |
| Phoenix | Self-hostable trace inspection, LLM evals, RAG debugging, dataset and span analysis | Deterministic unit tests, permissions governance, or source-authority registry |
The architectural rule is: use vendor tools as runners and inspectors, but keep the corporate benchmark pack in repo-native or warehouse-native form. A vendor dashboard can be useful; it should not be the durable source of truth.
Custom Corporate Eval Design
Corporate evals should be built as benchmark packs, not one-off prompts.
| Pack component | Requirement |
|---|---|
pack_id and version |
Stable identifier, semantic version, owner, review date, deprecation policy |
| Workflow family | RAG QA, support, coding agent, browser agent, SQL/data agent, legal memo, finance analysis, executive synthesis |
| Persona | Role, expertise level, permissions, expected decision context |
| Source bundle manifest | Documents, repo snapshot, tickets, emails, policies, tables, dashboards, ACL snapshot |
| Task cases | JSONL/YAML cases with prompt, expected artifact, allowed tools, disallowed actions, and risk tier |
| Deterministic validators | schema checks, tests, exact extraction, SQL equivalence, citation span match, policy-id match, numeric recomputation |
| Judge rubrics | subjective dimensions, judge model/version, prompt version, threshold, calibration labels |
| Human calibration | SME labels, reviewer identity or role, inter-rater notes, disagreement policy |
| Trace requirements | required span kinds, retrieval chunks, tool calls, model calls, approvals, costs, latency, errors |
| Failure taxonomy | normalized failure code and release-blocking severity |
| Baselines | model/application/index/tool versions that establish current production behavior |
Build Sequence
- Harvest candidates. Pull real tasks from production traces, support escalations, incident reports, code-agent failures, stale KB answers, and SME-authored examples.
- Normalize into golden work packets. Each case gets persona, source bundle, permission boundary, output contract, checks, rubric, trace contract, and failure taxonomy.
- Choose runners by layer. Use DeepEval for local/CI tests, RAGAS for RAG component metrics, LangSmith or Phoenix for trace/dataset workflows, OpenAI Evals where JSONL grader workflows fit.
- Calibrate judges. Run human labels against LLM judges, record disagreement, and require judge version/rubric version in every score.
- Promote to benchmark pack. Once a workflow has enough stable examples, freeze a versioned pack with baseline scores and release gates.
- Refresh from failures. Every production failure should become a candidate eval case or an explicit reason why the eval suite cannot catch it.
Corporate Benchmarks To Build First
| Benchmark pack | Minimum cases | Primary tools | Release gates |
|---|---|---|---|
corp_kb_policy_qa_v1 |
100 | RAGAS, DeepEval, LangSmith/Phoenix traces | zero permission leaks; citation span accuracy threshold; stale-source error threshold |
corp_coding_agent_repo_v1 |
50 | DeepEval/custom pytest, repo tests, trace export | tests pass; no unsafe commands; bounded diff; no secret access |
corp_support_resolution_v1 |
100 | LangSmith datasets, DeepEval multi-turn metrics, human review | policy grounding; escalation accuracy; tone; no false promise |
corp_sql_analyst_v1 |
75 | execution validators, RAGAS SQL metrics, Phoenix traces | SQL equivalence/execution pass; no forbidden table access; numeric correctness |
corp_deep_research_v1 |
50 | source-discovery scoring, citation validators, human review | source quality; contradiction handling; no uncited material claims |
This is the missing enterprise bridge between public benchmark literature and deployment. Public benchmarks identify candidate capability. Corporate benchmark packs decide whether a system is safe to ship.
Failure Taxonomy
Enterprise evals should classify failures, not just count wrong answers.
| Failure | Domain | Description | User classification path | Required context |
|---|---|---|---|---|
| Unsupported answer | Grounded generation | Output contains claims not supported by allowed evidence | Mark as grounding.unsupported_claim; attach unsupported claim and expected source span |
retrieved chunks, answer claims, judge/rubric version |
| Wrong source | Retrieval / knowledge freshness | Output is grounded, but in stale, non-canonical, or low-authority material | Mark as retrieval.wrong_source; identify canonical source that should have won |
source authority registry, index version, recency metadata |
| Retrieval miss | Retrieval quality | The necessary source existed but was not retrieved | Mark as retrieval.miss; add missing source to golden evidence set |
retriever version, top-k list, query rewrite trace |
| Citation mismatch | Citation and provenance | Citation exists but points to the wrong page, span, table, or document | Mark as citation.bad_span; record expected and actual span |
citation parser version, document OCR/source version |
| Stale fact | Knowledge freshness | Agent uses an old policy, deprecated API, superseded decision, or retired runbook | Mark as knowledge.stale_fact; link superseding source |
source timestamps, owner, review date |
| Permission leak | Safety / retrieval | Agent used or exposed data outside persona scope | Mark as permission.leak; treat as release-blocking |
identity, groups, ACL snapshot, forbidden source id |
| Bad abstention | Safety / utility | Agent refused when evidence was sufficient, or answered when evidence was insufficient | Mark as abstention.false_refusal or abstention.false_answer |
evidence sufficiency label, rubric version |
| Tool misuse | Tool calling | Wrong tool, wrong parameters, repeated failed calls, or side effects without approval | Mark as tool.misuse; include tool schema and call args |
tool catalog version, MCP/server version, approval policy |
| Trajectory drift | Agent trajectory | Agent starts on-task but drifts, loops, over-plans, or ignores observations | Mark as agent.trajectory_drift; attach trace segment |
trace id, span sequence, planner version |
| Persona mismatch | Persona utility | Output is technically correct but not useful for the role’s decision | Mark as persona.mismatch; identify target role and missing decision need |
persona id, task rubric, human reviewer role |
| Format breach | Output contract | Output fails schema, citation, style, or artifact contract | Mark as format.contract_breach; include validator errors |
schema version, parser version, output artifact |
| Unsafe code | Coding-agent work | Patch introduces security, data-loss, or operational risk | Mark as code.unsafe_change; require engineering review |
repo SHA, diff, tests, static analysis, secret scan |
| Test illusion | Coding-agent work | Agent passes narrow tests while missing behavior or breaking adjacent flows | Mark as code.test_illusion; add regression task |
test suite version, coverage gap, failing hidden check |
| Over-action | Browser / workflow / agent | Agent takes an irreversible or unnecessary action | Mark as action.overreach; classify side effect severity |
approval log, tool/action trace, rollback status |
| Cost blowout | Cost and latency | Task succeeds but exceeds acceptable token, runtime, or review cost | Mark as cost.blowout; compare to budget |
token log, latency, model route, cache status |
| Judge disagreement | Evaluator reliability | LLM judge, human reviewer, and deterministic check disagree | Mark as judge.disagreement; route to calibration |
judge model/version, rubric version, human labels |
| Trace gap | Trace / audit | The outcome cannot be audited because evidence, tool calls, or approvals are missing | Mark as trace.missing_context; fail auditability gate |
trace exporter version, missing span kinds |
| Product-context miss | Automated error analysis / persona utility | The trace looks acceptable, but the product goal, channel, escalation rule, or user-experience requirement was violated | Mark as eval.product_context_miss; require human annotation and product-rule capture |
trace id, channel metadata, product rule, human annotation, auto-analysis output |
| False-positive eval finding | Evaluator reliability | Auto-analysis flags a convincing but invalid issue, overstates severity, or treats internal metadata as user-facing context | Mark as eval.false_positive; retain rejected issue and severity correction |
issue cluster, reviewer decision, severity calibration, cited span |
This taxonomy is more useful than a single 0-100 score because it tells the deployment team what to fix.
Failure Domain Routing
Classification should also decide who owns the fix:
| If the failure starts with | First owner | Typical fix |
|---|---|---|
retrieval.* |
Search / knowledge platform | chunking, query rewrite, index refresh, reranker, canonical-source metadata |
citation.* |
Knowledge governance / document pipeline | source-span extraction, OCR correction, citation formatter |
knowledge.* |
Business content owner | source cleanup, supersession links, owner review cycle |
permission.* |
Security / identity / platform | ACL resolver, connector permissions, gateway policy |
tool.* |
Agent platform / MCP owner | schema tightening, tool descriptions, tool allowlist, runtime validation |
agent.* |
Agent runtime owner | planner, retry policy, observation handling, step budget |
persona.* |
Business function owner | persona rubric, task framing, output contract |
format.* |
Product / QA | schema validation, structured output constraints |
code.* |
Engineering | hidden tests, static analysis, repo-specific skill, code review |
action.* |
Automation owner / risk | approval policy, side-effect containment, rollback |
cost.* |
FinOps / platform | model routing, caching, context compression, budget gate |
judge.* |
Evaluation owner | rubric revision, judge calibration, human review sampling |
eval.* |
Evaluation owner / product owner | annotation workflow, product-context capture, false-positive review, trace-visible vs product-context split |
trace.* |
Observability owner | OTel/OpenInference export, span coverage, retention |
Operating Model
- Pick five critical workflows per persona. Use real internal work, not generic prompts.
- Create 20-50 golden packets per workflow. Include source bundles, permission scopes, and expected artifacts.
- Add deterministic checks first. Tests, schemas, citation spans, calculation recomputation, policy IDs, file diffs.
- Add rubric judges only where judgment is unavoidable. Calibrate against human reviewers.
- Capture traces by default. Retrieval, tool calls, costs, approvals, errors, and final evidence.
- Gate deployment on failure families. For example: zero permission leaks, zero unsafe code, citation fidelity above threshold, abstention false-negative below threshold.
- Refresh continuously. Add examples from incidents, support escalations, failed code-agent runs, stale KB answers, and human corrections.
Procurement Questions
Ask every vendor:
- Can eval tasks be exported in a documented format, or are they trapped in the vendor UI?
- Can traces be exported with prompts, tool calls, retrieved chunks, costs, latency, approvals, and model versions?
- Can the system run private golden datasets without training on them?
- Can it score retrieval separately from generation?
- Can it enforce source-system ACLs during eval and report auditable evidence of forbidden-source non-access?
- Can human reviews be attached to tasks and used for judge calibration?
- Can evals block deployment in CI/CD or release workflows?
- Can code-agent evals run in isolated repo snapshots with deterministic tests?
- Can failures be classified by type, not only aggregated into an average score?
- Can evaluation records be retained in the customer’s warehouse for audit?
Bottom Line
The modern enterprise eval program is not a leaderboard subscription. It is a work-measurement system.
For knowledge bases, the test is whether the right persona gets the right answer from the right source under the right permission boundary. For coding agents, the test is whether the agent can move a real repository from failing state to verified state with an auditable trace and acceptable blast radius. For domain-specific assistants, the test is whether procedural and source knowledge changes outcomes, not whether the base model scores well on public exams.
The format that holds this together is the golden work packet: persona, task, source bundle, permission boundary, output contract, checks, rubric, trace, and failure taxonomy. Build that layer once, and vendors become replaceable candidates instead of the owners of the evaluation program.