Source files: sources/21-benchmarks/awesome-evals-primary-sources-2026.md · sources/21-benchmarks/awesome-evals-primary-sources-2026.jsonl · sources/21-benchmarks/awesome-evals-golden-exemplars-2026.jsonl · sources/21-benchmarks/eval-framework-native-schema-primary-sources-2026.md · sources/21-benchmarks/frontier-lab-eval-sdk-primary-sources-2026.md · sources/21-benchmarks/eval-native-schema-boundaries-2026.json · sources/21-benchmarks/parlance-auto-evals-production-trace-error-analysis-raw.md · sources/21-benchmarks/model-routing-meaningfulness-hse-robustness-raw.md · statebench/schemas/eval_result_v1.schema.json · statebench/schemas/failure_taxonomy_v1.schema.json.
Production companion pages: Enterprise Evaluation for Knowledge Bases and Coding Agents · Eval Framework Schema Crosswalk · Frontier Lab Evaluation Research and SDK Primitive Crosswalk · Decentralized GitLab Eval Federation
Production machine-readable artifacts: Awesome Evals source ledger · Awesome Evals source summary · Awesome Evals golden exemplars · PDF/Chandra queue
Ingestion status: The Awesome Evals ingestion materializes 318 cited rows, 318 golden-quality exemplar rows, 269 raw mirrors, 143 mirrored BenchFlow notes, and 83 Chandra-extracted PDF rows, including 2 chunked extractions. The 46 media rows were attempted through the transcript path; public captions were unavailable, and the local ASR pass is queued because audio download/transcription requires the podcast/video ASR pipeline.
Answer
An enterprise eval is not a score. It is an evidence contract that says: for this task, this model, this prompt, this retrieval index, this tool catalog, this runtime, and this judge/check, here is what happened, where it failed, and what evidence is retained.
The durable architecture is:
- Run evals repo-locally where application code, fixtures, prompts, and tool schemas live.
- Normalize only the result row into a firm-wide schema.
- Keep native framework exports as evidence, because DeepEval, RAGAS, LangSmith, Phoenix, Hugging Face, provider SDKs, and custom pytest suites do not share one stable native schema.
- Promote only reviewed failures into knowledge-base updates, SFT rows, preference pairs, or policy changes.
The Awesome Evals corpus strengthens the prior finding: evals are becoming a product capability, not a one-off QA activity. The field is moving from static benchmark scores toward verifier design, RL environments, benchmark integrity checks, agent trajectory scoring, security/adversarial harnesses, and observability-backed eval loops.
Where Evals Live In The Agentic Chat Cycle
| Cycle point | Eval artifact | Main failure domains | Framework or lab primitives |
|---|---|---|---|
| User/task intake | task card, persona, policy boundary, expected artifact | ambiguous intent, wrong persona, missing authority, unsafe request | OpenAI eval item schema, Anthropic task design guidance, Google ADK eval sets, custom golden work packets |
| Prompt/context assembly | prompt template, memory state, retrieved context, tool catalog | stale context, ACL leak, prompt drift, missing source, wrong tool exposure | RAGAS contexts, DeepEval context fields, LangSmith/Phoenix traces, Bedrock KB traces |
| Model generation | model call, sampling config, response | hallucination, refusal error, format drift, cost/latency blowout | OpenAI graders, model eval suites, Hugging Face LightEval/Evaluate, lm-eval-harness |
| Retrieval and grounding | retrieved chunks, citations, source bundle | wrong source, unsupported claim, stale source, citation span mismatch | RAGAS faithfulness/context metrics, DeepEval RAG metrics, Phoenix retrieval spans, FACTS/LongFact-style grounding |
| Tool use and action | tool schema, tool call, observation, approval | wrong tool, invalid args, side effect, failed recovery, permission bypass | provider traces, Claude Code/MCP hooks, AutoGen traces, Qwen-Agent function calls, custom deterministic checks |
| Model/tool routing | routed actor, candidate pool, route rationale, perturbation set | redundant model pool, unstable assignment, cost-only routing, accuracy/robustness split | AI gateways, RouterBench/EmbedLLM-style matrices, HSE, perturbation robustness |
| Multi-turn trajectory | state deltas, handoffs, retries, final answer | goal drift, loop failure, bad recovery, compounded stochastic failure | Anthropic agent eval concepts, Google trajectory/final response split, LangSmith/Phoenix spans, AutoGenBench |
| Judge and review | rubric, judge model/version, human label, rationale | judge bias, criteria drift, validator invalidity, label error | LLM-as-judge frameworks, EvalGen-style validator validation, human annotation queues |
| Governance rollup | normalized row, source bundle, failure code, review decision | unverifiable score, missing trace, benchmark overclaim, unapproved training promotion | eval_result_v1, failure taxonomy, GitLab component, warehouse rollups |
Eval Domains And Hard Boundaries
| Domain | What it measures | What it does not prove | Enterprise storage boundary |
|---|---|---|---|
| Public model benchmark | base capability on a known benchmark | product reliability or private workflow fit | model registry evidence only |
| Product task eval | output quality for a workflow | general model intelligence | repo-local eval pack plus normalized rows |
| RAG/KB eval | retrieval, grounding, citation, abstention | source authority, freshness, ACL compliance unless separately checked | source bundle plus retrieval trace |
| Coding-agent eval | diff correctness, test pass, repo navigation, tool behavior | maintainability or production safety unless reviewed | repo snapshot, patch, test logs, trace |
| Agent trajectory eval | tool choice, state updates, recovery, handoffs | final answer truth by itself | trace spans and action evidence |
| Judge/verifier eval | rubric or reward reliability | that the underlying model is safe or correct | judge config, judge model version, calibration set |
| RL environment eval | whether success can be verified cheaply and repeatedly | that the reward cannot be hacked | environment spec, verifier, exploit tests |
| Security/adversarial eval | jailbreak, prompt injection, sabotage, exfiltration | business-task success | red-team suite plus policy failure code |
| Observability eval | trace completeness and production drift | task correctness without labels | trace coverage, metrics, sampled reviews |
| Routing meaningfulness eval | whether a router is choosing among differentiated actors and making stable assignments | final-answer correctness, safety, or cost by itself | route logs, behavioral matrix, HSE score, perturbation robustness |
| Training/SFT promotion | reusable corrected behavior | eval result validity | redacted training example with lineage and approval |
The hardest boundary is faithfulness vs truth. A RAG answer can be faithful to an unauthorized, stale, or wrong source. Grounding metrics must therefore sit beside source authority, freshness, ACL, and citation-span checks.
Framework Overlap
| Concept | DeepEval | RAGAS | LangSmith | Phoenix | Hugging Face | Provider SDKs | Corporate schema |
|---|---|---|---|---|---|---|---|
| Case input | input |
user_input |
example/run input | span input | dataset sample | user message/task | task.input |
| Expected/reference | expected_output, context |
reference, reference_contexts |
reference output | annotation | reference labels | expected artifact/check | expected_artifact |
| Actual output | actual_output |
response |
run output | span output | prediction/generation | final response | attempt.actual |
| Retrieved evidence | retrieval_context |
retrieved_contexts |
trace docs | retrieval spans | dataset docs | KB/tool observations | evidence.role=retrieved |
| Metric/judge | metric object | metric | evaluator/feedback | evaluator | metric module/task | grader/check | metric_result |
| Trace | optional/external | external | native run trace | native trace | logs/bundles | native run spans | trace_ref |
| Model/runtime | metric/provider config | evaluator/provider config | run metadata | span metadata | model/backend revision | model/runtime call | model, harness, runtime_context |
| Failure class | custom reason | custom label | feedback tag | annotation | postprocess | error/event type | failure_code |
The overlap is enough for a shared warehouse row. It is not enough to force every team into one runner.
Lab And SDK Divergence
| Lab/platform | Eval posture | SDK/runtime primitive | Divergence to preserve |
|---|---|---|---|
| OpenAI | eval definitions, graders, agent traces, third-party eval trust | Evals API, graders, Agents SDK traces | strong grader abstractions; trace and eval artifacts still need warehouse linkage |
| Anthropic | agent-specific eval design, MCP/tooling, Claude Code-style permissioned agents | Agent SDK, MCP, tool traces, Claude Code hooks | emphasizes task design and trajectory/final outcome split more than one public schema |
| Google/DeepMind | grounding/factuality research, ADK evals, A2A protocols | ADK eval sets, Gen AI evaluation service, A2A task/artifact concepts | protocol-level conformance matters alongside model scoring |
| Microsoft | multi-agent and enterprise orchestration | AutoGen, Foundry monitoring, Magentic-style systems | team graph and handoff behavior are first-class eval surfaces |
| AWS | Bedrock Agents, Knowledge Bases, AgentCore-style observability | KB citations, action groups, Bedrock traces | source authority and enterprise controls are platform-specific |
| Meta/Hugging Face/open model ecosystem | model cards, datasets, LightEval/lm-eval style harnesses | Hub revisions, LightEval, Evaluate, TRL/SFT, PEFT | registry/training substrate is separate from product eval warehouse |
| NVIDIA | safety and guardrail eval/tooling | garak, NeMo Guardrails, runtime observability | adversarial and policy checks sit beside task quality |
| Chinese and second-tier labs | model reports, OpenAI-compatible APIs, function calling, local agent frameworks | Qwen-Agent, DeepSeek APIs, Kimi/GLM/Tencent/Baidu SDK surfaces | API compatibility does not mean behavior, tokenizer, tool policy, or eval compatibility |
What Awesome Evals Changed
The 318-row ingestion does not overturn the existing architecture. It sharpens it.
| Prior finding | Updated after Awesome Evals ingestion |
|---|---|
| Normalize result rows, not frameworks | Confirmed. The corpus spans runners, dashboards, papers, observability tools, talks, and RL environment work; one native schema would discard evidence. |
| RAG eval is a retrieval/faithfulness problem | Expanded. RAG eval is also source authority, permissioning, freshness, abstention, and citation-span evidence. |
| Agent eval needs traces | Strengthened. Several sources emphasize outcome-vs-trajectory, state deltas, pass@k/pass^k, recovery, stochasticity, and harness engineering. |
| LLM-as-judge is useful but risky | Strengthened. Judge bias, criteria drift, validator validation, label errors, and benchmark ground-truth mistakes need their own eval lane. |
| Automated trace analysis can replace error analysis | Rejected. Parlance Labs’ 100-trace production experiment shows strong recovery of trace-visible failures, but all tested systems missed product-context failures that looked acceptable in the trace. |
| Model routing is solved by cost/quality optimization | Rejected. Google DeepMind’s routing-meaningfulness paper shows clean accuracy can diverge from behavioral diversity and perturbation robustness. |
| Benchmark scores are procurement weak evidence | Strengthened. Benchmark integrity, label error, contamination, live/held-out tests, and leaderboard incentives are recurring themes. |
| GitLab runners can decentralize eval execution | Confirmed. Repo-local execution is the only place private fixtures, code, tools, and workflow traces are naturally available. |
| SFT promotion should come from eval failures | Expanded. The corpus points to verifier/RL environment design and trajectory training, not just prompt/completion SFT. |
What The Parlance Auto-Evals Experiment Adds
Parlance Labs adds a missing empirical lane: automated error-analysis tools over real production traces. Their July 2026 experiment compared 100 human-annotated apartment-leasing agent traces against Braintrust Loop, Arize AX Alyx, LangSmith chat-agent analysis, Codex, Factory Droid, and Claude Code. The best system recovered 87.2% of the 39 human-labeled failures; Arize had the highest reported precision at 91.0%; coding agents were competitive with dedicated platforms on finding failures.
The important result is not the vendor order. It is the boundary:
| Boundary | What automated systems caught | What they missed |
|---|---|---|
| Trace-visible failure | unsupported claims, repeated asks, tool-output contradictions, unbacked promised follow-ups | product objective failures that looked polite or complete inside the transcript |
| Product-context failure | only when the rule was present in trace or prompt | sales-objection handling, SMS Markdown, voice interruption behavior, handoff rules |
| Workflow value | platforms kept findings linked to traces and supported annotation/rerun workflows | every tested system treated the job as one-shot instead of interviewing the user or learning from annotations |
This sharpens the operating model: automated trace analysis should be a
review-acceleration loop, not an autonomous judge. The corporate schema should
record error_analysis_mode, failure_visibility, human_annotation_ref,
auto_issue_cluster, review_decision, and severity_calibration so teams can
measure which failures are trace-visible versus product-context-dependent.
What The Routing Meaningfulness Paper Adds
Google DeepMind’s July 2026 routing paper adds a structural eval for AI gateways and multi-model systems. A router is only meaningful if the candidate actors are behaviorally differentiated and the assignment policy is stable under meaning-preserving perturbations.
This adds four failure codes to the enterprise eval taxonomy:
| Failure code | Meaning | Evidence |
|---|---|---|
routing.redundant_society |
routed models/agents behave too similarly for routing to matter | actor list, behavioral matrix, HSE score |
routing.assignment_instability |
paraphrases or minor rewrites route to different actors | perturbation set, original route, perturbed routes, robustness score |
routing.coreset_overbuild |
extra models add governance/cost burden without meaningful diversity | HSE-vs-size curve, marginal diversity gain |
routing.accuracy_robustness_split |
clean accuracy improves while perturbation robustness falls | clean accuracy, perturbed accuracy, robustness by perturbation level |
The practical deployment rule: gateway evals should report accuracy, cost, and latency, but also the diversity of the model pool and route stability. Otherwise a firm can buy a sophisticated router that mostly arbitrages redundant models or routes equivalent requests inconsistently.
Golden JSONL Exemplar Shape
Each Awesome Evals citation now has one exemplar row that can seed a corporate golden work packet. A row is intentionally not a full benchmark. It is a typed seed that says how this source should influence failure classification and evidence retention.
{
"exemplar_id": "awesome_evals::ae-006-demystifying-evals-for-ai-agents",
"source_id": "ae-006-demystifying-evals-for-ai-agents",
"source_url": "https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents",
"source_type": "web_article",
"eval_domain": "agent_trajectory",
"recommended_failure_code": "trajectory_error",
"golden_case": {
"persona": "enterprise_eval_owner",
"task_id": "ae-006-demystifying-evals-for-ai-agents::classify_failure",
"input": "Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.",
"expected_artifact_type": "source_document",
"expected_fields": [
"harness.name",
"harness.version",
"model.provider",
"model.name",
"runtime_context.prompt_template_id",
"runtime_context.retrieval_index_id",
"attached_tools",
"trace_ref",
"failure_code",
"source_bundle_ref"
]
}
}
The exemplar contract forces the team to record the harness, model version, tools, retrieval index, trace reference, failure code, and source bundle. That is the minimum shape needed for a failure to roll up from repo CI into a team dashboard, firm-wide scorecard, KB update, or training-data promotion queue.
Decentralized Rollup Model
The target operating model is:
- Product repo owns
evals/<pack_id>/cases.jsonl, fixtures, source bundle manifest, and native runner configs. - GitLab runner executes the native framework close to the code and secrets.
- Runner emits JUnit, JSONL, trace bundles, annotations, and raw framework exports.
- Team collector normalizes rows to
eval_result_v1and adds review status. - Firm collector aggregates normalized rows, not private fixtures.
- Knowledge owners promote only reviewed, redacted, source-backed failures into wiki pages, KB patches, SFT rows, preference pairs, or policy gates.
This keeps private repo evidence decentralized while still giving leadership a firm-wide view of failure rates, model regressions, retrieval index drift, judge drift, and source-quality gaps.
Remaining Verification Backlog
The ingestion is enough to update the synthesis and wiki. It is not enough to declare every source fully processed through every modality-specific pipeline.
| Backlog item | Count | Required next step |
|---|---|---|
| Media items | 46 | Captions unavailable; run yt-dlp plus Whisper/WhisperX ASR through the podcast/video path and keep needs_asr until transcript artifacts exist. |
| Paper/PDF-like items | 83 | All 83 paper/PDF rows have Chandra text evidence, including 2 chunked extractions. |
| Metadata-only rows | 3 | Revisit blocked/indirect sources and either mirror allowed content or keep as metadata-only citations. |
| Native schema gaps | tracked separately | Add exact export fixtures for framework versions whose hosted exports are not stable public schemas. |
The practical conclusion is that the evaluation pillar now has a stronger map, a larger citation graph, and a machine-readable exemplar set. The remaining work is modality completion for the video/podcast/lecture rows.