AI evaluation operations is the enterprise practice of turning model, RAG, coding-agent, and workflow failures into auditable evidence that can run in product repos and roll up to team and firm-wide scorecards.
The core rule: centralize normalized results, not private fixtures or native runner formats. Each repo should keep its own DeepEval, RAGAS, LangSmith, Phoenix, Promptfoo, OpenAI Evals, Hugging Face LightEval, pytest, or custom harness configuration. The central platform should collect one canonical result row with source-bundle, trace, model, harness, retrieval-index, tool-catalog, judge, score, failure-code, and review metadata.
Operating Model
- Repo-local packs hold cases, fixtures, rubrics, source manifests, deterministic checks, and native runner configs.
- GitLab runners execute evals near code, secrets, retrieval indexes, and tool schemas.
- Team collectors normalize raw exports into
eval_result_v1rows. - Firm collectors aggregate rows into scorecards, governance gates, model regressions, retrieval-index drift, and knowledge-base promotion queues.
- Knowledge owners promote only reviewed and redacted failures into wiki updates, KB patches, SFT rows, preference pairs, or policy gates.
Failure Domains
The failure taxonomy should classify the point of failure, not just the bad final answer:
| Domain | Examples |
|---|---|
| Intake | ambiguous task, wrong persona, missing authority |
| Retrieval | wrong source, stale source, ACL leak, missing citation |
| Generation | hallucination, refusal error, format drift |
| Tooling | wrong tool, invalid arguments, failed recovery, permission bypass |
| Trajectory | goal drift, loop failure, bad handoff, compounding stochastic error |
| Judge | criteria drift, judge bias, invalid validator, label error |
| Benchmark | contamination, saturated benchmark, leaderboard overfit, wrong ground truth |
| Training promotion | unreviewed trace, secret leakage, missing lineage |
Source Evidence
The June 2026 Awesome Evals ingestion added 318 citation rows, 318 golden-quality exemplar rows, 269 raw mirrors, 143 mirrored BenchFlow notes, 83 Chandra-extracted PDF rows, including 2 chunked extractions, and 46 media rows that were attempted through the transcript path. Public captions were unavailable for those media rows, so they remain marked needs_asr for the podcast/video ASR pipeline.
Important boundary: queued or ASR-needed media is not the same as full local transcription. A media row should remain needs_asr until the podcast/video pipeline attaches transcript evidence. Likewise, a downloaded PDF is not a Chandra extraction; Chandra completion should be tracked separately in the PDF/Chandra queue.
The AI Engineer selected-session ingestion on 2026-07-02 adds a practitioner operating pattern for agent evals: evals should increasingly be built from production or UAT traces, not only synthetic static test cases. Braintrust’s eval-maturity talk frames the loop as capture production traces, annotate failure modes with human justifications, turn those justifications into rubrics or LLM-judge criteria, evaluate the judge against human decisions, and rerun production-like traces offline before promotion. Cline’s eval talk reinforces the same boundary from a benchmark angle: use evals, but treat public/static eval scores as directional and allocate failures by trace rather than only by final score.
Source: research/13-multimodal-sources/ai-engineer/selected-sessions-agent-operations-2026.md · Raw ledger: sources/13-multimodal-sources/ai-engineer-worlds-fair-2026-selected-sessions-raw.md
The full-channel AI Engineer caption pass expands this eval-operations source pool to 801 captured transcripts. The useful addition is breadth: evaluation topics now sit beside RAG production, agent observability, coding-agent benchmarks, voice-agent reliability, and model-infrastructure talks. That makes the corpus useful for finding eval cases that cross failure domains rather than staying inside static benchmark categories. The two uncaptured videos are explicitly recorded in the full ledger; they should not be treated as available transcript evidence until retried or transcribed by another path.
Source: research/13-multimodal-sources/ai-engineer/channel-caption-ingestion-index-2026.md · Full ledger: sources/13-multimodal-sources/ai-engineer-channel-full-captions-2026-raw.md
July 2026 Update: Trace-To-Skill Promotion
The Ramp/Snorkel/Red Hat/NVIDIA source pass adds one operating principle: evals should decide not only whether a model passes, but whether a failure becomes a governed improvement.
New eval contract:
- Task — the precise workflow, persona, source bundle, tools, policy boundary, and expected artifact.
- Trace — retrieval, tool calls, observations, state changes, approvals, retries, and deviations.
- Outcome — the business or system state change, including correct abstention or escalation.
- Rubric — SME-approved criteria, deterministic validators, judge version, and acceptable failure boundaries.
- Promotion — whether the failure becomes a prompt patch, retrieval fix, training row, or skill candidate.
The skill-promotion gate is the new piece. A trace-derived skill should carry a failure signature, when-to-use trigger, repair strategy, source provenance, required tools/permissions, negative examples, deterministic checks or SME rubric, rollback guidance, and regression evidence. Raw traces are evidence, not instructions.
Source: Corporate Evals, Trace Mining, and Automatic Skill Improvement · Raw ledger: sources/21-benchmarks/corporate-evals-trace-skill-improvement-2026-raw.md
See Also
July 2026 Update: Self-Improving Agent Evaluation
The survey Self-Improvements in Modern Agentic Systems adds a critical distinction to the evaluation registry: record both what changed and what signal justified the change. An agent can improve its foundation-model parameters or its scaffold (prompts, memory, tools, or control logic). These are different deployment objects with different rollback, leakage, and regression risks.
Every self-improvement result should therefore record the before/after agent versions, update target, learning signal, optimization and held-out splits, improvement budget, evaluator version, transfer score, regression rate, safety violations, human interventions, and rollback artifact. A score increase without those fields is not an enterprise improvement claim.
The survey also makes evaluator overfitting a first-class failure domain. When a judge participates in the improvement loop, use a separately versioned judge and hidden or temporally shifted tasks. Keep memory and skill changes in the agent version even when model weights do not change.
Source: Self-Improving Agent Systems: Evaluation and Governance Survey · primary paper · source ledger
- Enterprise AI Evals Field Guide
- Corporate Evals, Trace Mining, and Automatic Skill Improvement
- Enterprise Evaluation for Knowledge Bases and Coding Agents
- Eval Framework Schema Crosswalk
- Frontier Lab Evaluation Research and SDK Primitive Crosswalk
- Decentralized GitLab Eval Federation
- AI Engineer Selected Sessions: Agent Operations Signals
- AI Engineer Channel Caption Ingestion Index
- AI Model Evaluation and Benchmarks