AI evaluation operations is the enterprise practice of turning model, RAG, coding-agent, and workflow failures into auditable evidence that can run in product repos and roll up to team and firm-wide scorecards.
The core rule: centralize normalized results, not private fixtures or native runner formats. Each repo should keep its own DeepEval, RAGAS, LangSmith, Phoenix, Promptfoo, OpenAI Evals, Hugging Face LightEval, pytest, or custom harness configuration. The central platform should collect one canonical result row with source-bundle, trace, model, harness, retrieval-index, tool-catalog, judge, score, failure-code, and review metadata.
Operating Model
- Repo-local packs hold cases, fixtures, rubrics, source manifests, deterministic checks, and native runner configs.
- GitLab runners execute evals near code, secrets, retrieval indexes, and tool schemas.
- Team collectors normalize raw exports into
eval_result_v1rows. - Firm collectors aggregate rows into scorecards, governance gates, model regressions, retrieval-index drift, and knowledge-base promotion queues.
- Knowledge owners promote only reviewed and redacted failures into wiki updates, KB patches, SFT rows, preference pairs, or policy gates.
Failure Domains
The failure taxonomy should classify the point of failure, not just the bad final answer:
| Domain | Examples |
|---|---|
| Intake | ambiguous task, wrong persona, missing authority |
| Retrieval | wrong source, stale source, ACL leak, missing citation |
| Generation | hallucination, refusal error, format drift |
| Tooling | wrong tool, invalid arguments, failed recovery, permission bypass |
| Trajectory | goal drift, loop failure, bad handoff, compounding stochastic error |
| Judge | criteria drift, judge bias, invalid validator, label error |
| Benchmark | contamination, saturated benchmark, leaderboard overfit, wrong ground truth |
| Training promotion | unreviewed trace, secret leakage, missing lineage |
Source Evidence
The June 2026 Awesome Evals ingestion added 318 citation rows, 318 golden-quality exemplar rows, 269 raw mirrors, 143 mirrored BenchFlow notes, 83 Chandra-extracted PDF rows, including 2 chunked extractions, and 46 media rows that were attempted through the transcript path. Public captions were unavailable for those media rows, so they remain marked needs_asr for the podcast/video ASR pipeline.
Important boundary: queued or ASR-needed media is not the same as full local transcription. A media row should remain needs_asr until the podcast/video pipeline attaches transcript evidence. Likewise, a downloaded PDF is not a Chandra extraction; Chandra completion should be tracked separately in the PDF/Chandra queue.
The AI Engineer selected-session ingestion on 2026-07-02 adds a practitioner operating pattern for agent evals: evals should increasingly be built from production or UAT traces, not only synthetic static test cases. Braintrust’s eval-maturity talk frames the loop as capture production traces, annotate failure modes with human justifications, turn those justifications into rubrics or LLM-judge criteria, evaluate the judge against human decisions, and rerun production-like traces offline before promotion. Cline’s eval talk reinforces the same boundary from a benchmark angle: use evals, but treat public/static eval scores as directional and allocate failures by trace rather than only by final score.
Source: research/13-multimodal-sources/ai-engineer/selected-sessions-agent-operations-2026.md · Raw ledger: sources/13-multimodal-sources/ai-engineer-worlds-fair-2026-selected-sessions-raw.md
The full-channel AI Engineer caption pass expands this eval-operations source pool to 801 captured transcripts. The useful addition is breadth: evaluation topics now sit beside RAG production, agent observability, coding-agent benchmarks, voice-agent reliability, and model-infrastructure talks. That makes the corpus useful for finding eval cases that cross failure domains rather than staying inside static benchmark categories. The two uncaptured videos are explicitly recorded in the full ledger; they should not be treated as available transcript evidence until retried or transcribed by another path.
Source: research/13-multimodal-sources/ai-engineer/channel-caption-ingestion-index-2026.md · Full ledger: sources/13-multimodal-sources/ai-engineer-channel-full-captions-2026-raw.md
July 2026 Update: Trace-To-Skill Promotion
The Ramp/Snorkel/Red Hat/NVIDIA source pass adds one operating principle: evals should decide not only whether a model passes, but whether a failure becomes a governed improvement.
New eval contract:
- Task — the precise workflow, persona, source bundle, tools, policy boundary, and expected artifact.
- Trace — retrieval, tool calls, observations, state changes, approvals, retries, and deviations.
- Outcome — the business or system state change, including correct abstention or escalation.
- Rubric — SME-approved criteria, deterministic validators, judge version, and acceptable failure boundaries.
- Promotion — whether the failure becomes a prompt patch, retrieval fix, training row, or skill candidate.
The skill-promotion gate is the new piece. A trace-derived skill should carry a failure signature, when-to-use trigger, repair strategy, source provenance, required tools/permissions, negative examples, deterministic checks or SME rubric, rollback guidance, and regression evidence. Raw traces are evidence, not instructions.
Source: Corporate Evals, Trace Mining, and Automatic Skill Improvement · Raw ledger: sources/21-benchmarks/corporate-evals-trace-skill-improvement-2026-raw.md
July 2026 Update: Harbor Eval Engineering Skill
LangChain’s current eval-engineering skill makes eval creation a gated engineering loop: inspect the agent read-only, use selected traces only as discovery evidence, propose capabilities, obtain approval for the runtime and environment, build one Harbor task, test the verifier with positive and negative controls, run it, and audit both target and verifier trajectories. The unit is evals/<task-id>/ with task.toml, instruction.md, environment/, and tests/.
Generated tasks are drafts until the capability, runtime, environment, oracle, verifier, and review status are recorded. Harbor-native artifacts remain repo-local; central federation receives normalized result rows and immutable artifact references. Build, credential, reset, timeout, judge, and verifier failures are infrastructure failures, not model-quality failures.
Source: LangChain Eval Engineering Skill on Harbor · raw primary mirror
July 2026 Update: Competing Eval Authoring Paths
The authoring landscape now separates four layers: Harbor/LangChain creates executable agent environments; Google ADK and Copilot Studio generate or simulate agent scenarios; Pydantic Evals validates typed cases, outputs, tools, and artifacts; Weave, Braintrust, and LangSmith turn traces and feedback into versioned datasets and scorers. DeepEval and Ragas remain strongest for synthetic response/RAG coverage.
Pydantic belongs inside the verifier and adapter boundary. It should validate structured output, tool arguments, state transitions, and artifact metadata before semantic scoring. It does not replace sandbox isolation, trace storage, or a task runner.
Source: Competing Eval-Authoring Frameworks · primary ledger
July 2026 Update: Frameworks in Local GitLab Containers
The framework layer should be selected by evidence type: HELM and lm-evaluation-harness for model baselines; MLflow for trace-to-dataset and experiment tracking; Giskard, Deepchecks, and UpTrain for policy, RAG, data, and response-quality checks; Strands Evals for Strands-native trajectory, tool, simulation, and red-team evaluation; Harbor for environment-dependent coding-agent tasks.
In GitLab, each repo pins an eval Docker image, dependency lock, dataset/source hashes, evaluator versions, and harness adapter. validate_pack checks the contract, smoke runs a small subset, full fans out models/cases, audit runs positive/negative verifier controls, and publish emits JUnit, normalized JSONL, native exports, summaries, and artifact hashes. Cloud Code can build/debug the same image, but GitLab remains the authoritative repeatable runner. Codex CLI and Strands SDK are target harnesses; they do not replace the evaluator or result contract.
Source: Eval Frameworks for GitLab Containers and Agent Harnesses · primary ledger
See Also
July 2026 Update: Self-Improving Agent Evaluation
The survey Self-Improvements in Modern Agentic Systems adds a critical distinction to the evaluation registry: record both what changed and what signal justified the change. An agent can improve its foundation-model parameters or its scaffold (prompts, memory, tools, or control logic). These are different deployment objects with different rollback, leakage, and regression risks.
Every self-improvement result should therefore record the before/after agent versions, update target, learning signal, optimization and held-out splits, improvement budget, evaluator version, transfer score, regression rate, safety violations, human interventions, and rollback artifact. A score increase without those fields is not an enterprise improvement claim.
The survey also makes evaluator overfitting a first-class failure domain. When a judge participates in the improvement loop, use a separately versioned judge and hidden or temporally shifted tasks. Keep memory and skill changes in the agent version even when model weights do not change.
Source: Self-Improving Agent Systems: Evaluation and Governance Survey · primary paper · source ledger
- Enterprise AI Evals Field Guide
- Corporate Evals, Trace Mining, and Automatic Skill Improvement
- Enterprise Evaluation for Knowledge Bases and Coding Agents
- Eval Framework Schema Crosswalk
- Frontier Lab Evaluation Research and SDK Primitive Crosswalk
- Decentralized GitLab Eval Federation
- AI Engineer Selected Sessions: Agent Operations Signals
- AI Engineer Channel Caption Ingestion Index
- AI Model Evaluation and Benchmarks