← Benchmarks 🕐 2 min read
Benchmarks

Evals Skills, Harness Engineering, and Agent-Operable Evaluation

What this adds · March 2026

Source status: primary repository and primary OpenAI engineering article reviewed 2026-07-25; discovery began from an authenticated browser capture on X.

Source status: primary repository and primary OpenAI engineering article reviewed 2026-07-25; discovery began from an authenticated browser capture on X. The X post is not used as evidence where the linked primary source is available.

The existing framework crosswalk needs a separate evaluation skill layer. Frameworks expose data and execution primitives; skills encode the reasoning procedure that turns those primitives into a credible evaluation program.

Layer Responsibility Example
Harness Runs the agent or application under controlled conditions Claude Code, Codex CLI, Strands, Harbor
Instrumentation Emits traces, tool calls, logs, metrics, and artifacts OpenTelemetry, OpenInference, provider traces
Evaluation framework Defines datasets, evaluators, experiments, and result records MLflow, LangSmith, Ragas, DeepEval, HELM
Evaluation skill Guides diagnosis, judge calibration, data generation, review, and iteration eval-audit, error-analysis, validate-evaluator, evaluate-rag
Knowledge base Stores versioned task contracts, failure taxonomies, rubrics, and evidence Git repository plus indexed artifacts
Governance Controls promotion, provenance, access, and CI policy GitLab runner gates and team rollups

Audit model

The eval-audit skill’s six diagnostic areas are a useful minimum contract for our enterprise audit:

  1. Error analysis: failures are classified from traces rather than only aggregate scores.
  2. Evaluator design: criteria are explicit, scoped, and tied to the task contract.
  3. Judge validation: LLM judges are calibrated against human labels with held-out checks.
  4. Human review: ambiguous or high-impact cases have an annotation path.
  5. Labeled data: examples, labels, provenance, and splits are versioned.
  6. Pipeline hygiene: instrumentation, sampling, retries, caching, and result storage are reproducible.

Implication for the platform

An MCP server should be treated as an access adapter, not an evaluation methodology. A coding agent needs both: a server or SDK to query traces and experiments, and a domain-grounded skill that tells it which evidence to collect, how to classify failures, how to validate judges, and when a result is insufficient for promotion.

OpenAI’s harness-engineering account strengthens the repository-local model already used in this project. Agent-facing knowledge should be a small navigational map pointing to versioned schemas, task contracts, executable checks, and evidence records. Observability and evaluation artifacts must be queryable from the same isolated worktree or runner that executes the task. This supports decentralized GitLab execution while preserving firm-wide rollups.

Updated enterprise shape

Every evaluation package should carry:

  • harness: agent/runtime, version, model/provider, effort mode, tool policy, and environment digest.
  • task_contract: user persona, domain, task family, inputs, expected outputs, and prohibited actions.
  • trace_contract: event schema, tool calls, retrieval context, model calls, and timing.
  • failure_taxonomy: error class, severity, user impact, reproducibility, and owner.
  • evaluator: deterministic checks, judge rubric, human-label protocol, and calibration results.
  • dataset: source, license, split, mutation policy, and contamination controls.
  • result: score vector, confidence or uncertainty, cost, latency, artifact links, and promotion decision.

This separates a bad answer from a bad judge, a missing trace from a bad retrieval result, and a harness limitation from a model limitation.

Golden exemplar

{"schema":"eval_result_v1","eval_id":"rag-support-001","harness":{"name":"codex-cli","version":"recorded-at-run","model":"recorded-at-run","provider":"recorded-at-run","effort":"recorded-at-run","tool_policy":"read-only","environment_digest":"sha256:..."},"task_contract":{"domain":"customer-support","persona":"support-agent","task_family":"grounded-answer","input_ref":"dataset://support/v3/case-001","expected":"answer with cited policy evidence","prohibited":["inventing account actions"]},"trace_ref":"trace://run/001","failure":{"class":"unsupported_action_claim","severity":"high","impact":"false customer action","reproducible":true},"evaluator":{"deterministic":["citation_present","action_claim_supported"],"judge":"rubric://support-groundedness/v2","human_calibration_ref":"labels://support/v2/holdout"},"result":{"status":"fail","scores":{"groundedness":0.2,"action_safety":0.0},"latency_ms":1840,"cost_usd":0.012,"artifacts":["artifact://trace/001"]},"provenance":{"source":"gitlab-runner","commit":"abc123","retrieved_at":"2026-07-25T00:00:00Z"}}