Source mirror: harbor-langchain-eval-engineering-skill-2026-raw.md
Source files: sources/21-benchmarks/harbor-langchain-eval-engineering-skill-2026-raw.md.
Source status: primary public skill mirror plus official LangChain and Harbor documentation, retrieved 2026-07-22.
Primary sources: launch post, skill repository, Harbor task structure, Harbor repository.
LangChain’s July 22, 2026 eval-engineering skill turns repository context and selected production traces into reviewed, executable Harbor tasks. The important design decision is not automatic benchmark generation. It is a gated interview loop: map the agent read-only, propose capabilities, obtain user approval for one capability and its runtime/environment boundary, build one task, test the verifier, run the agent, inspect both trajectories, and revise.
Technical shape
| Layer | Required shape | Enterprise implication |
|---|---|---|
| Agent map | entrypoint, prompts, model/routing, tools, data, state, permissions, effects, evidence | capture harness, model, tool catalog, repo commit, index/data versions before scoring |
| Trace input | selected requests and dependency behavior; never assumed ground truth | traces are discovery evidence, not labels; redact and attach trace IDs |
| Scenario | one capability, realistic request, known state or source material | failure-domain labels should be attached to the capability and expected action |
| Runtime boundary | active entrypoint preferred; reconstruction only with disclosed loss | preserve production behavior unless isolation makes it impossible |
| Environment | live, frozen, or simulated dependencies with version/effects documented | freeze data and reset mutable state; classify credential and network assumptions |
| Task | task.toml, instruction.md, environment/, tests/ |
repo-local GitLab runners can emit this as a portable eval pack |
| Verifier | deterministic execution/state checks plus LLM judge for semantics | hidden truth and judge credentials stay outside target visibility |
| Evidence | target trajectory, tool/action/state record, verifier evidence, verdict, reason, reward, errors | normalize these into eval_result_v1; retain native Harbor artifacts |
| Failure boundary | build, credentials, reset, timeout, judge, verifier = infrastructure | do not turn harness failure into model-quality failure |
What this updates in our operating model
- Eval creation is an interactive engineering workflow. A generated task should carry
proposal_status,runtime_approval,environment_boundary, andreview_status; generation without those gates is draft material, not a benchmark. - Trace mining is discovery, not supervision. A trace can supply a realistic request, tool contract, error pattern, or state transition. It cannot establish the golden answer without an independently reviewed oracle.
- The verifier is a first-class evaluated component. Every new verifier needs a positive control, a realistic negative control, trajectory review, reward-hacking probes, and an infrastructure-error path.
- Harbor is a task execution contract, not a firm-wide result schema. Keep
task.tomland native artifacts with the repo-local pack; roll up only normalized result rows plus immutable artifact references. - A capability is the right unit for regression promotion. One capability per task makes failures classifiable, prevents broad prompts from hiding which behavior changed, and supports team-level aggregation across different agent frameworks.
Failure-domain mapping
| Observed issue | Classify as | Evidence needed |
|---|---|---|
| Wrong source or unsupported answer | retrieval/grounding or generation | retrieved documents, citations, answer, independent reference |
| Wrong tool or malformed arguments | tooling | tool schema, call, observation, state delta |
| Agent skips required action because fixture leaks answer | environment/task-design | environment contents, target trajectory, task revision |
| Verifier accepts irrelevant citations or claimed actions | judge/verifier | verifier evidence, exploit reproduction, corrected check |
| Docker/build/credential/reset/timeout failure | infrastructure | resolved config, logs, error class, retry/reset outcome |
| Trace-derived request has a wrong presumed answer | dataset/oracle | trace ID, source authority, human-approved reference |
Evidence Boundaries and Verification
The blog is an implementation announcement and the skill repository is early development. The public skill links to additional reference files whose exact contents are not included in this ingestion; those should be mirrored before treating the workflow as complete. The evidence supports the task shape and control boundaries, not a claim that generated tasks are valid without human review or that Harbor runs are equivalent across Docker and cloud backends.
Recommended integration
Add a harbor_eval_v1 adapter to the existing decentralized GitLab architecture with: task_id, task_version, capability, failure_domain, repo_commit, entrypoint_mode, environment_digest, data/index versions, agent/model/harness/tool versions, trace_refs, verifier_version, judge_model, reward, verdict, infrastructure_error, artifact_refs, and review_status. Do not centralize private fixtures, credentials, or raw traces. Promote a task only after the positive/negative verifier controls and trajectory audit pass.