Decentralized GitLab Eval Federation — 2026

Source files: sources/21-benchmarks/gitlab-decentralized-eval-federation-2026-raw.md · sources/21-benchmarks/model-routing-meaningfulness-hse-robustness-raw.md · research/21-benchmarks/enterprise-evaluation-knowledge-coding-agents-2026.md · statebench/ci/gitlab-enterprise-eval-component.yml · statebench/runners/enterprise_eval_contract_smoke.py · statebench/runners/contract_validator.py · docs/research-synthesis-verification-loop.md · .claude/skills/synthesis-research-verification/SKILL.md

See also: Enterprise Evaluation for Knowledge Bases and Coding Agents · Agent Observability Format Standard · Enterprise Desktop Agent Knowledge Graphs

Source status: This is an architecture pattern grounded in GitLab’s documented CI/CD runner, artifacts, downstream-pipeline, and CI/CD component capabilities. It is not a GitLab product claim. The recommendation is to keep eval execution decentralized in each repo and centralize only normalized eval facts, scorecards, and curated knowledge.

Adjacent benchmark evidence: EnterpriseClawBench independently validates the same privacy boundary: derive eval packets from private workplace sessions, keep sensitive task data local, and publish construction protocols, run-directory schemas, aggregate scorecards, and redacted exemplars rather than raw enterprise content.

Executive Answer

Run evals where the work happens: inside each repository’s GitLab pipeline, on runners controlled by that team or platform group. Do not centralize source code, prompts, proprietary fixtures, or secrets into one eval monorepo. Centralize only the normalized output:

repo-local eval pack
  -> GitLab runner job
  -> local artifacts: JUnit + JSONL + annotations + trace bundle
  -> team collector
  -> firm eval warehouse
  -> dashboards, release gates, and knowledge-base updates

This is the right shape because AI evals are contextual. A support repo, a coding-agent repo, a RAG service, a data-agent repo, and a research-synthesis repo need different fixtures, secrets, source bundles, and deterministic validators. The common layer is the result contract, not the runner implementation.

Why Decentralized

Central eval platforms fail when they require every team to move local reality into a central harness. The brittle parts of enterprise evals are repo-local:

• build/test commands;
• source bundles and ACL snapshots;
• private prompts and policies;
• tool catalogs and MCP servers;
• domain fixtures;
• human review rubrics;
• hidden regression cases;
• model/runtime versions;
• secrets and network boundaries.

The centralized layer should receive evidence, not own execution.

GitLab Architecture

Repo-Local Contract

Every participating repo should contain:

evals/
  packs/
    corp_kb_policy_qa_v1/
      manifest.yaml
      cases.jsonl
      source_bundle_manifest.yaml
      rubric.md
      validators/
  runners/
    run_eval.sh
  reports/
    .gitkeep

Minimum manifest.yaml:

pack_id: corp_kb_policy_qa_v1
pack_version: 1.3.0
owner_team: hr-platform
repo_path: group/hr-assistant
workflow_family: knowledge_base_qa
persona: hr_business_partner
risk_tier: high
review_date: 2026-06-21
source_bundle_manifest: source_bundle_manifest.yaml
cases: cases.jsonl
runner:
  command: ./evals/runners/run_eval.sh
  expected_outputs:
    - reports/eval-results.jsonl
    - reports/junit.xml
    - reports/eval-summary.md
gates:
  permission_leak: 0
  citation_bad_span_rate_max: 0.02
  stale_fact_rate_max: 0.01
  unsupported_claim_rate_max: 0.03
context_required:
  - harness_name
  - harness_version
  - model_provider
  - model_name
  - model_version
  - prompt_template_version
  - retrieval_index_version
  - tool_catalog_version
  - judge_model_version

Normalized Result Contract

Each eval attempt should emit one JSONL row per task attempt:

{
  "schema_version": "eval_result_v1",
  "result_id": "res_policy_017_claude_001",
  "run_id": "run_hr_policy_20260622_001",
  "team_id": "hr-platform",
  "repo_path": "group/hr-assistant",
  "commit_sha": "abc123",
  "pipeline_id": "123456",
  "job_id": "987654",
  "pack_id": "corp_kb_policy_qa_v1",
  "pack_version": "1.3.0",
  "task_id": "policy_017",
  "attempt_id": "attempt_policy_017_001",
  "persona": "hr_business_partner",
  "workflow_family": "knowledge_base_qa",
  "risk_tier": "high",
  "harness": {
    "name": "deepeval",
    "version": "4.0",
    "runner_kind": "framework_sdk",
    "native_export_ref": "reports/deepeval-results.json",
    "native_export_hash": "sha256:deepeval"
  },
  "model": {
    "provider": "anthropic",
    "name": "claude-sonnet",
    "version": "vendor_snapshot_or_release",
    "weights_revision": null,
    "adapter_id": null,
    "serving_stack": "api",
    "tokenizer": null,
    "chat_template": null
  },
  "runtime_context": {
    "prompt_template_version": "hr_policy_qa_v3",
    "retrieval_index_version": "confluence_hr_2026_06_21",
    "tool_catalog_version": "mcp_tools_2026_06_21",
    "judge_model_version": "gpt-4o",
    "execution_environment": "gitlab-runner:python3.13:macos",
    "dependency_lock_ref": "requirements-evals.lock",
    "dependency_lock_hash": "sha256:requirements",
    "container_image_ref": null,
    "container_image_digest": null
  },
  "attached_tools": [
    {
      "tool_id": "retriever_confluence_hr",
      "tool_type": "retriever",
      "name": "confluence_hr_vector_search",
      "version": "2026-06-21",
      "source": "internal_mcp_registry",
      "schema_ref": "schemas/tools/confluence_search.json",
      "schema_hash": "sha256:confluence_tool_schema",
      "permission_profile": "hr_policy_readonly",
      "network_access": "allowlist",
      "data_scope": "hr-policy-space"
    },
    {
      "tool_id": "mcp_policy_citations",
      "tool_type": "mcp_server",
      "name": "policy-citation-mcp",
      "version": "0.8.2",
      "source": "internal_mcp_registry",
      "schema_ref": "schemas/mcp/policy-citation-tools.json",
      "schema_hash": "sha256:policy_mcp_schema",
      "permission_profile": "citation_readonly",
      "network_access": "workspace",
      "data_scope": "source-bundle:hr_policy_bundle_2026_06_21"
    }
  ],
  "trace_ref": "trace_hr_policy_017",
  "source_bundle_id": "hr_policy_bundle_2026_06_21",
  "scores": {
    "faithfulness": {
      "score": 0.91,
      "threshold": 0.85,
      "passed": true,
      "metric_type": "llm_judge",
      "judge_model": "gpt-4o",
      "rubric_version": "grounding_v3",
      "reason_ref": "reports/reasons/policy_017.md"
    }
  },
  "failures": [
    {
      "code": "knowledge.stale_fact",
      "domain": "knowledge_retrieval",
      "severity": "release_blocking",
      "owner_role": "knowledge_owner",
      "evidence_ref": "hr_policy_bundle_2026_06_21",
      "remediation_status": "open"
    }
  ],
  "artifact_refs": {
    "summary_md": "reports/eval-summary.md",
    "trace_bundle": "reports/trace-bundle.jsonl"
  },
  "created_at_utc": "2026-06-22T10:05:00Z"
}

This row is the federation boundary. Teams can use LangSmith, DeepEval, RAGAS, OpenAI Evals, pytest, custom scripts, or human review internally as long as they emit this contract.

Contract Validation

The first-pass schemas now live under statebench/schemas/. Repo-local eval jobs should run the contract smoke command before upload:

scripts/.venv-tts/bin/python3 -m statebench.runners.enterprise_eval_contract_smoke

That command validates passing case, result, GitLab annotation, native fixture, team rollup, and firm rollup artifacts. It also verifies that eval-results.fail.jsonl and native-fixtures.fail.jsonl fail as expected. The native negative fixture intentionally carries a bad real_run_export hash, so the smoke suite demonstrates that real-run payload hashes are recomputed instead of accepted as decorative strings.

The lower-level entrypoint remains statebench/runners/contract_validator.py for individual files. The smoke command is the shared GitLab gate; the single-file validator is the debug and adapter-development tool.

Rollups are generated, not hand-maintained. Once a repo-local runner emits eval-results.jsonl, the collector can create team and firm scorecards:

scripts/.venv-tts/bin/python3 -m statebench.runners.eval_rollup_collector \
  --eval-results reports/evals/eval-results.jsonl \
  --team-id "$TEAM_ID" \
  --team-rollup-id "team_${CI_PROJECT_ID}_${CI_PIPELINE_ID}" \
  --firm-rollup-id "firm_${CI_PIPELINE_ID}" \
  --window-start-utc "$CI_PIPELINE_CREATED_AT" \
  --window-end-utc "$CI_JOB_STARTED_AT" \
  --created-at-utc "$CI_JOB_STARTED_AT" \
  --team-output reports/evals/team-rollup.json \
  --firm-output reports/evals/firm-rollup.json

GitLab CI Pattern

Repos should include a pinned shared component once the component is published. The repo artifact statebench/ci/gitlab-enterprise-eval-component.yml is the first-pass template to promote into that CI component project:

include:
  - component: $CI_SERVER_FQDN/ai-platform/eval-components/repo-eval@1.0.0
    inputs:
      stage: test
      pack_glob: "evals/packs/*/manifest.yaml"
      output_dir: "reports/evals"
      fail_on_release_blocking: "true"
      publish_annotations: "true"

The component expands to jobs like:

repo_ai_eval:
  stage: test
  image: registry.example.com/ai-platform/eval-runner:1.0.0
  script:
    - ./evals/runners/run_eval.sh --pack-glob "$PACK_GLOB" --output-dir reports/evals
    - scripts/.venv-tts/bin/python3 -m statebench.runners.enterprise_eval_contract_smoke
    - scripts/.venv-tts/bin/python3 -m statebench.runners.contract_validator --schema eval_result_v1 --input reports/evals/eval-results.jsonl --jsonl
    - scripts/.venv-tts/bin/python3 -m statebench.runners.contract_validator --schema gitlab_eval_annotation_v1 --input reports/evals/annotations.json
  artifacts:
    when: always
    paths:
      - reports/evals/
    reports:
      junit: reports/evals/junit.xml
      annotations: reports/evals/annotations.json

Use GitLab JUnit for merge-request visibility. Use JSONL for warehouse ingestion. Use annotations to link the job to team dashboards, trace viewers, or longer HTML reports. The component validator statebench/runners/validate_gitlab_eval_component.py checks that the template still emits JUnit and annotations and still runs the StateBench contract gates.

Bubbling Up

Repo Level

The repo decides whether a merge request can pass. Gates should be local and strict:

• zero permission leaks;
• zero unsafe tool calls;
• no release-blocking coding-agent failures;
• citation and stale-source thresholds;
• required trace coverage;
• routing stability checks for repos using AI gateways, model routers, or multi-agent dispatch.

Team Level

The team collector reads artifacts from all repos in the team group and produces:

• pack coverage by workflow and persona;
• latest pass/fail by repo and branch;
• top failure families;
• stale eval packs;
• judge disagreement rates;
• cost and latency trends;
• new incident-derived eval candidates.

Firm Level

The firm collector receives normalized JSONL from team collectors or directly from GitLab APIs. It should not need repo secrets or source bundles. It aggregates:

• critical workflow coverage;
• model/vendor scorecards by workflow;
• release-blocking failure trends;
• risk-tier compliance;
• unowned knowledge failures;
• teams missing current eval packs;
• benchmark-pack drift and stale rubric versions.

Knowledge Base Level

Only curated learnings should enter the knowledge base:

• recurring failure families and fixes;
• approved benchmark-pack templates;
• source-authority rules;
• model/tool deprecation notes;
• incident-derived eval cases after redaction;
• “do not recommend” patterns, such as stale defaults or unsupported vendor claims.

Raw traces should not be dumped into the knowledge base. They should stay in trace storage or the eval warehouse with access controls.

Security And Governance

Required Warehouse Tables

Implementation Sequence

1. Use the v1 schemas. Validate golden_work_packet_v1, eval_case_v1, eval_result_v1, gitlab_eval_annotation_v1, and failure_taxonomy_v1 in CI.
2. Publish the CI component. Promote statebench/ci/gitlab-enterprise-eval-component.yml into a pinned GitLab CI/CD component project after adapting the runner image and product-repo command.
3. Pilot three repos. Pick one RAG service, one coding-agent workflow, and one synthesis/research repo.
4. Create team collector. Scheduled job reads latest artifacts and writes team_eval_rollup_v1 scorecards.
5. Create firm warehouse ingest. Load normalized JSONL and firm_eval_rollup_v1 scorecards into warehouse tables.
6. Promote learnings. Only redacted, reviewed failure learnings enter knowledge bases.
7. Govern pack changes. Require code review for benchmark-pack changes and visible diff summaries for cases/rubrics.

Bottom Line

The enterprise should standardize the eval result contract, not the eval runner. GitLab runners let every repo execute its own context-specific checks. CI/CD components distribute the shared job shape. Artifacts carry local visibility. JSONL carries federation. Team and firm collectors turn repo-local evidence into governance, scorecards, and knowledge-base updates without centralizing secrets or flattening every workflow into one generic benchmark.