Source files: sources/21-benchmarks/eval-frameworks-mlflow-helm-giskard-deepchecks-uptrain-strands-2026-raw.md.
Source status: primary documentation and research ledger, retrieved 2026-07-23.
These frameworks were previously present at uneven depth. Giskard, Deepchecks, UpTrain, and Strands were mostly source-corpus mentions; HELM, lm-evaluation-harness, MLflow, and Hugging Face Evaluate had broader adjacent coverage. This ingestion adds dedicated primary-source capability and deployment-boundary records. Strands is the largest new addition because its Evals SDK and Eval SOP directly address test-data generation, trajectory evaluation, simulation, and failure diagnosis.
Framework crosswalk
| Framework | What it owns | Input/output shape | GitLab Docker fit | Main limitation |
|---|---|---|---|---|
| MLflow GenAI | trace-to-dataset, scorers, experiments, monitoring | traces or inputs/outputs/expectations, EvaluationResult |
sidecar MLflow Tracking Server with SQL backend; runner emits native export and IDs | not a sandbox or agent harness; server dependency |
| HF Evaluate | metrics/evaluators and Hub publication | metric module plus model/dataset/predictions/results | fully local Python container; optionally publish to Hub | weak agent trajectory/environment semantics |
| HELM | standardized benchmark scenarios/adaptations/metrics | model adapter + scenario + metrics + raw generations/results | containerized runner with cached datasets/models; good scheduled baseline job | broad benchmark comparison, not private repo task generation |
| lm-evaluation-harness | YAML task configs, model adapters, samples/results | lm-eval run, validate, YAML task definitions, result files |
excellent local Docker/CI fit; pin commit and task YAML | primarily model-level evaluation, not arbitrary agent state |
| Giskard | LLM tests, policies, ground truths, red-team cases | test cases, reports, tags, judge configuration | local OSS Python job; managed features require service access | product/managed boundary and judge behavior need pinning |
| Deepchecks | data/RAG quality, root-cause analysis, monitoring | datasets, retrieval/response signals, reports | local Python job; use as pre-eval data gate and RAG lane | not a complete agent runner |
| UpTrain | preconfigured and custom LLM metrics | question, context, response, metric results |
local Python job with optional dashboard | mainly response/RAG metrics; limited environment semantics |
| Strands Evals | cases, experiments, trajectory/tool/interaction evaluators, simulation, red team | JSON-serializable experiments/reports, OTel sessions | natural fit for Strands harness in Docker; simulated tools avoid production writes | Strands-centered APIs; LLM judge and simulator require calibration |
Harness mapping
| Harness | Recommended framework path | Runner contract |
|---|---|---|
| Codex CLI | Harbor task adapter or generic repo-local runner; Strands Evals when target is a Strands agent | invoke CLI in isolated container, capture stdout/stderr, tool trace, filesystem diff, exit status, and model/harness versions |
| Google Cloud Code | Treat Cloud Code as IDE/developer entrypoint; execute the same Docker image and GitLab job contract in CI | Cloud Code should select/build/debug the image, while GitLab remains the authoritative repeatable runner |
| Strands SDK | Strands Evals Experiment, simulators, OTel traces, custom evaluators |
execute agent inside container, emit Strands report plus normalized eval_result_v1 |
| LangGraph/Claude Code | LangChain eval-engineering + Harbor | one capability per Harbor task, hidden verifier truth, trajectory audit |
GitLab Docker implementation
Each repository owns a pinned eval image and pack:
evals/
Dockerfile
requirements-evals.lock
manifest.yaml
cases/
runners/run.sh
verifiers/
reports/
The GitLab pipeline stages should be:
validate_pack: validate YAML/JSON, task definitions, image digest, dataset/source hashes, evaluator versions, and required metadata.smoke: run 1-5 cases with network allowlists and masked credentials.full: fan out cases/models/harnesses using GitLab parallel jobs or child pipelines.audit: run positive/negative verifier controls, simulator checks, and reward-hacking probes.publish: emit JUnit, JSONL normalized rows, native framework export, Markdown summary, and immutable artifact hashes.rollup: team collector validates schema and forwards only approved metadata/artifact references to firm storage.
Keep raw prompts, private source bundles, hidden answers, tool credentials, and raw traces in protected repo artifacts or an approved team store. The firm rollup should contain repo, commit, pack, task, persona, failure_domain, harness, model, framework, framework_version, dataset/index version, judge, score, verdict, infrastructure_error, and artifact hashes.
Recommended composition
- Use lm-evaluation-harness or HELM for stable model baselines.
- Use MLflow as an optional trace/dataset/experiment system of record when a SQL-backed tracking server is acceptable.
- Use Giskard, Deepchecks, and UpTrain as specialized data, RAG, policy, and response-quality lanes.
- Use Strands Evals for Strands-native trajectory, tool, simulation, and red-team evaluation.
- Use Harbor for coding-agent and environment-dependent tasks.
- Normalize every framework into the existing result contract; do not force one native schema to represent all evaluator evidence.
Evidence Boundaries
Cloud Code is treated here as a development and container orchestration entrypoint, not as an independent evaluation framework. Codex CLI and Strands SDK are harnesses/targets, while the evaluator framework determines scoring and evidence. Managed cloud dashboards may require network access and accounts; local Docker runs should be the conformance baseline.