Source files: sources/21-benchmarks/enterprise-custom-eval-tooling-research-2026-raw.md · sources/21-benchmarks/llm-evaluation-compendium-2026-raw.md · sources/21-benchmarks/eval-framework-native-schema-primary-sources-2026.md · sources/21-benchmarks/eval-native-schema-boundaries-2026.json · sources/21-benchmarks/awesome-evals-primary-sources-2026.md · sources/21-benchmarks/awesome-evals-golden-exemplars-2026.jsonl · statebench/schemas/native_eval_fixture_v1.schema.json · statebench/fixtures/enterprise-eval-contract/v1/native-fixtures.pass.jsonl · research/21-benchmarks/enterprise-evaluation-knowledge-coding-agents-2026.md · research/21-benchmarks/frontier-lab-eval-sdk-crosswalk-2026.md · research/21-benchmarks/enterprise-ai-evals-field-guide-2026.md · sources/21-benchmarks/frontier-lab-eval-type-matrix-2026.json · Hugging Face Evaluate, LightEval, Dataset Card, and TRL SFT documentation retrieved June 21-22, 2026.
Production sibling pages: Enterprise Evaluation for Knowledge Bases and Coding Agents · Frontier Lab Evaluation Research and SDK Primitive Crosswalk · Decentralized GitLab Eval Federation
Production machine-readable artifacts: adapter boundary JSON · typed source-gap backlog JSONL · Awesome Evals source ledger · Awesome Evals golden exemplars · native fixture schema · eval result schema · source-gap schema · native fixture rows · GitLab eval component · contract smoke runner
Source status: This is a schema and operating-model crosswalk, not a claim that each framework exposes a stable public JSON schema for all internals. Where a project has a native format, the table names the observed record shape. Where a project is a runtime or dashboard, the table maps the data it manages into the enterprise canonical schema. Framework versions and hosted SaaS export formats can drift; runners should store raw exports alongside normalized rows.
Answer
We know the conceptual schema for the major frameworks, but we should not treat vendor-native formats as the durable enterprise schema.
The overlap is strong:
- every framework needs a case/task input;
- most need an output/response;
- RAG frameworks add retrieved contexts/chunks;
- judge frameworks add rubric, judge model, score, and rationale;
- agent frameworks add tool calls, trajectory spans, handoffs, and approvals;
- training/SFT systems add prompt/completion or messages, dataset split, tokenizer/chat template, and model lineage;
- observability systems add trace IDs, span events, latency, token/cost, and metadata.
The practical answer is to keep three layers:
- Native framework files in each repo so engineers can run DeepEval, RAGAS, OpenAI Evals, LightEval, Promptfoo, pytest, or provider SDK evals locally.
- Canonical corporate benchmark packs in a shared format: golden work packet YAML/JSON plus JSONL cases and source bundle manifests.
- Central warehouse rows using
eval_result_v1, with raw vendor exports retained as evidence.
Framework Schema Crosswalk
| Framework / surface | Native schema grain | Typical native fields | Best eval domain | Maps into canonical fields | Boundary |
|---|---|---|---|---|---|
| DeepEval | test case plus metrics | input, actual output, expected output, context, retrieval context, tools, metric config, threshold, score, reason | repo-native CI, RAG, agents, multi-turn, safety, custom metrics | task, response, expected artifact, source bundle, metric, judge, score, failure code | Good test runner; not the only warehouse |
| RAGAS | evaluation sample/dataset plus metric result | question/user input, answer/response, contexts, reference/ground truth where available, retrieved contexts, rubric/custom metric, score | RAG and agentic RAG | retrieval evidence, context precision/recall, faithfulness, answer relevance, tool/goal metrics | RAG faithfulness is not source authority or permission safety |
| LangSmith | dataset example plus experiment/run trace | dataset, example input/output, reference output, evaluator, run ID, trace, feedback, annotation queue | trace-to-dataset loop, online/offline app evals | task, attempt, trace_ref, human review, evaluator result, production feedback | Strong workflow UI; export normalized rows for durability |
| OpenAI Evals / Evals API | eval definition plus JSONL/input rows and graders | eval/task definition, data source, test inputs, testing criteria/graders, model under test, run result | model/prompt regression, grader-based task evals | suite, task, model, grader, score, metadata, artifact refs | Not a complete trace warehouse by itself |
| Phoenix / Arize | trace/dataset/eval record | spans, prompt, response, retrieved docs, annotations, evaluator outputs, experiment/session metadata | observability, RAG debugging, LLM eval inspection | trace root, spans, evidence, scores, human labels | Needs deterministic test gates outside UI |
| Hugging Face Evaluate | metric/comparison/measurement package | predictions, references, metric/comparison/measurement module, compute result | model/dataset metric computation | metric_result, model/dataset IDs, split, score, config | General ML metric substrate; exact suite/evaluator output schemas still need capture |
| Hugging Face LightEval | benchmark task/run/sample runner | task, model/backend, prompt/generation, metric, output bundle | open/local model benchmark runs | model candidate, benchmark task, backend, sample score, run artifact | Great for model eval; exact task/output bundle schemas still need capture |
| Hugging Face Hub model card | README/YAML metadata plus reported evals | model metadata, intended use, limitations, datasets, metrics, eval results metadata | model-card benchmark disclosure | candidate registry, reported benchmark evidence, caveats | Author-reported scores are not deployment evidence |
| Hugging Face dataset card | README/YAML metadata | language, license, task categories, tags, data files, dataset description, bias/limitations | dataset/source-bundle governance | dataset/source metadata, license, task family, provenance | Metadata helps discovery, not sufficient for internal ACLs |
| Hugging Face TRL SFTTrainer | training dataset rows | text, messages, prompt, completion; standard or conversational format; chat template | SFT and post-training data | training_example, source_eval_case_ref, model lineage, chat template, split | SFT data is not an eval result; it must be lineage-linked to source eval failures |
| Promptfoo / OpenEvals-style suites | config suite plus assertions | prompts, providers, vars/cases, assertions, scorers, thresholds | prompt regression and provider comparison | suite, provider/model, test case, assertion, score | Good adapter into CI; needs trace/source evidence for agents |
| lm-evaluation-harness / BigCode harness | task config plus model outputs | task, dataset split, prompt construction, metric, model output, aggregate score | public model benchmarks | candidate model benchmark result, task, score, config | Public benchmark calibration only |
| Provider agent SDKs: OpenAI, Anthropic, Google ADK, Microsoft AutoGen, AWS Bedrock, Mistral, Qwen-Agent | run/trace/tool trajectory | model call, tool call, handoff, retrieval, approval, span, final response, cost/latency | agent trajectory and tool-use evals | attempt, trace, spans, tool calls, final answer, failure domain | Runtime trace shape differs; normalize after capture |
Canonical Overlap Fields
These are the fields every framework can map into, even if it names them differently:
| Canonical field | DeepEval | RAGAS | LangSmith | OpenAI Evals | Phoenix | Hugging Face / LightEval | Provider SDKs |
|---|---|---|---|---|---|---|---|
suite_id |
test module/suite | evaluation dataset | dataset/project | eval ID | project/session | task bundle | app/eval pack |
task_id |
test case name | sample ID | example ID | input row ID | span/run metadata | sample/doc ID | request/run ID |
input |
input | user input/question | run input | prompt/test input | prompt/input | prompt/doc | user message/task |
expected |
expected output | reference/ground truth | reference output | grader target/reference | annotation/label | reference | expected artifact/check |
actual |
actual output | response/answer | run output | model output | response | generation | final answer/action result |
contexts |
context/retrieval context | contexts | retrieved docs in trace | optional metadata | retrieved docs/spans | dataset docs | retrieval/tool observations |
metric |
metric object | metric | evaluator | testing criterion/grader | evaluator | metric | deterministic or judge check |
score |
score | score | evaluator score/feedback | grader result | eval result | metric value | normalized score |
judge |
model-backed metric config | LLM metric config | evaluator model | grader model | evaluator model | usually none | judge span or external judge |
trace_ref |
trace if enabled | external | run/trace ID | external/API run | trace ID | bundle/log path | native trace/run ID |
failure_code |
custom reason mapping | custom classification | feedback tag | grader label | annotation tag | postprocess | normalized failure taxonomy |
Hard Technical Boundaries
The main risk is not that the frameworks are incompatible. The risk is that teams collapse different grains into the same table and lose evidence.
| Boundary | Do not merge | Why it matters | Canonical fix |
|---|---|---|---|
| Eval case vs model attempt | task_id and attempt_id |
One case can be attempted by many models, prompts, retrieval indexes, and tool catalogs | separate eval_tasks and eval_attempts |
| Reference context vs retrieved context | DeepEval context/RAGAS reference_contexts and runtime retrieved chunks |
Reference evidence is the expected truth set; retrieved evidence is what the system actually found | store both in eval_evidence with `evidence_role=reference |
| Faithfulness vs truth | RAGAS/DeepEval faithfulness and source authority/freshness | A response can be faithful to stale or unauthorized context | add source authority, ACL, freshness, and citation-span checks |
| Tool schema vs tool call | available tool JSON schema and executed tool call | A valid tool schema does not mean the agent chose or used the tool correctly | store tool_catalog_version separately from tool_calls |
| Trace root vs metric result | LangSmith/Phoenix/provider trace and eval score | A score without trace coverage is not audit-ready; a trace without scoring is not an eval | require trace_ref on attempts and span coverage on governance gates |
| Judge config vs human label | LLM judge output and SME/human calibration | Judge drift and rubric ambiguity are separate failure modes | store judge model/rubric and human labels in separate review tables |
| Benchmark disclosure vs private evidence | model-card/public leaderboard score and internal eval result | Public benchmark scores are candidate evidence only | store model-card scores in model_registry, not eval_scores |
| SFT example vs eval failure | training row and failed eval case | Training on raw failures can leak secrets or reinforce bad behavior | promote only reviewed, redacted, approved examples with lineage |
| Prompt template vs user input | full prompt construction and case input | DeepEval warns not to put prompt templates into input; provider traces often include full prompt |
store clean input plus prompt_template_id and raw trace separately |
| OpenAI-compatible API vs behavior compatibility | request shape and output semantics | OpenAI-compatible serving normalizes transport, not model behavior, tokenizer, tool policy, or chat template | record provider, serving stack, tokenizer, chat template, and model revision |
Native Schema Shapes To Verify Before Adapter Work
The next implementation step should harvest exact schemas from primary docs or APIs into machine-readable adapters. Current verified shapes:
| Source | Verified shape | Adapter consequence |
|---|---|---|
DeepEval LLMTestCase |
input, actual_output, optional expected_output, context, retrieval_context, tools_called, expected_tools, token_cost, completion_time; tool calls include name plus optional description, reasoning, output, and input parameters |
Directly maps to task input, actual output, expected output, reference evidence, retrieved evidence, tool calls, cost, latency |
RAGAS SingleTurnSample |
user_input, retrieved_contexts, reference_contexts, retrieved context IDs, response/reference fields; MultiTurnSample uses message lists and optional reference tool calls/rubrics/topics |
Needs separate single-turn and multi-turn adapters; tool-call eval requires message/tool-call preservation |
| LangSmith evaluation | datasets/examples, evaluators, experiments, online/offline evaluation, human/code/LLM/pairwise evaluator types, annotation/feedback queues | Treat LangSmith dataset/example IDs and run/trace IDs as foreign keys, not the canonical warehouse ID |
| OpenAI Evals API | evals use data_source_config and testing_criteria; custom data source has JSON Schema item_schema; graders can reference sample output and item fields |
Adapter should store item schema, grader config, sample schema, eval ID, and raw run result |
| Hugging Face TRL SFTTrainer | language modeling text, conversational messages, prompt-completion prompt/completion, conversational prompt-completion message lists; tool-calling SFT adds tool_calls, tool role messages, and tools JSON schemas |
Training-example schema must record row format, chat template, loss mask mode, tool schema, dataset revision, model revision, and approval lineage |
| Hugging Face Evaluate | evaluator compute contract accepts model/pipeline, data, split/subset, metric, task-specific columns, strategy, and confidence parameters |
Store evaluator invocation plus raw metric result; exact result keys are metric-specific |
| Hugging Face LightEval | LightevalTaskConfig captures name, prompt function, Hub repo/subset, metrics, splits, few-shot settings, generation settings, and version; Doc captures query, choices, gold index, instruction, generation settings, few-shot docs, and images |
Store as model-candidate calibration evidence; output bundle/sample-result schema still needs deeper capture |
The detailed primary-source ledger is sources/21-benchmarks/eval-framework-native-schema-primary-sources-2026.md, with structured adapter boundaries in sources/21-benchmarks/eval-native-schema-boundaries-2026.json. It upgrades DeepEval, RAGAS, LangSmith, Phoenix, Promptfoo output behavior, OpenEvals evaluator return objects, Hugging Face Evaluate evaluator/suite boundaries, Hugging Face LightEval result bundles, TRL/SFT, and several provider SDK trace/event surfaces from conceptual coverage to concrete native field coverage. Remaining gaps are now narrower: version-pinned run fixtures, selected metric-specific mappings, and platform-specific exports that are not fully documented as stable public JSON schemas.
The Awesome Evals ingestion adds a second, broader source graph: 318 cited rows, one golden exemplar per citation, 269 raw mirrors, 143 mirrored BenchFlow notes, 46 media rows attempted through transcript acquisition and marked needs_asr, and 83 Chandra-extracted PDF rows, including 2 chunked extractions. That does not replace the native-schema ledger; it provides domain coverage and exemplar seeds for eval packs. The schema consequence is that verifier/RL-environment sources, benchmark-integrity sources, judge-calibration sources, and agent-security sources now need first-class eval_domain, recommended_failure_code, source_bundle_ref, and trace_ref fields even when no vendor-native export exists.
Provider Interplay
The runner and the provider should be separate. A DeepEval suite can call OpenAI, Anthropic, Gemini, Bedrock, Mistral, Qwen, a local Hugging Face model, or a staging app endpoint. RAGAS can score the RAG output from any provider if the retrieval contexts and final answer are captured. LangSmith or Phoenix can hold the trace even when DeepEval or RAGAS owns the metric. LightEval can run Hugging Face Transformers, vLLM, TGI, Inference Endpoints, LiteLLM, and inference-provider backends, then push bundles locally or to the Hub.
| Provider / platform | Native strength | External framework pairing | What to centralize |
|---|---|---|---|
| OpenAI | Evals API, graders, Agents SDK traces | DeepEval for CI, RAGAS for RAG metrics, Phoenix/LangSmith for traces | raw eval run, grader config, trace ID, model snapshot |
| Anthropic / Claude Code | prompt evals, Agent SDK, hooks, permissions, MCP, OTel | DeepEval/custom pytest for coding tasks, LangSmith/Phoenix for traces | repo snapshot, tool trace, permission events, final diff |
| Google / Gemini / ADK | trajectory/final-response eval split, Gen AI eval service, A2A | RAGAS for grounding, protocol conformance packs for A2A | ADK eval set, A2A task/artifact records, trace spans |
| AWS Bedrock | Agents, Knowledge Bases, AgentCore Observability | RAGAS for KB metrics, Phoenix/LangSmith for trace inspection, custom ACL checks | KB query trace, citations, action group calls, ACL decision |
| Microsoft / AutoGen / Foundry | multi-agent architectures and monitoring | AutoGenBench plus canonical result adapter | team graph, stop reason, handoffs, per-agent scores |
| Mistral | agents, conversations, handoffs, document library, MCP | RAGAS for citations/grounding, DeepEval for tool tests | conversation ID, handoff trace, document refs |
| Qwen / local OSS | Qwen-Agent, function calling, MCP, vLLM/OpenAI-compatible serving | DeepEval, LightEval, lm-eval-harness, RAGAS | serving template, tokenizer/chat template, tool traces |
| Hugging Face local models | Hub datasets/models, Evaluate, LightEval, TRL, PEFT | DeepEval/RAGAS around app layer; LightEval around base model | model revision, dataset revision, adapter hash, eval bundle |
Hugging Face Role
Hugging Face is not one eval framework. It is four different things in this architecture:
- Model/dataset registry. Store model cards, dataset cards, model revisions, dataset revisions, licenses, and benchmark disclosures.
- Evaluation libraries. Use Evaluate for metrics and LightEval for model benchmark runs with multiple backends.
- Execution backends. Run local or hosted models through Transformers, vLLM, TGI, Inference Endpoints, Inference Providers, or LiteLLM-compatible backends.
- Post-training substrate. Use TRL/SFTTrainer, PEFT, DPO/GRPO/reward trainers, and dataset formats for training and adaptation.
The key distinction: eval cases are governance artifacts; SFT rows are training artifacts. They can share lineage, but they should not be the same table.
SFT And Post-Training Schemes
SFT should be fed from reviewed eval failures and successful expert traces, not directly from raw production logs.
| Scheme | Native row shape | Use | Store as | Promotion rule |
|---|---|---|---|---|
| Standard language modeling | {"text": "..."} |
domain style/language adaptation | training_examples with format=text |
only from clean licensed corpus |
| Conversational language modeling | {"messages": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]} |
chat behavior and assistant style | training_examples with format=messages |
strip secrets; preserve role labels and template version |
| Standard prompt-completion | {"prompt": "...", "completion": "..."} |
instruction following or extraction | training_examples with format=prompt_completion |
link to source eval case and SME approval |
| Conversational prompt-completion | {"prompt": [{"role": "..."}], "completion": [{"role": "..."}]} |
multi-turn assistant behavior | training_examples with format=conversation_pair |
require conversation redaction and policy review |
| Preference pair / DPO | prompt plus chosen/rejected responses | preference tuning | preference_pairs |
chosen/rejected must include rubric reason |
| Reward model / judge data | prompt, response, score/rating/rationale | evaluator or reward model calibration | judge_calibration_examples |
keep human labels and inter-rater metadata |
| Process reward / trajectory | step sequence, actions, observations, step scores | agent/coding trajectory improvement | trajectory_training_examples |
only from traces with complete tool/action evidence |
Do not store SFT examples only in Hugging Face Datasets. Mirror them in the corporate artifact store with lineage:
training_example:
id: sft_hr_policy_000123
source_eval_case_id: corp_kb_policy_qa_v1/case_017
source_trace_id: trace_abc
source_failure_code: retrieval.wrong_source
format: conversational_prompt_completion
dataset_revision: hf_dataset_sha_or_internal_snapshot
model_family_target: qwen3_or_claude_distillation_or_internal_slm
license_scope: internal_only
pii_status: redacted
approved_by: hr_policy_sme
approved_at: 2026-06-21
Where To Store And Centralize
Use a layered storage model:
| Layer | Location | Contents | Owner |
|---|---|---|---|
| Repo-local eval pack | evals/<pack_id>/ in each product repo |
cases, fixtures, source bundle manifest, local runner configs, deterministic checks | repo team |
| Shared schema package | statebench/schemas/ or internal package |
golden_work_packet_v1, eval_result_v1, trace_ref_v1, failure taxonomy |
eval platform |
| Artifact/object store | S3/R2/GCS/Blob | raw traces, raw vendor exports, source bundles, rendered docs, screenshots, code diffs | platform/security |
| Warehouse tables | Snowflake/BigQuery/Postgres/DuckDB for local | normalized runs, tasks, attempts, scores, failures, evidence, reviews, model/dataset lineage | data platform |
| Hugging Face private org | private datasets/models/spaces where appropriate | model candidates, dataset cards, LightEval bundles, training datasets, adapters | ML platform |
| Knowledge base | curated wiki/docs only | reviewed failure patterns, benchmark guidance, approved runbooks | knowledge owners |
The warehouse should be canonical for cross-team reporting. Repos remain canonical for executable tests. Hugging Face is canonical for model/dataset artifacts only when the organization chooses it as the ML registry, but the eval warehouse should still store dataset/model revision IDs.
Recommended Directory Contract
evals/
corp_kb_policy_qa_v1/
pack.yaml
cases.jsonl
source_bundle.yaml
rubrics/
grounding_v3.yaml
checks/
citation_span.py
acl_policy.py
runners/
deepeval.yaml
ragas.yaml
openai_evals.yaml
lighteval.yaml
baselines/
2026-06-21.jsonl
exports/
README.md
training/
datasets/
corp_kb_sft_v1/
dataset_card.md
train.jsonl
validation.jsonl
lineage.jsonl
redaction_report.md
adapters/
qwen3-corp-kb-lora-v1/
adapter_config.json
eval_before_after.jsonl
Warehouse Minimum Tables
| Table | Grain | Required fields |
|---|---|---|
eval_runs |
suite run | run_id, suite_id, runner, runner_version, git_sha, started_at, provider, model, model_version |
eval_tasks |
case/task | task_id, pack_id, persona, domain, risk_tier, source_bundle_id, expected_artifact_type |
eval_attempts |
one attempt | attempt_id, run_id, task_id, input_ref, output_ref, trace_ref, latency, tokens, cost |
eval_scores |
one metric | attempt_id, metric_id, metric_type, score, threshold, pass, judge_model, rubric_version |
eval_failures |
one classified failure | attempt_id, failure_code, severity, owner, evidence_ref, remediation_status |
eval_evidence |
source evidence | evidence_id, source_uri, source_version, span_ref, acl_scope, freshness_status |
eval_traces |
trace root | trace_ref, exporter, span_count, required_span_coverage, raw_export_ref |
eval_human_reviews |
reviewer label | review_id, attempt_id, reviewer_role, rubric_version, score, rationale_ref |
model_registry |
model candidate | provider, model_id, revision, serving_stack, tokenizer, chat_template, license |
dataset_registry |
eval/training dataset | dataset_id, revision, split, license, pii_status, source_lineage_ref |
training_examples |
SFT/RL example | example_id, format, source_eval_case_id, source_trace_id, approval_status, dataset_revision |
Verified Gaps From Fan-Out Review
The deeper verification pass found a clean split between covered concepts and adapter-ready contracts. The report should not imply we already have exact native export schemas for every framework.
| Area | Current coverage | Hard gap | Adapter consequence |
|---|---|---|---|
| DeepEval | LLM test-case fields, metric concepts, and deepeval==4.0.6 local no-key TestRun JSON are captured |
hosted Confident AI export shape, component trace schema, and built-in LLM judge metric exports still need primary capture | adapter can map local TestRun results now if it stores raw exports and keeps hosted/trace exports separate |
| RAGAS | single-turn and multi-turn sample shapes are captured, and ragas==0.4.3 EvaluationDataset.to_jsonl() is now pinned as a real JSONL fixture |
exact scored EvaluationResult object/dataframe/JSON export, metric-required-field matrix, and cost/traces serialization still need primary capture |
adapter must keep sample schema and metric result schema separate |
| LangSmith | dataset/example/evaluator/trace workflow is captured | exact run, feedback, annotation queue, and experiment export JSON need primary capture | use LangSmith IDs as foreign keys until export schemas are mirrored |
| Phoenix / OpenInference | trace/eval/annotation role is captured | exact dataset row, eval result, annotation, and export shapes need primary capture | OpenInference spans can be normalized, but eval results need raw-export refs |
| Promptfoo / OpenEvals | Promptfoo OutputFile, EvaluateSummaryV3, EvaluateResult, GradingResult, JUnit projection, provider response, red-team metadata, and OpenEvals Python EvaluatorResult are captured from primary source |
no standalone published Promptfoo JSON Schema; Promptfoo red-team report UI schema, OpenEvals TypeScript result types, and LangSmith serialized feedback payload need more capture | adapter can ingest JSON/JSONL/OpenEvals result rows now if it stores raw exports and pins package/version |
| Hugging Face Evaluate | evaluator invocation, EvaluationSuite result annotation, metric module result boundary, distributed cache metadata, and evaluate==0.4.6 evaluate.save() JSON serialization are captured |
metric-module execution, selected per-metric output fixtures, EvaluationSuite, and distributed cache artifacts still need pinned captures |
do not treat Evaluate output as a generalized agent eval row; map metric keys only after declaring metric-specific adapters |
| Hugging Face LightEval | task config, aggregate result JSON sections, detail artifact layout, Doc, ModelResponse, and a lighteval==0.13.0 EvaluationTracker.save() logger-level bundle fixture are captured |
full CLI run over a real model/dataset still needed to confirm populated config_tasks and backend-specific export variations |
LightEval belongs in model-candidate calibration unless linked to app cases |
| TRL / PEFT / SFT | row families are captured | exact TRL row examples for SFT, prompt-completion, preference/DPO, KTO, reward modeling, process reward, VLM, and tool-calling are not fully ingested | training examples need their own schema and approval lineage |
| Provider SDK traces | OpenAI Agents SDK trace/span data, Google ADK runtime/eval events, AutoGen OpenTelemetry runtime attributes, Mistral conversation output entries, and Qwen-Agent message/function-call stream shapes are captured | Google Agent Engine exports, Microsoft Foundry eval exports, Mistral observability/judge OpenAPI objects, and real exported trace fixtures remain incomplete | canonical trace adapter must retain raw payload refs, semantic convention versions, and source SDK commit/package versions |
| GitLab federation | decentralized run pattern plus first-pass schemas are captured | JUnit element mapping and production collector implementation still need work | repo-local evals can emit cases, annotations, normalized results, and warehouse rows through enforceable contracts |
StateBench Current Schema Boundary
StateBench already has useful model/runtime comparability fields. It records model artifact identity, adapter state, artifact format, precision, backend kind, backend hardware, context length, prompt version, and inference parameters in run_manifest.schema.json. Its task result schema records result_id, suite_run_id, task_id, slice, artifact_id, backend_id, model, acceptance, check results, timing, tokens, cost, and a narrow runner-oriented failure_mode.
That is enough for benchmark-score hygiene. It is not enough for enterprise eval governance.
| Missing contract | Why it matters |
|---|---|
statebench/schemas/golden_work_packet_v1.schema.json |
defines the durable case/source/rubric/persona packet that teams can run locally |
statebench/schemas/eval_case_v1.schema.json |
defines one cases.jsonl row with input grain, expected artifact, source bundle, rubrics, persona, risk tier, and failure-domain coverage |
statebench/schemas/eval_result_v1.schema.json |
normalizes results from DeepEval, RAGAS, Promptfoo, OpenAI Evals, LightEval, pytest, and custom runners; requires harness name/version/runner kind, model provider/name/version, runtime context, and attached tool inventory |
statebench/schemas/eval_source_gap_v1.schema.json |
turns source-ledger gaps into tracked rows with owner, priority, status, verification requirement, related fixtures, and raw export refs |
statebench/schemas/source_bundle_v1.schema.json |
preserves source URI, version, authority, ACL, freshness, citation spans, and bundle hash |
statebench/schemas/trace_ref_v1.schema.json |
stores native trace/export refs, span coverage, semantic convention version, redaction version, and raw payload hash |
statebench/schemas/failure_taxonomy_v1.schema.json |
separates retrieval, reasoning, tool, safety, policy, code, UX, infra, and judge failures with owner and gate behavior |
statebench/schemas/gitlab_eval_annotation_v1.schema.json |
defines merge-request and pipeline annotation rows emitted alongside JSONL results and JUnit |
statebench/schemas/kb_promotion_v1.schema.json |
governs how a failed eval or expert trace becomes a curated knowledge-base learning |
statebench/schemas/training_example_v1.schema.json |
links SFT/RL examples to source failures, approvals, redaction, dataset revision, and model target |
statebench/schemas/native_eval_fixture_v1.schema.json |
verifies source-observed or real-run native framework payloads before adapter authors normalize them into StateBench rows |
statebench/schemas/team_eval_rollup_v1.schema.json |
defines the team collector scorecard window after ingesting repo-local eval artifacts |
statebench/schemas/firm_eval_rollup_v1.schema.json |
defines the firm-wide collector scorecard window after ingesting team rollups |
statebench/migrations/001_enterprise_eval_warehouse.sql |
turns the minimum warehouse tables into enforceable keys, enums, indexes, retention, and uniqueness constraints |
statebench/runners/eval_rollup_collector.py |
deterministic collector that reads eval_result_v1 JSONL and emits validated team and firm rollup JSON |
statebench/runners/enterprise_eval_contract_smoke.py |
single GitLab-runner-friendly smoke gate that demonstrates pass fixtures validate and fail fixtures fail before upload |
statebench/ci/gitlab-enterprise-eval-component.yml |
reusable GitLab CI component template that runs repo-local eval packs, validates normalized outputs, and publishes JUnit plus annotation artifacts |
These contracts are now first-pass repository artifacts. They are intentionally narrower than a vendor export schema: they define the enterprise boundary that repo-local runners must emit. The normalized result boundary is now explicit: every eval_result_v1 row must identify the harness name, harness version, runner kind, model provider/name/version, prompt template version, retrieval index version, tool catalog version, judge model version, execution environment, and attached tools. Attached tool rows can carry the tool type, tool version, source registry, schema ref/hash, permission profile, network access class, and data scope. That is the hard line between an interesting score and an audit-ready enterprise eval result.
The native fixture layer is separate from normalized results: statebench/fixtures/enterprise-eval-contract/v1/native-fixtures.pass.jsonl currently pins source-observed DeepEval LLMTestCase, RAGAS SingleTurnSample, LangSmith run format, Phoenix/OpenInference spans, Promptfoo EvaluateResult, OpenEvals EvaluatorResult, Hugging Face Evaluate result serialization, and LightEval result-bundle shapes with adapter expectations and explicit limitations. It also includes a package-pinned DeepEval 4.0.6 local TestRun JSON fixture stored as statebench/fixtures/enterprise-eval-contract/v1/deepeval-test-run-realrun-4.0.6.json, package-pinned Promptfoo 0.121.17 JSON and JSONL real_run_export fixtures generated with the local echo provider and stored as statebench/fixtures/enterprise-eval-contract/v1/promptfoo-realrun-0.121.17.json and statebench/fixtures/enterprise-eval-contract/v1/promptfoo-realrun-0.121.17.jsonl, a package-pinned RAGAS 0.4.3 EvaluationDataset.to_jsonl() fixture stored as statebench/fixtures/enterprise-eval-contract/v1/ragas-dataset-realrun-0.4.3.jsonl, a package-pinned Hugging Face Evaluate 0.4.6 evaluate.save() fixture stored as statebench/fixtures/enterprise-eval-contract/v1/evaluate-realrun-0.4.6.json, and a package-pinned LightEval 0.13.0 EvaluationTracker.save() logger-level bundle manifest stored as statebench/fixtures/enterprise-eval-contract/v1/lighteval-logger-realrun-0.13.0.json. A fixture can be marked source_observed_contract, documentation_example, or real_run_export; only real_run_export should be used as proof that a package-versioned run produced that exact payload. The schema now requires real_run_export rows to carry a sha256: payload hash, statebench/runners/contract_validator.py recomputes that hash and any referenced raw JSON/JSONL export hash so stale or fabricated real-run fixtures fail validation, and statebench/runners/validate_eval_source_ledger.py checks that sources/21-benchmarks/eval-native-schema-boundaries-2026.json lists the same external real-run exports as the fixture file. The open/captured remainder is also machine-readable: sources/21-benchmarks/eval-native-schema-gap-backlog-2026.jsonl records every source-ledger remaining_gaps item as an eval_source_gap_v1 row with status, gap type, owner, priority, related fixtures, and verification requirement; statebench/runners/validate_eval_gap_backlog.py fails when that backlog drifts from the source ledger.
The remaining ingestion work is no longer a blanket “find every schema” task; it is a typed backlog. The highest-priority open rows are DeepEval hosted Confident AI exports, DeepEval component trace exports, built-in LLM judge metric runs, RAGAS scored EvaluationResult frames and traces, RAGAS metric-required-field matrices, Promptfoo red-team report exports, full LightEval CLI runs with populated task configs, OpenAI custom trace processor fixtures for ZDR-compatible deployments, Google Agent Engine exports, and provider-specific platform exports from Microsoft Foundry and Mistral.
The shared smoke entrypoint is statebench/runners/enterprise_eval_contract_smoke.py; the single-file debug entrypoint is statebench/runners/contract_validator.py. The pass/fail fixtures in statebench/fixtures/enterprise-eval-contract/v1/ demonstrate that repo-local cases.jsonl, normalized eval-results.jsonl, native fixture rows, native real-run hashes, GitLab annotation artifacts, and team/firm rollup scorecards can be checked before upload. The rollup fixtures are no longer disconnected examples: statebench/runners/eval_rollup_collector.py regenerates them from eval-results.pass.jsonl, and the smoke suite fails if generated rollups drift from the checked-in fixtures. native-fixtures.fail.jsonl intentionally carries a bad real-run hash and should fail. The GitLab component template is checked by statebench/runners/validate_gitlab_eval_component.py and by the all-up smoke suite.
Failure Domain Shape
Users should classify failures by domain first, then by mechanism. A useful failure code answers: what system boundary failed, what evidence proves it, and who can fix it?
| Domain | Example failure codes | Required evidence | Typical owner |
|---|---|---|---|
| Knowledge retrieval | retrieval.no_source, retrieval.wrong_source, retrieval.stale_source, retrieval.acl_leak |
query, retrieved chunks, reference source bundle, ACL/freshness metadata | search/RAG owner |
| Grounded answer quality | answer.unsupported_claim, answer.missing_citation, answer.wrong_citation_span, answer.over_refusal |
final answer, citation spans, reference evidence, judge/human label | app team + domain SME |
| Tool and action use | tool.not_called, tool.wrong_tool, tool.bad_args, tool.bad_result_handling, tool.unsafe_action |
tool catalog version, tool call request/result, approval event, trace spans | agent/platform owner |
| Agent trajectory | trajectory.loop, trajectory.premature_stop, trajectory.bad_handoff, trajectory.context_loss |
full trace, turn IDs, handoff/task IDs, stop reason | agent runtime owner |
| Coding agent output | code.patch_failed, code.test_failed, code.unrelated_change, code.missing_file, code.security_regression |
git SHA, diff, test logs, sandbox/tool trace | repo team |
| Safety/policy/compliance | policy.disallowed_content, policy.pii_exposure, policy.license_violation, policy.audit_gap |
policy version, classifier/judge result, redaction state, source license | safety/compliance |
| Model/runtime | runtime.timeout, runtime.api_error, runtime_context_overflow, runtime.model_drift, runtime.backend_mismatch |
provider request ID, model revision, backend config, token/cost/latency | platform/ML infra |
| Evaluator quality | judge.rubric_ambiguous, judge.model_drift, judge.low_agreement, judge.prompt_leak |
judge model/version, rubric version, human calibration, disagreement examples | eval platform |
Each failure row should store domain, code, severity, gate_behavior, owner_role, evidence_ref, trace_ref, rubric_version, remediation_status, and review_required.
Decision
We should centralize normalized results and lineage, not framework configs. Framework configs stay close to the code that runs them. The enterprise-owned objects are:
golden_work_packet_v1;eval_result_v1;failure_taxonomy_v1;trace_ref_v1;source_bundle_v1;model_candidate_v1;training_example_v1.
This lets DeepEval, RAGAS, LangSmith, Phoenix, OpenAI Evals, Hugging Face LightEval/Evaluate, and provider SDKs interoperate without forcing teams to abandon the tools that fit their local workflow.
Evidence Boundaries
- The raw data now captures several exact source-level native shapes, and the first machine-validated native fixture layer covers DeepEval, RAGAS, LangSmith, Phoenix/OpenInference, Promptfoo, OpenEvals, Hugging Face Evaluate, and LightEval. DeepEval now has an external package-pinned local TestRun JSON fixture; Promptfoo now has external package-pinned JSON and JSONL real-run fixtures; RAGAS now has an external package-pinned dataset JSONL real-run fixture; Hugging Face Evaluate now has an external package-pinned
evaluate.save()fixture; LightEval now has an external package-pinnedEvaluationTracker.save()logger-level bundle fixture. Public schemas remain uneven. Production adapters should pin tool versions, store raw exports, and maintain per-tool fixture tests that distinguish source-observed contracts from real-run exports. - Hugging Face model cards and community leaderboards are candidate evidence; they are not private-enterprise deployment evidence.
- SFT examples should be lineage-linked to eval cases and human approvals. Raw failures should not flow directly into training.
- RAG metrics should be paired with source authority, ACL, freshness, and citation-span checks.
- Provider SDK traces need span coverage validation before they are used as governance artifacts.