This synthesis is backed by primary source cards under sources/21-benchmarks/
for AgenticDataBench, semantic metadata and agentic retrieval, LongDA,
AgenticOpsEval, kRAIG, DataGovBench, DCA-Bench, Replayable Financial Agents,
and FinReasoning.
Evidence Boundaries:
Sources are promoted at different confidence levels. DCA-Bench and FinReasoning have primary GitHub evidence inspected in this pass. AgenticOpsEval has a reachable public GitLab project shell but needs deeper file inspection. AgenticDataBench, semantic metadata, LongDA, kRAIG, and DataGovBench are arXiv-primary source cards until code/data artifacts are found and locally verified. Reported model scores remain source-reported snapshots unless a StateBench fixture reproduces the harness.
Executive Answer
The new sources change the eval map in one concrete way: agents should not be graded only on final answers or chat quality. Enterprise evals need a separate DataOps lane that scores whether the agent found executable data, understood metadata, navigated documentation, produced valid transformations, and grounded diagnostic reasoning in real evidence.
The expanded core:
| Source | What It Adds | Hard Boundary |
|---|---|---|
| AgenticDataBench | Broad data-agent tasks across 15 vertical domains, 5 fintech B2B use cases, skill-level data-science operations | Abstract-only until code/data are inspected |
| Do Agents Need Semantic Metadata? | FAIR-mapped retrieval eval showing structured metadata beats open-web retrieval for usable datasets | LLM-judge caveat; not a full data-quality system |
| LongDA | Long-document data analysis over surveys, codebooks, raw files, and publications | General data-agent pattern, not finance-specific |
| AgenticOpsEval | Microservice failure diagnosis with localization, identification, and evidence-grounded reasoning | Diagnosis benchmark, not autonomous remediation authority |
| kRAIG | Natural-language DataOps pipeline generation into Kubeflow Pipelines with clarification and validation | Abstract-only until runnable code is verified |
| DataGovBench | Data governance workflows with realistic noise, transformation code, and pipeline reliability scoring | Abstract-only until code/data are verified |
| DCA-Bench | Dataset quality issue discovery across online dataset platforms, with GitHub/HF artifacts and output schema | LLM-evaluator and licensing/ethics caveats |
| Replayable Financial Agents | Determinism, decision replay, faithfulness, and DataOps exception handling for tool-using agents | GitHub artifact reported but not locally run |
Research Map
These sources sit between RAG evals and coding-agent evals. They evaluate the middle work that most corporate agents actually perform:
find data
-> verify metadata and access
-> interpret schema/docs/codebooks
-> clean/transform
-> compute
-> cite evidence
-> diagnose failures
-> produce auditable artifact
That means the eval harness must preserve more than a prompt and response. Minimum useful records include dataset URL, download URL, schema fields, license/access note, documentation spans, transformations, code execution logs, artifact hashes, evaluator labels, and final answer.
Source Findings
AgenticDataBench
AgenticDataBench expands the benchmark unit from “answer this question” to “execute a data-science workflow.” Its important design move is skill-level coverage: data tasks are organized around recurring operational patterns, with task-solution diversity selected by skill composition.
StateBench implication: a GitLab-runner eval should report per-skill performance for data discovery, schema inspection, cleaning, joining, aggregation, visualization, modeling, and artifact validation. A single aggregate score hides the failure.
Source card:
sources/21-benchmarks/agenticdatabench-data-agent-benchmark-raw.md.
Semantic Metadata for Agentic Data Retrieval
This paper names the failure our Samaya miss exposed: Last-Mile Utility. The agent can find a plausible page but not the machine-usable artifact. The paper reports that the semantic metadata path delivers higher precision for metadata-rich registries and machine-readable downloads, while open-web retrieval often returns prose pages or portal landing pages.
StateBench implication: source ingestion must distinguish:
- page found;
- canonical source found;
- machine-readable artifact found;
- schema/metadata found;
- access/license recorded;
- data is computationally usable.
Source card:
sources/21-benchmarks/semantic-metadata-agentic-data-retrieval-raw.md.
LongDA
LongDA is the benchmark pattern for enterprise knowledge bases with attached datasets. It requires agents to read heterogeneous documentation before writing analysis code. This is the corporate data-estate reality: the answer depends on codebooks, schema notes, survey methods, raw files, and prior publications.
StateBench implication: knowledge-base evals should include documentation navigation and codebook-to-data alignment, not only answer faithfulness.
Source card:
sources/21-benchmarks/longda-long-document-data-analysis-agents-raw.md.
AgenticOpsEval
AgenticOpsEval brings the same idea into operations. It scores localization, identification, and reason/evidence grounding over microservice failures. This is useful for DataOps because data pipelines fail through logs, metrics, runtime events, database/middleware issues, and application logic.
StateBench implication: operational evals need evidence-grounding labels on the reasoning trace. Final root-cause correctness is insufficient.
Source card:
sources/21-benchmarks/agenticopseval-microservice-failure-diagnosis-raw.md.
kRAIG
kRAIG moves from evaluating analytical answers to evaluating generated DataOps pipelines. The key design pattern is ReQuesAct: the agent clarifies ambiguous intent before synthesizing a Kubeflow Pipeline, then validates pipeline integrity before execution.
StateBench implication: pipeline-generation evals should score clarification quality, KFP validity, extraction/loading success, transformation accuracy, and pre-execution validation separately.
Source card:
sources/21-benchmarks/kraig-dataops-pipeline-generation-raw.md.
DataGovBench
DataGovBench frames governance as executable transformation under data quality, security, and compliance constraints. Its reversed-objective methodology synthesizes realistic noise, which is exactly the kind of controlled corruption enterprise data-quality evals need.
StateBench implication: add governance-specific tasks where the answer is not a paragraph but a corrected dataset or transformation pipeline with validation proof.
Source card:
sources/21-benchmarks/datagovbench-data-governance-workflows-raw.md.
DCA-Bench
DCA-Bench is the most concrete data-quality artifact in this pass. It has a
GitHub repo, Hugging Face dataset, task metadata, four hint levels, and a
Curator output schema with issue, evidence, and description.
StateBench implication: source-ingestion evals should include hidden defect discovery. A good agent should identify incomplete documentation, inaccurate labels, licensing gaps, ethical/legal risks, and outdated information before those issues contaminate downstream retrieval, evals, or training data.
Source card:
sources/21-benchmarks/dca-bench-dataset-curation-agents-raw.md.
Replayable Financial Agents
The replayability harness adds an audit metric often missing from eval design: the same inputs should produce repeatable decisions and evidence-aligned trajectories. The source also includes a DataOps exceptions lane, making it a bridge between finance governance and operational data failures.
StateBench implication: each GitLab-runner eval should record repeated-run variance, trace determinism, decision determinism, and faithfulness separately from accuracy.
Source card:
sources/21-benchmarks/replayable-financial-agents-determinism-faithfulness-raw.md.
Failure Domains To Add
| Failure Domain | What To Classify | Eval Evidence Needed |
|---|---|---|
| Data discoverability | Could the agent find the right dataset, not just a page? | dataset URL, registry metadata, search trace |
| Last-mile utility | Did it reach a machine-readable file/API? | download/API URL, file type, row count, schema |
| Metadata fidelity | Did it preserve schema, units, time, entity keys, and license? | schema fields, units, dates, access/license notes |
| Documentation alignment | Did it use the right codebook or methodology note? | cited doc span, variable mapping, method constraint |
| Transformation correctness | Did cleaning/join/aggregation code match the task? | code, execution log, intermediate artifact hash |
| Computation correctness | Did output values reconcile? | deterministic checker, tolerances, recomputation trace |
| Evidence-grounded diagnosis | Did the reasoning cite the relevant logs/metrics/spans? | trace/log refs, fault category, localization label |
| Remediation boundary | Did the agent avoid unauthorized fixes? | diagnosis vs action flag, approval state |
| Replay drift | Does the same input produce the same trajectory and decision? | repeated runs, trace hashes, decision hashes |
Harness Implications
The current source-ingestion pipeline should add a DataOps artifact contract:
{
"source_system": "web|hf|github|gitlab|registry|internal",
"landing_page_url": "...",
"machine_readable_artifact_urls": ["..."],
"metadata_urls": ["..."],
"schema_fields": ["..."],
"row_count": 0,
"license_or_access": "...",
"documentation_refs": ["..."],
"code_or_harness_refs": ["..."],
"execution_required": true,
"eval_dimensions": [
"discoverability",
"last_mile_utility",
"metadata_fidelity",
"documentation_alignment",
"transformation_correctness",
"computation_correctness",
"evidence_grounding"
],
"caveats": ["..."]
}
That contract is intentionally separate from training exports. The raw evidence must stay in source cards and trace artifacts; SFT or preference rows should be late exports after redaction, dedupe, rights review, and held-out split assignment.
Watchlist From This Pass
- FinReasoning was promoted into the finance cluster because it is a financial research-report benchmark with concrete GitHub schema evidence.
- Hugging Face search surfaced multiple low-evidence community finance/data mirrors. These should remain watchlist until primary org, code, license, schema, and reproducibility are verified.
- AgenticDataBench claims an open-sourced testbed, but this pass did not verify a concrete repository. Treat it as abstract-level evidence until found.
Sources
- AgenticDataBench — arXiv:2607.01647. Source card:
sources/21-benchmarks/agenticdatabench-data-agent-benchmark-raw.md. - Do Agents Need Semantic Metadata? — arXiv:2605.28787. Source card:
sources/21-benchmarks/semantic-metadata-agentic-data-retrieval-raw.md. - LongDA — arXiv:2601.02598. Source card:
sources/21-benchmarks/longda-long-document-data-analysis-agents-raw.md. - AgenticOpsEval — arXiv:2606.29193 and public GitLab shell. Source card:
sources/21-benchmarks/agenticopseval-microservice-failure-diagnosis-raw.md. - FinReasoning — arXiv:2603.19254 and TongjiFinLab GitHub. Source card:
sources/21-benchmarks/finreasoning-hierarchical-financial-research-reporting-raw.md. - kRAIG — arXiv:2603.20311. Source card:
sources/21-benchmarks/kraig-dataops-pipeline-generation-raw.md. - DataGovBench — arXiv:2512.04416. Source card:
sources/21-benchmarks/datagovbench-data-governance-workflows-raw.md. - DCA-Bench — arXiv:2406.07275, GitHub, and Hugging Face dataset. Source
card:
sources/21-benchmarks/dca-bench-dataset-curation-agents-raw.md. - Replayable Financial Agents / DFAH — arXiv:2601.15322. Source card:
sources/21-benchmarks/replayable-financial-agents-determinism-faithfulness-raw.md.