← Benchmarks 🕐 10 min read
Benchmarks

Corporate Evals, Trace Mining, and Automatic Skill Improvement — 2026

Answer · March 2026

The cutting edge in enterprise evals is no longer "which model scored highest." It is **which workflow environment produces trustworthy evidence, which failures become reusable improvements, and which

Source files: sources/21-benchmarks/corporate-evals-trace-skill-improvement-2026-raw.md · sources/21-benchmarks/parlance-auto-evals-production-trace-error-analysis-raw.md · research/21-benchmarks/enterprise-ai-evals-field-guide-2026.md · research/21-benchmarks/eval-framework-schema-crosswalk-2026.md · research/21-benchmarks/enterprise-evaluation-knowledge-coding-agents-2026.md · research/19-agent-frameworks/nvidia-data-flywheel-slm-factory-2026.md · research/21-benchmarks/trace-to-training-pair-conversion-2026.md.

Source status: Current as of 2026-07-03. The source set combines primary/open research and repository metadata with vendor blogs. Vendor blogs are treated as architecture signals, not independent effectiveness evidence.

The cutting edge in enterprise evals is no longer “which model scored highest.” It is which workflow environment produces trustworthy evidence, which failures become reusable improvements, and which improvements are allowed back into production.

Ramp, Snorkel, Red Hat, and the NVIDIA/ASPIRE robotics thread converge on the same architecture:

  1. Represent the business workflow, not just the prompt. Ramp’s Finance Intelligence Layer points at the practical substrate: GL accounts, reporting logic, policies, source systems, paper workflows, and KPI-linked actions.
  2. Evaluate the task, trace, outcome, and rubric together. Snorkel’s federal/eval material is explicit: public benchmark scores miss workflow specificity, process evidence, and staleness.
  3. Run evals in executable environments. Terminal-Bench/Harbor-style patterns show why containerized rollouts, tools, rewards, and traces are more useful than static prompt/answer rows for agents.
  4. Mine traces into skills only after validation. ASPIRE shows the strongest current pattern: fine-grained execution traces localize failures; agents synthesize repairs; a coordinator promotes only reusable, validated fixes into a shared skill library.
  5. Package skills like governed software. Red Hat’s prodsec-skills repository shows the packaging layer: triggerable SKILL.md files, categories, indexes, provenance, installation paths, scanners, and pre-merge checks.
  6. Run red-team evals as governed jobs. TrustyAI’s llama-stack-provider-trustyai-garak repository is the missing Red Hat eval tool: it wraps NVIDIA Garak as an Eval-Hub/Llama Stack provider with Kubernetes and Kubeflow execution, named OWASP/AVID/CWE/quality/intents profiles, JSONL/HTML/AVID reports, MLflow integration, and attack-success metrics.
  7. Evaluate domain agents with tool provenance. Red Hat’s aegis-ai repository shows a security-agent pattern: CVE/advisory/component analysis runs against context tools such as OSIDB, RHTPAv2, osv.dev, CWE, NVD, GitHub, CISA KEV, and MCP servers, and returns structured outputs that include confidence, tools used, and a human-validation disclaimer.

The enterprise implication: build a private eval stack that treats every agent run as either evidence, regression, or candidate skill material. Do not let raw traces become instructions until they pass source redaction, failure diagnosis, deterministic or SME validation, and regression checks.

Net-New Concepts

Concept Why it matters Source basis
Task-trace-outcome-rubric contract Final-answer scoring is too thin for consequential workflows. The trace and the business state change must be scored beside the answer. Snorkel federal/eval transcript; existing eval schema crosswalk.
Finance Intelligence Layer Corporate agents need a semantic layer over GL, policy, reporting logic, systems, and workflow artifacts before they can be evaluated or routed intelligently. Ramp Applied AI launch; existing Ramp AI Index corpus for spend measurement pattern.
Benchmark-as-environment Agent benchmarks are moving toward executable environments with tools, state, validators, rewards, and reproducible rollouts. Terminal-Bench 2.0, Harbor, Snorkel benchmark-design posts.
Longitudinal erosion evals Agents can pass a one-shot task while degrading code or process quality over repeated edits. SlopCodeBench and broader coding-agent eval discussion.
Validated repair to skill promotion The useful training unit is not the whole trace; it is a reusable repair pattern that passed validation and has an explicit trigger condition. ASPIRE, Workflow-to-Skill, Red Hat prodsec-skills.
Skill graph / skill contract A skill should carry preconditions, postconditions, evaluator hooks, failure signature, repair strategy, provenance, and rollback guidance. ASPIRE; Skill Graph paper; Red Hat skill packaging.
Eval-Hub red-team provider LLM safety/security evals can run as cloud-native jobs with benchmark profiles, probe selection, report artifacts, and attack-success metrics instead of ad hoc prompt tests. TrustyAI Garak Llama Stack provider.
Tool-provenance security-agent eval For CVE and advisory agents, the eval must check which vulnerability/source tools were used, whether the model was allowed to access secure internal systems, and whether outputs include confidence and human-validation boundaries. Red Hat Aegis-AI repository.
Continuous private model routing Model routing should be based on current private workflow evals with cost, latency, and scale constraints, not static public leaderboards. Ramp Applied AI; existing enterprise eval field guide.
BYO knowledge priority eval RAG evals must test source authority and source priority: internal policy should beat generic vendor docs when both are available. Red Hat Ansible BYO-knowledge post; existing RAG failure-mode wiki.
Human-seeded automated error analysis Automated tools can find many trace-visible failures, but miss product-context failures unless a human teaches the criteria through annotations. Parlance Labs 100-trace production experiment.

Evolved Existing Concepts

Existing concept How it evolved
Golden datasets Evolved into golden work packets: task card, source bundle, tool catalog, policy boundary, expected artifact, trace expectations, deterministic checks, rubric, and reviewer authority.
LLM-as-judge Evolved into judge calibration and drift management: the judge prompt, model, rubric version, human disagreement rate, and known blind spots must be recorded.
RAG faithfulness Evolved into source-governed grounding: faithfulness is insufficient unless freshness, source authority, ACL, citation span, and source priority are checked.
Failure taxonomy Evolved from final-output error labels into process-localized failure codes: intake, retrieval, tool, trace, judge, benchmark, and training-promotion errors.
Automated eval agents Evolved from one-shot trace summarizers into a required human annotation loop: auto-analysis proposes clusters, humans accept/reject, and accepted labels update the taxonomy and eval suite.
Prompt optimization Evolved into environment and verifier design: for agents, the reward/verifier and reproducible rollout often matter more than another prompt variant.
Training data creation Evolved into trace-to-skill and trace-to-training pipelines with redaction, segmentation, validation, provenance, and regression gates.
Public benchmarks Evolved into frontier-shaping infrastructure: good benchmarks are unsaturated, expert-validated, diverse, reproducible, easy to run, and useful for RL/tuning.
Skills Evolved from static prompts into governed procedural modules that can be installed, triggered, tested, versioned, and retired.

Best Practices for Corporate Dataset Evals

1. Start with Work Packets, Not Questions

A corporate eval case should look like a real work packet:

{
  "case_id": "ap-hold-review-042",
  "workflow": "accounts_payable_exception_review",
  "persona": "ap_manager",
  "source_bundle": ["invoice", "po", "vendor_master", "policy_excerpt", "ticket_history"],
  "tool_catalog": ["erp_lookup", "policy_search", "duplicate_invoice_check"],
  "policy_boundary": ["do_not_release_payment", "cite_policy", "escalate_sanctions_hit"],
  "expected_artifact": "decision_memo",
  "trace_expectations": ["retrieves_policy", "checks_vendor_master", "uses_duplicate_check"],
  "deterministic_checks": ["schema_valid", "no_payment_action", "policy_citation_present"],
  "rubric_id": "ap_exception_review_v3",
  "reviewer_role": "finance_ops_sme"
}

The question text is only one field. The eval must preserve the documents, tools, authority boundaries, expected artifact, and trace expectations needed to reproduce the task.

2. Split the Eval Suite by Decision Use

Suite Purpose Run cadence Examples
Smoke Catch broken prompts/tools/schema quickly Every PR schema validity, tool availability, basic retrieval
Regression Prevent known failures returning Nightly/weekly stale policy, wrong tool, citation drift, refusal errors
Release holdout Gate promotion Before model/prompt/index release unseen work packets, SME-reviewed labels
Shadow production Measure live drift without automation authority Continuous sample real traces scored offline
Red-team/safety Find unacceptable behavior Release and quarterly injection, exfiltration, unauthorized action
Training promotion Convert failures into skill/SFT candidates After review validated repairs, rubric revisions, preference pairs

3. Score the Process, Not Just the Artifact

Minimum metrics:

  • Task success: did the required work complete?
  • Trace compliance: did the agent use the required tools, sources, and approvals?
  • Outcome correctness: did the business state change correctly, or was action withheld correctly?
  • Source governance: were sources authorized, fresh, prioritized, and cited at the right granularity?
  • Cost and latency: did the task meet cost/latency limits for its workflow tier?
  • Recovery: did the agent detect and recover from missing context, tool errors, conflicting sources, or ambiguity?
  • Abstention/escalation: did it ask for help when policy, confidence, or authority required it?
  • Longitudinal quality: did repeated edits, retries, or follow-up tasks erode the codebase, knowledge base, or workflow state?

4. Use Deterministic Validators First

Use executable checks whenever possible:

  • JSON schema validity, required fields, ID match, date format, and citation span checks.
  • SQL/dataframe recomputation for finance and operations metrics.
  • Unit/integration tests for coding agents.
  • Policy rule checks for authorization, payment, claims, credit, HR, or security workflows.
  • Diff checks for code, configuration, wiki, and knowledge-base updates.

Use LLM judges only where the real decision is semantic or qualitative. Even then, record judge model, prompt, rubric version, calibration set, human disagreement, and known failure cases.

5. Preserve Real Enterprise Mess

Corporate datasets should include:

  • stale and conflicting policies;
  • partial, redundant, and mislabeled records;
  • ACL-restricted documents;
  • missing evidence where abstention is correct;
  • near-duplicate vendors/customers/tickets;
  • exception cases and overrides;
  • ambiguous requests from different personas;
  • source-priority conflicts between vendor docs, internal policy, and local practice;
  • repeated-task sequences where the agent should improve or avoid compounding errors.

Clean datasets are useful for debugging. They are not sufficient for production readiness.

Trace Mining and Automatic Skill Improvement

The ASPIRE pattern generalizes if the enterprise has the right guardrails:

flowchart LR
  A["Production or UAT trace"] --> B["Redact and classify"]
  B --> C["Localize failure"]
  C --> D["Synthesize repair"]
  D --> E["Validate on debug cases"]
  E --> F["Run regression suite"]
  F --> G["Promote skill candidate"]
  G --> H["Install governed skill"]
  H --> I["Monitor future traces"]
  I --> B

Promotion Contract

A trace-derived skill should not be admitted unless it has:

  • a failure signature;
  • a when-to-use trigger;
  • a repair strategy;
  • required tools and permissions;
  • source provenance;
  • examples from validated traces;
  • negative examples or do-not-use cases;
  • deterministic checks or SME rubric;
  • rollback or escalation guidance;
  • regression evidence showing it does not damage adjacent workflows.

This is the software analogue of ASPIRE’s robotics loop: traces expose the failing primitive, repair is validated in an executable environment, and only reusable patterns enter the skill library.

Where This Applies Outside Robotics

Domain Trace primitive Skill candidate
Coding agent failing test, patch diff, command trace, review comment “When schema migration changes a required enum, update fixtures and backward-compat tests before touching API handlers.”
Finance ops invoice, PO, ERP lookup, policy retrieval, approval trace “When invoice/vendor address conflicts with vendor master, hold payment and cite vendor-master policy before routing.”
Security review PR diff, scanner output, data-flow trace “When MCP server accepts tool arguments from model output, check prompt-injection and command-injection boundaries.”
LLM red-team eval Garak profile, probes, target model endpoint, threshold, JSONL/HTML/AVID report “When red-teaming an LLM endpoint, preserve benchmark profile, probes, attack-success rate, vulnerable responses, threshold, and run artifact URI.”
CVE/security analysis CVE ID, OSIDB/RHTPA/NVD/osv.dev lookups, CWE/CVSS output, tools-used metadata “When suggesting CVE impact or CVSS, preserve context-source provenance, confidence, model disclaimer, and human validation requirement.”
RAG support query, retrieved chunks, answer, citation spans, user correction “When internal policy conflicts with vendor docs, prefer dated internal policy and surface conflict.”
Procurement contract clause, vendor claim, pricing sheet, risk register “When vendor cites public benchmark only, request private workflow eval evidence and reject score-only claims.”

Corporate Eval Schema Implications

The current StateBench schema family already points in the right direction. The July 2026 source pass adds three fields that should be first-class in future packs:

Field Why
skill_candidate_ref links a failure or repair to a proposed skill, without auto-promoting it
promotion_status separates raw trace, reviewed failure, validated repair, accepted skill, rejected skill, and retired skill
longitudinal_effect captures whether repeated use improved, degraded, or left unchanged the target workflow

The warehouse should continue to centralize normalized rows, not raw private fixtures. But promotion metadata must be centralized enough to prevent two teams from independently adopting the same bad repair.

Procurement and Governance Questions

Ask vendors and internal platform teams:

  1. Which private workflow evals decide model routing today?
  2. Do evals capture full traces, or only final outputs?
  3. Which failures become prompt patches, retrieval fixes, training rows, or skills?
  4. Who approves a trace-derived skill before it is installed?
  5. How are source authority, freshness, and ACLs scored?
  6. Which deterministic validators run before LLM judges?
  7. Which evals test repeated-task degradation?
  8. How are judge prompts and rubrics versioned?
  9. Can a public benchmark score be traced to task specs, verifier logic, and failure modes?
  10. What is the rollback path when a promoted skill causes regression?

Evidence Boundaries

  • Ramp Applied AI and Red Hat AI posts are vendor architecture signals. They are not independent proof of customer ROI or benchmark superiority.
  • Snorkel benchmark/blog posts are high-signal for current eval design patterns, but leaderboard scores require benchmark-specific method review before procurement use.
  • Red Hat prodsec-skills proves a public packaging pattern for skills. It does not prove that 138 skills improve security outcomes without measurement.
  • TrustyAI llama-stack-provider-trustyai-garak proves a public Red Hat-adjacent eval-provider pattern for LLM red-teaming. It does not prove any model is safe or unsafe unless tied to a concrete Garak run artifact, target endpoint, profile, probes, threshold, and report output.
  • Red Hat aegis-ai proves a public security-agent implementation and eval scaffold with context-source tooling. It does not prove autonomous CVE impact/CVSS accuracy without exact eval files, held-out cases, and human validation results.
  • ASPIRE provides strong evidence for robotics trace-to-skill loops. Applying the same loop to corporate knowledge work is a design hypothesis until tested in StateBench or production/UAT traces.
  • Public benchmark scores remain weak evidence for private deployment readiness unless the private workflow, tools, data, policies, and risk tolerance are represented in the eval.

Create trace-to-skill-promotion-v0 with:

  • before/ raw traces from coding-agent, RAG, and finance-workflow failures;
  • expected/ validated skill candidates with failure signature, trigger, repair, negative cases, and regression checks;
  • schema checks for skill_candidate_ref, promotion_status, and longitudinal_effect;
  • forbidden phrases that prevent raw trace summaries from being promoted as skills;
  • a regression fixture where a plausible skill fixes one workflow but breaks another.

That eval would turn this synthesis into an executable guardrail for the repo.