EnterpriseClawBench: Enterprise Agent Evals from Real Workplace Sessions

Source

Hugging Face paper page: https://huggingface.co/papers/2606.23654
arXiv: https://arxiv.org/abs/2606.23654
PDF: https://arxiv.org/pdf/2606.23654
Project: https://frontisai.github.io/EnterpriseClawBench/
Code: https://github.com/FrontisAI/EnterpriseClawBench

Local mirrors:

• sources/21-benchmarks/enterpriseclawbench-hf-paper-2606.23654-raw.md
• sources/21-benchmarks/enterpriseclawbench-arxiv-2606.23654-raw.md
• sources/21-benchmarks/enterpriseclawbench-project-2606.23654-raw.md
• sources/21-benchmarks/enterpriseclawbench-github-readme-2606.23654-raw.md
• research/21-benchmarks/raw/enterpriseclawbench-workplace-agent-sessions-2606.23654.pdf
• research/21-benchmarks/raw/enterpriseclawbench-workplace-agent-sessions-2606.23654.chandra.txt

What It Is

EnterpriseClawBench is a benchmark construction and evaluation protocol for enterprise agents operating in persistent workspaces. It starts from proprietary workplace sessions where employees chat with an agent, upload files, invoke tools, and expect deliverables to appear in a Linux workspace. The benchmark pipeline converts those sessions into reproducible task packets with recovered fixtures, rewritten prompts, role classes, skill subclasses, expected deliverables, hard rules, semantic rubrics, and sandbox preflight.

The paper reports 852 full tasks and a manually audited 120-task Lite subset. The benchmark data itself is not released because the source sessions contain internal enterprise content. The reusable contribution is therefore the construction protocol, local evaluation pipeline, sandbox run-directory protocol, sanitized public example, reference run, aggregate leaderboard, and visualizations.

Why It Matters

This is a useful missing layer between public agent benchmarks and enterprise eval operations. Most public agent benchmarks either use synthetic tasks, open-source software issues, or simulated SaaS environments. EnterpriseClawBench instead treats real proprietary work sessions as the raw material, while acknowledging that those sessions cannot be published. That makes it directly relevant to decentralized corporate eval programs: each firm can build private benchmark packets from its own work logs, publish only schemas and aggregate metrics, and keep source content local.

The benchmark also reinforces a key StateBench boundary: evaluate the harness-model combination, not just the base model. The paper explicitly reports 32 harness-model combinations and argues that scores depend on the interaction among model behavior, approval flow, trace handling, tool execution, artifact writing, and runtime constraints.

Construction Pipeline

The reported pipeline includes:

1. Raw session recovery from workplace agent logs.
2. Splitting and merging session turns into task candidates.
3. Mechanical gates: length filtering, fixture lookup, redaction recovery, and network-dependency checks.
4. Self-contained review and prompt rewriting into single-turn benchmark tasks.
5. Role-class and skill-subclass assignment.
6. Expected deliverable detection.
7. Hard-rule generation and semantic rubric generation.
8. Sandbox preflight for input upload, agent execution, artifact download, and judge routing.

The paper reports raw TaskInstances flowing through mechanical and self-contained gates before producing the final pool used for role-balanced sampling and full-pool evaluation.

Task and Artifact Coverage

The reported role classes map well to enterprise personas:

Input fixtures include Markdown, docs, images, PDFs, sheets, code, notebooks, slides, and text. Expected deliverables include business artifacts rather than only final answers. That makes the benchmark closer to internal agent productivity evals than QA/coding-only benchmarks.

Evaluation Shape

EnterpriseClawBench scores:

• harness-model combination performance;
• file/artifact delivery;
• text semantic quality;
• visual semantic quality;
• cost;
• runtime;
• skill-transfer behavior;
• role-class and artifact-type slices.

The public repo defines an agent run directory protocol:

evaluation_runs/<run_name>/<task_id>/
  task.json
  prompt.txt
  response.txt
  artifacts/
  result.json
  token_usage.json
  history.jsonl

That run shape is especially useful for our enterprise eval federation design. It cleanly separates task packet, prompt actually sent, final response, generated artifacts, optional runtime metadata, token usage, and audit trace.

Findings

Single scores hide the important behavior. The paper reports that the best Lite configuration reaches 0.663, but its more important claim is that enterprise evals must report cost, runtime, artifact delivery, visual quality, and skill-transfer behavior alongside semantic score.

Harness-model compatibility can dominate model-only capability. The paper describes a Claude-family drop under Hermes relative to Claude Code, DeepAgents, and OpenClaw, attributing the gap to approval checks, blocked execution, delegated subtasks, truncated traces, and incomplete artifact writing. This supports treating approval policy and harness semantics as eval dimensions.

Skill transfer is high variance. The paper reports that skill creator quality, consumer behavior, creator-consumer fit, and baseline score all affect whether skill injection helps or hurts. A single averaged skill score is therefore too lossy; skill evals need a creator-consumer matrix and task-class slices.

Visual artifact evaluation remains immature. Text judge agreement with humans is stronger than visual judge agreement. The paper reports a text-route human agreement audit with MAE 0.134 and Spearman 0.790, while visual agreement is much weaker. That argues for keeping multimodal artifact scoring flagged as high-risk until human calibration improves.

Private-data benchmarks need release-safe patterns. The paper does not release the private benchmark data. It instead publishes the construction/evaluation protocol and release-safe examples. That is probably the right model for enterprise eval programs over sensitive internal chat, files, and tool traces.

Caveats

• The full benchmark data is not public, so external reproduction of reported aggregate scores is limited.
• Some model names and dates in the paper are future/current-cycle claims; treat scores as paper-reported, not independently reproduced.
• The benchmark is valuable as a construction/evaluation protocol even where the leaderboard cannot be independently audited.
• The paper’s visual judge results show the benchmark’s own multimodal scoring is less mature than text scoring.

StateBench Implications

EnterpriseClawBench should update our eval architecture in four ways:

1. Add harness_id, model_id, approval_policy, and runtime_profile as first-class result dimensions.
2. Treat generated artifacts as primary outputs, not attachments to a final answer.
3. Store private task packets locally while publishing only schemas, aggregate scorecards, and redacted exemplars.
4. Evaluate skills by task class and creator-consumer pairing, not a single global skill score.

For GitLab runner federation, this argues for repo-local private task fixtures plus normalized rollups: each repository can run sensitive eval packets inside its own runner and publish only eval_result_v1, aggregate slices, and redacted failure categories to team/firm dashboards.