failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

⭐ Must-read starter set (read these first)


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

An LLM-as-Judge Won't Save the Product, Fixing Your Process Will


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Hidden Technical Debt: Agent Evaluation Infrastructure


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Asymmetry of Verification and Verifier's Law


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

How to Build Good Language Modeling Benchmarks


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Benches 2026 — "LLM benchmarks in the era of agents"


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

A Shared Playbook for Trustworthy Third-Party Evaluations


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

A Field Guide to Rapidly Improving AI Products


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

What We Learned from a Year of Building with LLMs


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Why AI evals are the hottest new skill for product builders


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

How evals drive the next chapter in AI for businesses


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Beyond vibe checks: A PM's complete guide to evals


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Predicting model behavior before release by simulating deployment (Deployment Simulation)


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

evals are surprisingly often all you need


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

2 · "If you can eval it, you have built it" — eval ⇄ capability ⇄ RL environment


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

A Taxonomy of RL Environments for LLM Agents


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

What Comes Next with Reinforcement Learning


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Tülu 3: Pushing Frontiers in Open Language Model Post-Training


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Natural Emergent Misalignment from Reward Hacking in Production RL


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Environments Hub: A Community Hub To Scale RL To Open AGI


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

How to fully automate software engineering


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

An FAQ on Reinforcement Learning Environments


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

RL Environments and RL for Science: Data Foundries and Multi-Agent Architectures


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

tau2-bench (τ²-Bench): A Benchmark for Tool-Agent-User Interaction in Real-World Domains


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

3 · The model / harness / skill decomposition


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Measuring AI Ability to Complete Long Tasks


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Turing Post interview ("Open Models Won't Catch Up")


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Harness engineering: leveraging Codex in an agent-first world


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Equipping agents for the real world with Agent Skills


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Effective context engineering for AI agents


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Writing effective tools for agents — with agents


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Same Model, Different Results: Why Coding Agents Aren't Interchangeable


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

What comes next with open models (weights / tools / harness decomposition)


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

4 · Observability & the output / eval space (the surfaces you can grade)

Hidden Technical Debt: Agent Evaluation Infrastructure


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

AI Agent Metrics: How Elite Teams Evaluate


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

OpenTelemetry GenAI Semantic Conventions (agent & framework spans)


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Semantic Conventions for GenAI agent and framework spans


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Inside the LLM Call: GenAI Observability with OpenTelemetry


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

W&B Weave — tracing & evaluation toolkit


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Laminar — open-source observability for AI agents


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

5a · Eval frameworks & harnesses (code-first test-runners)


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

5d · LLM-as-judge / reward / verifier libraries


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

5e · RL-environment / verifiable-reward toolkits (eval ⇄ training)


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

5f · Observability + eval platforms (tracing · datasets · online/offline · CI)


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

OpenTelemetry GenAI semantic conventions


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming)

How to Build Good Language Modeling Benchmarks


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Why We No Longer Evaluate SWE-bench Verified


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

The SWE-bench Illusion: When SOTA LLMs Remember Instead of Reason


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Establishing Best Practices for Building Rigorous Agentic Benchmarks (ABC)


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

About 30% of Humanity's Last Exam Answers Are Wrong


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

SWE-bench: Can LMs Resolve Real-World GitHub Issues?


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Task-Specific LLM Evals that Do & Don't Work


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

A Careful Examination of LLM Performance on Grade School Arithmetic (GSM1k)


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

LiveCodeBench: Holistic and Contamination-Free Evaluation of LLMs for Code


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

LiveBench: A Challenging, Contamination-Limited LLM Benchmark


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

The LLM Evaluation Guidebook (Open LLM Leaderboard team)


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Gaming the System: Goodhart's Law Exemplified in the AI Leaderboard Controversy


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

A Shared Playbook for Trustworthy Third-Party Evaluations


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

7 · Evals & RL environments (verifiers, reward design, difficulty calibration, lifecycle)


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Spurious Rewards: Rethinking Training Signals in RLVR


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Reward Hacking in Reinforcement Learning


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Specification gaming: the flip side of AI ingenuity


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Multi-Turn RL for Multi-Hour Agents — with Will Brown (Prime Intellect)


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Position: The Hidden Costs and Measurement Gaps of RLVR


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

RewardBench 2: Advancing Reward Model Evaluation


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Curriculum RL from Easy to Hard Tasks Improves LLM Reasoning (E2H Reasoner)


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulators


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable)

Evaluating the Effectiveness of LLM-Evaluators


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Creating an LLM-as-a-Judge That Drives Business Results


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

LLM-as-a-Judge: Rethinking Model-Based Evaluations


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

LLMs Instead of Human Judges? A Large-Scale Study


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Statistics for AI/ML, Part 3 — Cohen's Kappa


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

LLM Evaluators Recognize and Favor Their Own Generations


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Weaver: Closing the Generation-Verification Gap with Weak Verifiers


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Agent-as-a-Judge: Evaluate Agents with Agents


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

VerifyBench: A Systematic Benchmark for Evaluating Reasoning Verifiers Across Domains


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Enhancing LLM-as-a-Judge with Grading Notes / From Pilot to Production with Custom Judges


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (CALM framework)


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization)


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

GAIA: A Benchmark for General AI Assistants


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Patterns for Building Cybersecurity Evals


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Statistics for AI/ML, Part 4 — pass@k and Unbiased Estimator


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

WebArena: A Realistic Web Environment for Building Autonomous Agents


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command-Line Interfaces


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risk of Language Models


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Why Do Multi-Agent LLM Systems Fail? (MAST taxonomy)


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

LocAgent: Graph-Guided LLM Agents for Code Localization


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Online-Mind2Web (An Illusion of Progress? Assessing the Current State of Web Agents)


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

WebGames: Challenging General-Purpose Web-Browsing AI Agents


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Berkeley Function Calling Leaderboard (BFCL) V4


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

GTA: A Benchmark for General Tool Agents


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

WindowsAgentArena: Evaluating Multi-Modal OS Agents at Scale


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

ST-WebAgentBench: Evaluating Safety and Trustworthiness in Web Agents


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Remote Labor Index: Measuring AI Automation of Remote Work


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

CORE-Bench: Computational Reproducibility Agent Benchmark


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

BixBench: A Comprehensive Benchmark for LLM-based Agents in Computational Biology


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Gaia2 and ARE: Scaling Up Agent Environments and Evaluations