failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization)

ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

TRAIL: Trace Reasoning and Agentic Issue Localization


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing)

BenchJack: Systematically Auditing AI Agent Benchmarks


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Towards Building Safe & Secure Agentic AI


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Dawn Song — ICLR 2025 keynote on LLM safety


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Adding Error Bars to Evals (A Statistical Approach to LM Evaluations)


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated LLM Agents


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Defeating Prompt Injections by Design (CaMeL)


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

The lethal trifecta for AI agents: private data, untrusted content, and external communication


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Agentic Misalignment: How LLMs Could Be Insider Threats


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

PyRIT — Python Risk Identification Tool for generative AI


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

OWASP Top 10 for Agentic Applications (2026) + LLM Applications (2025)


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

MITRE ATLAS — Adversarial Threat Landscape for AI Systems


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Gray Swan x UK AISI Agent Red-Teaming Challenge


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

How to Construct Domain Specific LLM Evaluation Systems


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Using LLMs as Judges: Insights, Challenges, Best Practices


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Building Metrics that actually work (workshop)


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

From Self-driving to Autonomous Voice Agents


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

On Engineering AI Systems that Endure the Bitter Lesson


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Strategies for LLM Evals (harnesses workshop)


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

3 Key Ideas in AI in 2025 (Verifier's Law)


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Some Intuitions About Large Language Models


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Building LLM Applications for Production


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Evals, error analysis, and better prompts


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Ep 50: A Field Guide to Rapidly Improving AI Products


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Artificial Analysis: Independent LLM Evals


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

#372 It's 2026 and We're Still Talking Evals


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Evaluating AI, Designing for Non-Determinism


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Karpathy: RL is terrible, why benchmarks mislead


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

How to Build AI Evals in 2026 (Step-by-Step)


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Towards Building Safe & Trustworthy AI Agents


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Towards Building Safe and Secure Agentic AI


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Measuring Agent Capabilities and Anthropic's RSP


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Open-Source and Science in the Era of Foundation Models


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.


failure is classified where it occurs in the agentic cycle

raw source evidence is retained outside the normalized row

public benchmark claims are not treated as private enterprise evidence

judge, harness, model, and tool versions are recorded when applicable

Input: Classify a failed AI-agent run using this source's eval concept and preserve the evidence boundary.

Companies & landscape (eval / RL-environment market)