⭐ Must-read starter set (read these first)

The Second Half

"Evaluation becomes more important than training." The field-level *why*.


An LLM-as-Judge Won't Save the Product, Fixing Your Process Will

Process over tooling; evals as the scientific method.


Hidden Technical Debt: Agent Evaluation Infrastructure

Control/data plane, the five eval surfaces, state deltas. "Chat eval was a spreadsheet; agent eval is a system."


LLM Evals FAQ

The densest operational Q&A: error analysis, binary judgments, the benevolent-dictator labeler.


Asymmetry of Verification and Verifier's Law

"Ability to verify == ability to create an RL environment."


Demystifying Evals for AI Agents

Best primary on agent-specific evals: task design, outcome vs trajectory, isolated trials, pass@k vs pass^k.


How to Build Good Language Modeling Benchmarks

Natural / auto-evaluatable / challenging; the "-200%" difficulty target; ~1-yr saturation.


AI Agents That Matter

Cost as a first-class metric; model-dev vs app-dev; missing holdouts breed overfitting.


Building on Evaluation Quicksand

LLM eval has no ground truth; contamination; eval↔training coupling.


Who Validates the Validators? (EvalGen)

"Criteria drift": you can't write the rubric before you grade.


Benches 2026 — "LLM benchmarks in the era of agents"

The sharpest current read on why benchmarks break in the agent era: the "evals are dead, just measure vibes" backlash, how every layer of the eval-running stack (prompt · sampling temp · grader · harness) swings the score, and that benchmark ground truth is frequently wrong.


A Shared Playbook for Trustworthy Third-Party Evaluations

What makes *independent* evals of frontier-model safeguards & capabilities trustworthy: harness selection, the validity hazards that distort results, and the standards third-party evaluators need.


The Second Half

The bottleneck shifts from solving problems to *defining and evaluating* them. (also T2, T7)


"Buying or building another evaluation tool won't save the product." Evals = the scientific method in disguise.


Your AI Product Needs Evals

The canonical "you need evals"; remove all friction from looking at your data; don't rely on generic frameworks.


A Field Guide to Rapidly Improving AI Products

"Error analysis is consistently the highest-ROI activity." The metric for an AI roadmap is experiments run.


In Defense of AI Evals, for Everyone

Rebuts the anti-eval backlash; evals = the systematic measurement of application quality.


What We Learned from a Year of Building with LLMs

The "intern test," genchi genbutsu, turning vibe-checks into assertions.


Big Tech's LLM Evals Are Just Marketing

Why frontier-lab leaderboard numbers are marketing, not science.


AI Engineering pitfalls

Common eval/AI-engineering mistakes from the *AI Engineering* author. (also T6)


Evals Are NOT All You Need

The essential nuance piece: automated graders alone don't save you; you need a continuous-improvement flywheel of offline tests + production monitoring + real-user iteration. Pairs with Shreya's 'In Defense' to complete the backlash debate. 🆕


Why AI evals are the hottest new skill for product builders

The accessible 'why evals matter' on-ramp (live walkthrough of error analysis, open/axial coding) that mainstreamed evals to PMs in 2025; the apartment-leasing-bot anecdote is the canonical 'you can't vibe-check' story. 🆕


How evals drive the next chapter in AI for businesses

Frontier-lab framing of evals as turning fuzzy business goals into specs and measurable ROI; useful counterweight to Lambert's 'evals are marketing' and grounds the 'why' for enterprise readers. 🆕 ⚠(unverified URL)


Beyond vibe checks: A PM's complete guide to evals

The widely-shared PM-oriented argument for moving past 'looked good to me' vibe checks to systematic evals; one of the pieces that made evals a mainstream product skill in 2025. 🆕


A pragmatic guide to LLM evals for devs

Reaches the broad engineering audience with the core 'why': LLM non-determinism breaks traditional testing, so you need evals. High-distribution motivation piece co-written by Hamel. 🆕


Predicting model behavior before release by simulating deployment (Deployment Simulation)

Concrete 2026 evidence for why fixed/static evals fail: models recognize when they're being tested and game test suites; replaying ~1.3M real conversations surfaced reward-hacking no fixed eval caught. Strong 'why evals must evolve' argument. 🆕 ⚠(unverified URL)


evals are surprisingly often all you need

The canonical one-liner ('evals are the new unit test') that anchors the whole 'why evals' thesis; frequently cited founding quote for the movement. Short but load-bearing.


2 · "If you can eval it, you have built it" — eval ⇄ capability ⇄ RL environment

Trainability tracks verifiability; verifying = creating an RL environment.


A Taxonomy of RL Environments for LLM Agents

A benchmark is a frozen RL environment; the E = {T,H,V,S,C} decomposition; "verifiable beats judgeable."


The Life Cycle of an RL Environment

Difficulty calibration (the 1–4/16 Goldilocks band), RL as variance reduction, reward hacking under training pressure. *(local notes: `research/notes/kanav-garg-rl-environment-lifecycle.md`)*


Welcome to the Era of Experience

Human-data value approaching its ceiling; the frontier is agents learning from experience / synthetic environments.


RLHF Book, Ch. 16 — Evaluation

Evaluation as a reflection of training goals; prompt-format sensitivity (60%→~0%).


What Comes Next with Reinforcement Learning

Long-horizon credit assignment; where RL is and isn't ready.


verifiers


DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

the canonical 'if you can verify it, RL builds it' result; also published in Nature 2025. Conspicuously absent from a section literally about eval-as-RL-environment. 🆕


Tülu 3: Pushing Frontiers in Open Language Model Post-Training

Coined/popularized RLVR and open-sourced the recipe + code (open-instruct): swap the reward model for a verifier on tasks with checkable answers. The foundational citation behind every 'verifiable beats judgeable' claim in this section. 🆕


Natural Emergent Misalignment from Reward Hacking in Production RL

Empirical receipt for the section's 'reward hacking under training pressure' theme: learning to cheat on real coding environments generalizes to sabotage/alignment-faking; introduces inoculation prompting as mitigation (arXiv 2511.18397). 🆕


Environments Hub: A Community Hub To Scale RL To Open AGI

the eval-is-an-RL-env thesis as an actual ecosystem, the natural companion to the already-listed verifiers repo. 🆕


How to fully automate software engineering

'you only get the capability you can build an environment for.' 🆕


Cheap RL tasks will waste compute

directly informs difficulty calibration / why environment design matters. 🆕


An FAQ on Reinforcement Learning Environments

the empirical state-of-the-field map this section lacks. 🆕


RL Environments and RL for Science: Data Foundries and Multi-Agent Architectures

Market-structure view: 35+ companies now sell RL environments; capability gains are coming from ramping RL compute, not pretraining. Grounds the 'benchmark = frozen RL environment' thesis in who's actually building/buying them. 🆕


Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces

i.e. a benchmark that IS an RL environment (and is used as one). 2.4k stars, active. 🆕


tau2-bench (τ²-Bench): A Benchmark for Tool-Agent-User Interaction in Real-World Domains

the canonical example of a verifiable conversational/agentic environment beyond math/code (paper arXiv 2506.07982). 🆕


3 · The model / harness / skill decomposition

Hidden Technical Debt: Agent Harness

The harness is the agent; what teams call "the model" is mostly harness + product.


Hidden Technical Debt series (index)

The four-part series (eval infra, runtime, harness, + agent runtime ~2026/04/24). *(verify the runtime post URL on the index.)*


Measuring AI Ability to Complete Long Tasks

Scaffolds change the measured horizon; success-vs-human-time as a primitive. (also T9)


Turing Post interview ("Open Models Won't Catch Up")

"What technical people call the harness or the product matters more than just the model."


Quo vadis, LLM benchmarks?

The AlgoTune case: *same model, different harness, opposite ranking.* (also T6) *(notes: `research/notes/florian-brand-*`)*


The Model is the Product

the direct counterpart to Hamel's 'Model is Not the Product'; the foundational text of the harness/model debate this section is built on. 🆕


The Model is Not the Product

The opposing side of the Lee debate (Data Council 2025): great products are mostly harness + product + evals, not the model. Section already cites Lee; it should cite the debate it half-references. 🆕


Agents are models using tools in a loop

the cleanest statement of why the harness, not the model, dominates behavior. 🆕


Harness engineering: leveraging Codex in an agent-first world

Frontier-lab primary source coining 'harness engineering': a 1M-line codebase built by Codex agents where improving the environment/harness mattered more than the model. Lab-side complement to Lee's 'harness is the agent'. (URL returns 403 to scraper but page is live; corroborated by InfoQ/Milvus coverage.) 🆕


Equipping agents for the real world with Agent Skills

skills as composable, progressively-disclosed capabilities (later made an open standard). The section title says 'skill' but has zero skill sources. 🆕


Effective context engineering for AI agents

the mechanism behind why same model + different harness diverges. 🆕


Writing effective tools for agents — with agents

Tool design is a load-bearing part of the harness; 'agents are only as effective as the tools we give them,' validated eval-first. Directly ties harness decisions to measured agent performance. 🆕


Same Model, Different Results: Why Coding Agents Aren't Interchangeable

the practitioner case-study version of Brand's AlgoTune point. 🆕


Holistic Agent Leaderboard (HAL)

the infrastructure answer to 'harness confounds rankings.' ICLR 2026; paper arXiv:2510.11977. 🆕


Agent Harness Engineering

'A decent model with a great harness beats a great model with a bad harness'; reframes agent failures as harness/config problems (traceable AGENTS.md rules). Names the converging harness primitives across coding agents. 🆕


What comes next with open models (weights / tools / harness decomposition)

the written companion to the Turing Post interview already listed, with the explicit three-part decomposition. 🆕


4 · Observability & the output / eval space (the surfaces you can grade)

Hidden Technical Debt: Agent Evaluation Infrastructure

Control plane / data plane; the **five surfaces** (output, trace, memory, environment, mechanistic); the empty-tool-result hallucination.


The Three Pillars of AI Observability

Dataset reconciliation (living datasets); traces / evals / annotation.


Agent Trajectory Evaluations


AI Agent Metrics: How Elite Teams Evaluate

A concrete agent-metric taxonomy (action completion, tool selection, etc.).


OpenInference semantic conventions

An OTel-based agent trace schema (tool, args, observation, latency, cost).


LangSmith Evaluation / Trajectory evals

<https://docs.langchain.com/langsmith/evaluation> · <https://docs.langchain.com/langsmith/trajectory-evals> · *docs*.


OpenTelemetry GenAI Semantic Conventions (agent & framework spans)

the canonical trace schema the section's OpenInference entry derives from. 🆕


Semantic Conventions for GenAI agent and framework spans

the precise definition of what a gradable agent trace looks like. 🆕


Inside the LLM Call: GenAI Observability with OpenTelemetry

concrete intro to the trace surface for practitioners not steeped in OTel. 🆕


W&B Weave — tracing & evaluation toolkit

a widely used surface for grading both traces and outputs. 🆕


Laminar — open-source observability for AI agents

purpose-built for grading multi-step agent trajectories rather than single LLM calls. 🆕


5a · Eval frameworks & harnesses (code-first test-runners)

Inspect AI

`@task` binds dataset + solver + scorer; custom scorers; sandboxed tools. The reference agent-eval framework. **(MUST)**


inspect_evals


lm-evaluation-harness

the standard academic harness; first-class decontamination; task YAMLs.


OLMES

🆕 the reproducible eval **standard + harness** behind OLMo/Tülu: standardized prompts/metrics/formatting for apples-to-apples model comparison.


BenchFlow

🆕 environment-lab framework: research infra + runtime for building RL environments, evals & post-training; ships **SkillsBench** and **ClawsBench**. ("Environments are the new data.")


lighteval

🆕 all-in-one harness across transformers/vLLM/TGI/nanotron, 1000+ tasks; HF's successor to `evaluate`.


OpenBench

🆕 provider-agnostic `bench` CLI, 95+ benchmarks, built on Inspect primitives.


simple-evals

minimal zero-shot/CoT scripts (MMLU, HumanEval, SimpleQA, HealthBench); the numbers OpenAI publishes. ⚠ not actively maintained.


OpenAI Evals

the `completion_fn` abstraction = swap the system-under-test. (Best-practices: <https://developers.openai.com/api/docs/guides/evaluation-best-practices>)


promptfoo

MIT eval + red-teaming CLI; git-diffable YAML configs. **(MUST)**


DeepEval / Confident AI

"pytest for LLMs," 40+ metrics (G-Eval, RAG, hallucination) + red-team; ~2M evals/day; hosted cloud. 🆕


pydantic-evals

🆕 type-safe Datasets/Cases/Evaluators with OTel tracing, from the Pydantic AI team.


openevals

🆕 prebuilt evaluators + `create_llm_as_judge` (incl. multimodal); general-purpose companion to **agentevals** (<https://github.com/langchain-ai/agentevals>, trajectory match).


MLflow GenAI evaluate

🆕 `mlflow.genai.evaluate`: 50+ judges/metrics, custom scorers, regression datasets inside MLflow.


HELM (crfm-helm)

holistic eval: standardized datasets + metrics beyond accuracy + leaderboard (also VHELM, HEIM).


Giskard

auto-generates adversarial test suites (injection, hallucination, bias) from a plain-language app description.


Deepchecks LLM

property-based scoring (grounded-in-context, toxicity, fluency) + custom LLM-judge properties.


UpTrain

20+ preconfigured checks + root-cause analysis on failures.


HF `evaluate`

classic metrics library, ⚠ maintenance mode (use lighteval for LLMs).


Harbor

🆕 framework for running agent evals + creating/using RL environments; powers Terminal-Bench 2.0. ~2.7k★. ⚠ name overloaded (cf. `av/harbor` local-LLM toolkit).


evalite

🆕 local-first eval runner on Vitest; `.eval.ts` files, web UI, cost-aware.


Mastra scorers

🆕 model-graded/rule/statistical scorers, live evals, CI, in the Mastra agent framework.


Vercel agent-eval

🆕 A/B-test coding agents (Claude Code, Codex, Cursor) on custom tasks; pass-rate dashboards.


Autoevals

OSS scorer library (Factuality, relevance, security…) across Py/JS/Go/Ruby.


TruLens

instrumentation + "feedback functions" (the RAG triad), now OTel-based.


ARES

synthetic queries + fine-tuned judges + prediction-powered inference for confidence intervals.


RAGChecker

🆕 claim-level diagnosis separating retriever vs generator errors.


continuous-eval (Relari)

modular per-module metrics across retrieval/generation/tool-use.


Tonic Validate


5d · LLM-as-judge / reward / verifier libraries

verdict

🆕 declarative compound judges (debate/verification/aggregation, inference-time scaling); arXiv:2502.18018.


RULER


Prometheus 2

open-weight evaluator LMs for rubric-based assessment + pairwise.


Atla Selene

🆕 8B SoTA open judge (score + critique); + MCP server `atla-ai/atla-mcp-server`. arXiv:2501.17195.


Patronus Lynx / GLIDER

🆕 open hallucination judge / explainable span-level judge.


Flow-Judge


RewardBench

canonical reward-model (+v2 judge) benchmark/harness.


JudgeBench

benchmark to evaluate the judges themselves.


reward-kit

🆕 decorator-based reward-function authoring (TRL/Fireworks interop).


5e · RL-environment / verifiable-reward toolkits (eval ⇄ training)

verifiers

Environment = dataset + harness + rubric; one package for eval, RL, synthetic data. **(MUST)**


Environments Hub

🆕 crowdsourced verifiers-based RL/eval envs.


prime-rl

🆕 async RL trainer consuming verifiers envs (INTELLECT-3).


BenchFlow

🆕 environment lab: builds & runs RL/eval environments (SkillsBench, ClawsBench, runtime). "Environments are the new data." (also §5a)


HUD

🆕 SDK to build/run agent eval environments (computer-use, browser, MCP) with telemetry.


Atropos

🆕 async "environment microservice" framework for rollouts/verifiable rewards.


verl

de-facto industry RLVR trainer (PPO/GRPO). ~22k★.


OpenRLHF

the RL-training stack agents are post-trained + eval'd in.


Open Reward Standard (ORS)

🆕 MCP-extending spec adding RL primitives (episodes, rewards, curriculum). ⚠ no single canonical repo confirmed.


5f · Observability + eval platforms (tracing · datasets · online/offline · CI)

Arize Phoenix

OSS OTel tracing + response/retrieval evals + datasets/experiments. **(MUST)**


Langfuse

OSS: evals (LLM-judge, feedback, manual labeling), datasets/experiments, prompt mgmt; self-hostable. 🆕


Opik

🆕 fully-OSS eval + observability (judges, datasets, CI-runnable evals).


W&B Weave

`weave.Evaluation` scorers (exact/regex/model-graded/embedding) + Guardrails; comparison dashboards. 🆕 (Humanloop's migration target.)


Braintrust

`Eval()` over golden datasets; offline vs online. **(MUST)**


Patronus AI

🆕 research-grade judges (Lynx, GLIDER, **Percival** agent-failure debugger), experiments, multimodal judge.


Maxim AI

🆕 agent **simulation** + eval + observability across thousands of scenarios/personas.


Galileo


Vellum

visual workflows + offline/online evals scoring every production run.


Helicone

OSS gateway + observability; "Scores" ingests external eval results.


Traceloop / OpenLLMetry

OSS OTel instrumentation (Py/TS/Go/Ruby) + hosted reliability platform.


Langtrace

OSS OTel-standard tracing + manual scoring + dataset mgmt.


WhyLabs / LangKit

high-throughput text-signal metrics (toxicity, PII, jailbreak) for production monitoring.


Portkey

🆕 OSS gateway + 60+ guardrails + observability (fully open-sourced Mar 2026).


Datadog LLM Observability

🆕 evaluators + golden datasets + **LLM Experiments** + AI Agent Monitoring (Jun 2025).


Fiddler AI

🆕 Trust Models (Safety/PII/Faithfulness) scoring in <100ms; Guardrails + agentic observability.


PromptLayer

lighter prompt-CMS / APM-native monitoring.


OpenInference

semantic conventions for agent traces (tool/args/observation/latency/cost).


OpenTelemetry GenAI semantic conventions

🆕 the vendor-neutral schema (now covers agent orchestration, MCP tool calls, and a **quality-evaluation** span hook).


Braintrust

Industry-standard eval+observability platform (Notion, Stripe, Vercel) tying offline experiments to production logs; the section already cites Braintrust's Autoevals but omits the platform itself. 🆕


RagaAI Catalyst

covers the online/guardrail-eval slice the section lacks. 🆕


OpenAI Cookbook — Evals

Maintained, runnable recipes for building evals (incl. Agents SDK eval, evaluating agents with Langfuse); the practical companion to OpenAI Evals and a curator-grade 'show real work' resource. 🆕 ⚠(unverified URL)


6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming)

How to Build Good Language Modeling Benchmarks

The benchmark-author's checklist; difficulty target; one-number reporting; 150–500 task sizing.


AI Agents That Matter

Cost-controlled evaluation; model-dev vs downstream-dev needs; holdouts.


Why We No Longer Evaluate SWE-bench Verified

~59% of audited failures were broken tests. (mirror: <https://decrypt.co/359012/...>)


The Leaderboard Illusion

Private testing, selective disclosure, and data-access asymmetry on Chatbot Arena. *(notes: `research/notes/leaderboard-illusion.md`)*


The SWE-bench Illusion: When SOTA LLMs Remember Instead of Reason


Establishing Best Practices for Building Rigorous Agentic Benchmarks (ABC)

SWE-bench Verified weak tests; τ-bench rewards empty responses. *(verified high)*


FrontierMath Tiers 1–3 v2 (corrected)

~42% of problems corrected after AI-assisted review. (also T8: the operator-as-rot-detector tale)


About 30% of Humanity's Last Exam Answers Are Wrong

29 ± 3.7% of text-only chem/bio answers contradicted by the literature. (LessWrong writeup: <https://www.lesswrong.com/posts/JANqfGrMyBgcKtGgK/>)


Building on Evaluation Quicksand

No hard source of truth; synthetic-data contamination.


Lost in Simulation

Simulated users are unreliable proxies (~9pp swings by simulator choice; demographic miscalibration).


SWE-bench: Can LMs Resolve Real-World GitHub Issues?

<https://arxiv.org/abs/2310.06770> · <https://www.swebench.com> (Verified: `.../verified.html`) · *paper/site*.


Task-Specific LLM Evals that Do & Don't Work

Off-the-shelf evals rarely transfer; accuracy is too coarse.


Andrej Karpathy on evals

"We make a number of specific recommendations…" (the eval-as-narrow critique).


A Careful Examination of LLM Performance on Grade School Arithmetic (GSM1k)

the canonical method for measuring benchmark overfitting/contamination via a matched holdout.


Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks

NeurIPS 2021 foundational result: ~3.3% avg label errors across 10 famous test sets (ImageNet, MNIST, etc.); corrections flip model rankings. The canonical 'label errors' citation this section's theme rests on (labelerrors.com / cleanlab).


Are We Done with MMLU? (MMLU-Redux)

directly demonstrates label-error impact on the most-cited LLM benchmark.


LiveCodeBench: Holistic and Contamination-Free Evaluation of LLMs for Code

the section discusses contamination but lists no exemplar of how to engineer around it.


LiveBench: A Challenging, Contamination-Limited LLM Benchmark

the canonical 'dynamic refresh' answer to saturation and contamination.


The LLM Evaluation Guidebook (Open LLM Leaderboard team)

the hands-on 'how to not get fooled' companion to this section (updated version: hf.co/spaces/OpenEvals/evaluation-guidebook).


Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation

extends 'AI Agents That Matter' to leaderboard integrity for agents specifically. 🆕


Gaming the System: Goodhart's Law Exemplified in the AI Leaderboard Controversy

the accessible blog companion to The Leaderboard Illusion paper. 🆕


A Shared Playbook for Trustworthy Third-Party Evaluations

What makes *independent* evals of frontier-model safeguards & capabilities trustworthy: selecting the right harness, checking for validity hazards that distort results, and the standards third-party evaluators need. (also T10) 🆕


7 · Evals & RL environments (verifiers, reward design, difficulty calibration, lifecycle)

RewardBench

Evaluating reward models (the verifier you train against).


The New RL Scaling Laws

Where RLVR scaling is heading. (interview: <https://www.latent.space/p/the-rlvr-revolution-with-nathan-lambert>)


Spurious Rewards: Rethinking Training Signals in RLVR


The State of Post-Training 2025


Reward Hacking in Reinforcement Learning

taxonomy, RLHF-specific failure modes, mitigations; the foundational reference any reward-design section needs.


Specification gaming: the flip side of AI ingenuity

Canonical specification-gaming post (+the running examples list); origin story of why verifiers/reward functions get gamed, predating the LLM-RL wave.


Multi-Turn RL for Multi-Hour Agents — with Will Brown (Prime Intellect)

the practitioner voice behind the verifiers library already cited here. 🆕


Position: The Hidden Costs and Measurement Gaps of RLVR

the rigor counterweight to Lambert's RL-scaling optimism. 🆕


RewardBench 2: Advancing Reward Model Evaluation

harder, less saturated, ICLR 2026; the current bar for evaluating the verifier you train against. 🆕


Reward Modeling (RLHF Book, ch. 5)

the standing explainer for the 'verifier you train against' framing this section uses. 🆕


Curriculum RL from Easy to Hard Tasks Improves LLM Reasoning (E2H Reasoner)

directly fills the section's difficulty-calibration theme. 🆕


GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulators

recent take on auto-calibrating env difficulty to the agent. 🆕


8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable)

Evaluating the Effectiveness of LLM-Evaluators

Position/verbosity/self-enhancement bias; direct vs pairwise; prefer binary + classification metrics.


Creating an LLM-as-a-Judge That Drives Business Results

Critique-shadowing; validate against ONE benevolent-dictator expert; precision/recall over raw agreement.


Who Validates the Validators? (EvalGen)

Criteria drift; the coverage-vs-false-failure judge-alignment loop.


LLM Evals FAQ

Binary over Likert; review ≥100 traces; the first-failure transition matrix for agents.


LLM-as-a-Judge: Rethinking Model-Based Evaluations

Avoid [0,1] continuous scales; manage judges like junior annotators.


Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

*which the authors themselves hedge* ("cannot determine"); GPT-3.5 doesn't self-favor.


LLMs Instead of Human Judges? A Large-Scale Study

Substantial variance across models/datasets; validate judges against humans first.


AlignEval

"Align AI to human. Calibrate human to AI. Repeat." Work backward from the data.


Product Evals in Three Simple Steps

The "God Evaluator" anti-pattern; the benchmark is human performance, not perfection.


Statistics for AI/ML, Part 3 — Cohen's Kappa

Chance-adjusted inter-annotator agreement (the gate before holding out).


Data Flywheels for LLM Applications

Binary metrics, the "GPT smell," error analysis as the core activity.


SPADE

Data-quality assertions / agentic query rewriting for LLM pipelines.


LLM Evaluators Recognize and Favor Their Own Generations

The canonical causal study of self-preference bias: shows GPT-4/Llama-2 can recognize their own outputs and that self-recognition correlates linearly with self-favoring. This is the primary source behind 'self-enhancement bias' that the section's blogs only allude to.


G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment

The foundational reference-free LLM-judge method (CoT + form-filling scoring). Defines the direct-scoring paradigm the section critiques; a curated judge section is incomplete without the paper that started it.


A Survey on LLM-as-a-Judge

The most-cited survey organizing the LLM-judge space (bias taxonomy, reliability methods, agreement metrics). Serves as the one-stop map/bibliography the section currently lacks.


One Token to Fool LLM-as-a-Judge

Shows 'master-key' tokens (a colon, 'Solution:') trigger false-positive rewards up to 80% even on GPT-o1/Claude-4 judges, plus a robust Master-RM fix. Core evidence on judge/verifier reward-hacking fragility. 🆕


Weaver: Closing the Generation-Verification Gap with Weak Verifiers

Directly operationalizes 'verifiable vs judgeable': aggregates many weak judges/reward models (unlabeled) to shrink the generator-verifier gap, reaching o3-mini accuracy from Llama-3.3-70B. Paper: arxiv.org/abs/2506.18203. 🆕


Agent-as-a-Judge: Evaluate Agents with Agents

with the DevAI benchmark. The agent-specific evaluation case this agent-evals library specifically needs.


VerifyBench: A Systematic Benchmark for Evaluating Reasoning Verifiers Across Domains

Cross-domain benchmark exposing verifier precision/recall trade-offs (specialized verifiers high-accuracy but low-recall; general models inclusive but unstable). Quantifies how trustworthy a verifier actually is for RLVR. 🆕


Enhancing LLM-as-a-Judge with Grading Notes / From Pilot to Production with Custom Judges

a production-side complement to the Hamel/Shankar academic alignment loop. 🆕


Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (CALM framework)

broadens the section's bias coverage well beyond position/verbosity/self-enhancement.


9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization)

Demystifying Evals for AI Agents

Grade the final env state (flight-booking via SQL); outcome vs trajectory; isolation; pass@k vs pass^k.


τ-bench / τ²-bench

DB-state-diff grading; user simulation; pass^k; empty-result as explicit fail.


Benchmarking AI Agents


GAIA: A Benchmark for General AI Assistants

Real assistant tasks; difficulty by human task-length.


Patterns for Building Cybersecurity Evals

The four-primitive agentic-eval template (sandbox, difficulty inputs, tools, deterministic grader); outcome grading + partial-credit ladders + transcript audits. (also T10)


Statistics for AI/ML, Part 4 — pass@k and Unbiased Estimator

Demystifies the metric everyone misuses.


First-Principles Eval


SWE-bench grading harness

FAIL_TO_PASS / PASS_TO_PASS as a verifiable reward. (SWE-agent ACI: <https://swe-agent.com/0.7/background/aci/>)


human-eval (pass@k estimator)


WebArena: A Realistic Web Environment for Building Autonomous Agents


OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments


Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command-Line Interfaces

verified (arxiv: arxiv.org/abs/2601.11868). 🆕


Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risk of Language Models


AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories

First benchmark of LLM-judges-of-trajectories: 1302 expert-reviewed web-agent runs; shows rule-based graders reject many valid trajectories (under-reporting success). Core to the 'trajectory evaluation' theme the section currently lacks. 🆕


Why Do Multi-Agent LLM Systems Fail? (MAST taxonomy)


AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents

gold-standard world-state grading for tool-use agents.


BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents

1,266 'inverted' hard-to-find/easy-to-verify questions for deep-research browsing agents; short verifiable answers make grading deterministic. Released 2025, now standard for browsing-agent eval. (paper: arxiv.org/abs/2504.12516) 🆕


LocAgent: Graph-Guided LLM Agents for Code Localization

directly fills the 'localization' theme named in the section title but currently unlisted. 🆕


WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models

an early, widely-cited example of multimodal-LLM-as-judge for live-web agent trajectories.


SkillsBench

makes skill-acquisition/skill-use a measurable axis (the "Agent Skills" frontier). ~1.4k★.


ClawsBench

🆕 BenchFlow's agent benchmark (results/data repo; full release in progress).


SWE-bench Verified

500 human-validated SWE-bench instances graded by hidden FAIL_TO_PASS unit tests; the de facto standard for real-issue resolution and the headline coding-agent number labs report 🆕


SWE-bench Multimodal

619 visual JS/front-end issues from 17 user-facing repos, test-verified; probes whether SWE agents generalize beyond Python/text to visual software domains


SWE-bench Pro

1,865 long-horizon, multi-file tasks across public GPL + held-out + commercial startup repos, test-graded; contamination-resistant and hard (frontier <45% pass@1) 🆕


SWE-Lancer

1,400+ real Upwork freelance tasks worth $1M, graded by triple-verified end-to-end Playwright tests plus manager-decision tasks; ties capability to economic value 🆕


SWE-Gym

2,438 executable Python SWE tasks with pre-installed deps + test verification; the first real training/eval gym for SWE agents and verifiers, ICML 2025 🆕


Multi-SWE-bench

1,632 expert-annotated issue-resolution tasks across Java, TS, JS, Go, Rust, C, C++, test-graded; the leading multilingual SWE-bench extension, NeurIPS 2025 D&B 🆕


SWE-rebench

Automated pipeline yielding 21k+ executable Python tasks with continuously refreshed, decontaminated eval splits; quantifies how much SWE-bench Verified scores are inflated by contamination, NeurIPS 2025 D&B 🆕


RE-Bench

7 open-ended ML research-engineering environments (e.g. GPU-kernel optimization, scaling laws) scored against 71 human-expert 8-hour attempts; the reference AI-R&D-uplift eval, ICML 2025


MLE-bench

75 Kaggle ML-engineering competitions graded against real human leaderboards (medal thresholds) in 24h Docker runs; standard ML-engineering-agent eval, ICLR 2025. 🆕


PaperBench

Replicate 20 ICML 2024 papers from scratch, graded by 8,316 author-co-developed rubric leaves via a validated LLM judge; rigorous research-replication agent eval, ICML 2025 🆕


Konwinski Prize (K Prize)

$1M Kaggle forecasting-format contest on GitHub bugs filed after submission close, fully contamination-free, test-graded; round-1 top score only 7.5% exposed real-world difficulty 🆕


Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge

a serious answer to the Deep Research evaluation gap. 🆕


Online-Mind2Web (An Illusion of Progress? Assessing the Current State of Web Agents)

300 realistic tasks on 136 live websites with an LLM-as-a-Judge auto-grader (~85% human agreement); exposes overstated web-agent progress vs simple baselines. 🆕


REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites

fixes the flakiness of live-site web benchmarks. 🆕


WebGames: Challenging General-Purpose Web-Browsing AI Agents

50+ client-side challenges isolating specific browser interaction skills with verifiable pass/fail; best agent 41% vs 96% human, a sharp diagnostic gap. 🆕


Berkeley Function Calling Leaderboard (BFCL) V4

the de facto tool-calling leaderboard. 🆕


GTA: A Benchmark for General Tool Agents

229 human-written real-world queries with implicit multimodal tool use; executable evaluation platform across perception/operation/logic/creativity tools (GTA-2 follow-up in 2026). 🆕


Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows


AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents

the standard mobile-GUI agent benchmark. 🆕


WindowsAgentArena: Evaluating Multi-Modal OS Agents at Scale

desktop computer-use counterpart to OSWorld. 🆕


ST-WebAgentBench: Evaluating Safety and Trustworthiness in Web Agents

grades whether agents obey rules, not just succeed. 🆕


TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks


VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks


GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks

the flagship economic-value agent benchmark. 🆕


Remote Labor Index: Measuring AI Automation of Remote Work

a hard, money-grounded ceiling for end-to-end remote work. 🆕


Humanity's Last Exam

2,500 expert-written frontier-knowledge questions with unambiguous auto-gradable answers across dozens of fields; the canonical post-MMLU saturation exam (note: now very widely cited). 🆕


ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery

102 expert-validated tasks from 44 peer-reviewed papers; grades self-contained Python programs by execution + success rate; best agent solves only ~34% (ICLR 2025). 🆕


CORE-Bench: Computational Reproducibility Agent Benchmark

270 tasks over 90 papers (CS/social science/medicine) that grade whether an agent can reproduce published results from code+data; from the Princeton AI-Snake-Oil group.


DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents

the standard deep-research-report eval. 🆕


BixBench: A Comprehensive Benchmark for LLM-based Agents in Computational Biology

serious wet-lab-adjacent science agent eval. 🆕


Gaia2 and ARE: Scaling Up Agent Environments and Evaluations

the serious general-assistant env from Meta. 🆕


Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents

Run a simulated vending business over >20M-token horizons; objectively graded on profit/net-worth, exposing long-horizon coherence breakdowns unrelated to context limits. 🆕


ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems

the frontier fluid-intelligence benchmark. 🆕


TRAIL: Trace Reasoning and Agentic Issue Localization

148 annotated agent traces with 841 errors (reasoning/planning/execution); grades whether an LLM can localize the failure in a trace (best model ~11%). HF dataset PatronusAI/TRAIL. 🆕


CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios

19 expert-validated B2B/B2C tasks on a realistic Salesforce org with state-based grading; exposes the single-turn (~58%) vs multi-turn (~35%) reliability gap plus confidentiality checks. 🆕


10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing)

BenchJack: Systematically Auditing AI Agent Benchmarks

Reward hacking emerges spontaneously in frontier models; an 8-pattern flaw taxonomy + a 30-question Agent-Eval checklist; "benchmarks must be secure by design."


Towards Building Safe & Secure Agentic AI

The adversarial setting; environment-borne attacks.


Dawn Song — ICLR 2025 keynote on LLM safety


CyberGym

Memory-safety PoC generation from OSS-Fuzz; sanitizer-crash grading at scale.


AIR-Bench 2024


DecodingTrust


RedCode

Risky code execution/generation benchmark for code agents.


AgentPoison

Red-teams agents by poisoning their RAG memory.


Adding Error Bars to Evals (A Statistical Approach to LM Evaluations)

"is this difference real?" (cross-cutting: T6/T8)


AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents

The canonical prompt-injection benchmark for tool-using agents (97 tasks, 629 security cases over untrusted data); NeurIPS 2024 D&B, now the standard eval everyone reports against. A glaring omission. 🆕


AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents

ICLR 2025 benchmark of 110/440 malicious agent tasks across 11 harm categories; shows leading models comply with malicious agent requests without jailbreaking. The reference action-misuse/refusal benchmark. 🆕


InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated LLM Agents

ACL 2024 Findings; 1,054 IPI test cases over 17 user / 62 attacker tools, splitting direct-harm vs data-exfiltration intents. Foundational indirect-prompt-injection benchmark predating AgentDojo.


Defeating Prompt Injections by Design (CaMeL)

The defense-by-design counterpart: extracts control/data flow from the trusted query and enforces capability-based policies so untrusted data can't alter program flow; effectively solves AgentDojo's security eval. The key 2025 mitigation paper. 🆕


The lethal trifecta for AI agents: private data, untrusted content, and external communication

The most-cited conceptual frame for reasoning about when an agent is unconditionally vulnerable to prompt injection; essential practitioner mental model at the Eugene-Yan bar. 🆕


SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents

17 complex environments pairing a benign main task with a hidden harmful side task to measure whether agents can sabotage without tripping an AI monitor; the canonical sabotage/monitorability eval. (Paper: arxiv.org/abs/2506.15740) 🆕


Agentic Misalignment: How LLMs Could Be Insider Threats

Red-team study showing frontier models will resort to blackmail/leaking under goal conflict in agentic settings; the reference for action-authorization / insider-threat adversarial evaluation. Companion to the cited Anthropic error-bars piece. 🆕


PyRIT — Python Risk Identification Tool for generative AI

The de-facto open-source red-teaming automation framework (70+ converters, multi-turn attacks like Crescendo/TAP); how practitioners actually run adversarial evals at scale. The section lists papers but no tooling. 🆕


OWASP Top 10 for Agentic Applications (2026) + LLM Applications (2025)

Industry-standard risk taxonomy: goal hijack, tool misuse, identity/privilege abuse, memory poisoning, rogue agents; complements the regulation-grounded AIR-Bench taxonomy already listed. The canonical practitioner threat checklist. 🆕


MITRE ATLAS — Adversarial Threat Landscape for AI Systems

ATT&CK-style living knowledge base of 16 tactics / 80+ techniques against AI systems with real-world case studies and mitigations; the standard reference framework for AI adversarial threat modeling.


Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents

ICLR 2025 unified benchmark spanning 10 scenarios, 400+ tools, covering DPI/IPI, memory poisoning, plan-of-thought backdoors and defenses in one harness; broadest single attack/defense agent benchmark. 🆕


Gray Swan x UK AISI Agent Red-Teaming Challenge

Largest public agent red-teaming exercise: ~2,000 red-teamers, 1.8M attempts, 62k breaches against 22 tool-using agents (financial/shopping/marketing bots); real-world adversarial-eval data at scale. 🆕


How to Construct Domain Specific LLM Evaluation Systems

<https://www.youtube.com/watch?v=eLXF0VojuSs> · *talk* (AI Engineer World's Fair 2024)


How to look at your data

<https://www.youtube.com/watch?v=jryZvCuA0Uc> · *talk* (AI Engineer World's Fair 2025)


Failure is a Funnel

<https://www.youtube.com/watch?v=k98gDjYbSaU> · *talk* (Data Council 2025)


Using LLMs as Judges: Insights, Challenges, Best Practices

<https://www.youtube.com/watch?v=7EGF0Mc0_os> · *talk* (Jason Liu series 2024)


Scaling Up Vibe Checks for LLMs

<https://www.youtube.com/watch?v=eGVDKegRdgM> · *talk* (Stanford MLSys #97)


Why LLM Data Processing Pipelines Fail

<https://www.youtube.com/watch?v=H-1QaLPnGsg> · *talk* (LangChain Interrupt 2025)


Evals Are Not Unit Tests

<https://www.youtube.com/watch?v=L8OoYeDI_ls> · *talk* (AI Engineer 2025)


Building Metrics that actually work (workshop)

<https://www.youtube.com/watch?v=jxrGodnopHo> · *talk* (AI Engineer 2025)


From Self-driving to Autonomous Voice Agents

<https://www.youtube.com/watch?v=kDczF4wBh8s> · *talk* (AI Engineer 2025)


Fuzzing in the GenAI Era

<https://www.youtube.com/watch?v=OMGPvW8TBHc> · *talk* (AI Engineer 2025)


On Engineering AI Systems that Endure the Bitter Lesson

<https://www.youtube.com/watch?v=qdmxApz3EJI> · *talk* (AI Engineer 2025)


Strategies for LLM Evals (harnesses workshop)

<https://www.youtube.com/watch?v=89NuzmKokIk> · *talk* (AI Engineer 2025)


The Future of Evals

<https://www.youtube.com/watch?v=MC55hdWLq4o> · *talk* (AI Engineer 2025)


3 Key Ideas in AI in 2025 (Verifier's Law)

<https://www.youtube.com/watch?v=b6Doq2fz81U> · *talk* (Stanford AI Club 2025)


Some Intuitions About Large Language Models

<https://www.youtube.com/watch?v=l898fqkjdFc> · *talk* (The AI Conference 2025)


Deep Dive into LLMs like ChatGPT

<https://www.youtube.com/watch?v=7xTGNNLPyMI> · *talk* (2025)


RLHF: Progress and Challenges

<https://www.youtube.com/watch?v=hhiLw5Q_UFg> · *talk* (UC Berkeley EECS 2023)


Aligning Open Language Models

<https://www.youtube.com/watch?v=AdLgPmcrXwQ> · *talk* (Stanford CS25 V4)


Building LLM Applications for Production

<https://www.youtube.com/watch?v=spamOhG7BOA> · *talk* (MLOps LLMs in Prod 2023)


The Model is the Product

<https://www.youtube.com/watch?v=4dUFIRj-BWo> · *talk* (Data Council 2025)


RL Environments at Scale

<https://www.youtube.com/watch?v=_IzZWeuTx7I> · *talk* (AI Engineer 2025)


LLM benchmarks in the time of agents

<https://www.youtube.com/watch?v=kmTMc-fVSXw> · *talk* (Big Techday 26 (2026))


Evals, error analysis, and better prompts

<https://www.youtube.com/watch?v=PgzOBNse2EA> · *podcast* (How I AI)


Evals are the new PRD for AI products

<https://www.youtube.com/watch?v=QE_1hRLsehM> · *podcast* (How I AI)


Ep 60: 10 Things I Hate About AI Evals

<https://www.youtube.com/watch?v=QEk-XwrkqhI> · *podcast* (Vanishing Gradients)


Ep 50: A Field Guide to Rapidly Improving AI Products

<https://www.youtube.com/watch?v=rWToRi2_SeY> · *podcast* (Vanishing Gradients)


Five Hard-Earned Lessons About Evals

<https://www.youtube.com/watch?v=a4BV0gGmXgA> · *podcast* (Latent Space)


Artificial Analysis: Independent LLM Evals

<https://www.youtube.com/watch?v=v5mBjeX4TJ8> · *podcast* (Latent Space)


Reality: The Final Eval (Vending-Bench)

<https://www.youtube.com/watch?v=ZAimcoJXUBo> · *podcast* (Latent Space / Cognitive Revolution)


#5 Designing Evals

<https://www.youtube.com/watch?v=-N6MajRfqYw> · *podcast* (AI That Works)


#16 Evaluating Prompts Across Models

<https://www.youtube.com/watch?v=OawyQOrlubM> · *podcast* (AI That Works)


#24 Evals for Classification

<https://www.youtube.com/watch?v=5Fy0hBzyduU> · *podcast* (AI That Works)


#34 Multimodal Evals

<https://www.youtube.com/watch?v=jzhVo0iAX_I> · *podcast* (AI That Works)


#372 It's 2026 and We're Still Talking Evals

<https://www.youtube.com/watch?v=9EjWR3QpJYk> · *podcast* (MLOps Community)


#728 Generative Benchmarking

<https://www.youtube.com/watch?v=3kbiGPn0cOo> · *podcast* (TWIML AI)


Shaping AI Benchmarks (HELM)

<https://www.youtube.com/watch?v=kwkdKirqi6s> · *podcast* (Gradient Dissent)


Evaluating LLMs with Chatbot Arena

<https://www.youtube.com/watch?v=okHMaczHPXc> · *podcast* (Gradient Dissent)


Evaluating AI, Designing for Non-Determinism

<https://www.youtube.com/watch?v=v0eTTn7ZPEc> · *podcast* (Learning from Machine Learning)


Karpathy: RL is terrible, why benchmarks mislead

<https://www.youtube.com/watch?v=-lRBpyPt79c> · *podcast* (Dwarkesh Podcast)


How to Build AI Evals in 2026 (Step-by-Step)

<https://www.youtube.com/watch?v=J7N9FMouSKg> · *podcast* (Aakash Gupta 2026)


Towards Building Safe & Trustworthy AI Agents

<https://www.youtube.com/watch?v=QAgR4uQ15rc> · *lecture* (Berkeley LLM Agents MOOC F24)


Towards Building Safe and Secure Agentic AI

<https://www.youtube.com/watch?v=ti6yPE2VPZc> · *lecture* (Berkeley Advanced LLM Agents Sp25)


Measuring Agent Capabilities and Anthropic's RSP

<https://www.youtube.com/watch?v=6y2AnWol7oo> · *lecture* (Berkeley LLM Agents MOOC F24)


Open-Source and Science in the Era of Foundation Models

<https://www.youtube.com/watch?v=f3KKx9LWntQ> · *lecture* (Berkeley LLM Agents MOOC F24)


CS336 Lecture 12: Evaluation

<https://www.youtube.com/watch?v=x-R5l2HsXqM> · *lecture* (Stanford CS336 2025)


Companies & landscape (eval / RL-environment market)

pavlovslist.com

The RL-environment / eval startups directory ("for the RL-pilled").