AI Agents That Matter


Who Validates the Validators? (EvalGen)


Welcome to the Era of Experience


DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning


Tülu 3: Pushing Frontiers in Open Language Model Post-Training


The Leaderboard Illusion


The SWE-bench Illusion: When SOTA LLMs Remember Instead of Reason


Establishing Best Practices for Building Rigorous Agentic Benchmarks (ABC)


Lost in Simulation


SWE-bench: Can LMs Resolve Real-World GitHub Issues?


A Careful Examination of LLM Performance on Grade School Arithmetic (GSM1k)


Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks


Are We Done with MMLU? (MMLU-Redux)


LiveCodeBench: Holistic and Contamination-Free Evaluation of LLMs for Code


Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation


RewardBench


Spurious Rewards: Rethinking Training Signals in RLVR


Position: The Hidden Costs and Measurement Gaps of RLVR


RewardBench 2: Advancing Reward Model Evaluation


Curriculum RL from Easy to Hard Tasks Improves LLM Reasoning (E2H Reasoner)


GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulators


Who Validates the Validators? (EvalGen)


Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena


LLMs Instead of Human Judges? A Large-Scale Study


SPADE


LLM Evaluators Recognize and Favor Their Own Generations


G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment


A Survey on LLM-as-a-Judge


One Token to Fool LLM-as-a-Judge


Agent-as-a-Judge: Evaluate Agents with Agents


VerifyBench: A Systematic Benchmark for Evaluating Reasoning Verifiers Across Domains


Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (CALM framework)


τ-bench / τ²-bench


GAIA: A Benchmark for General AI Assistants


WebArena: A Realistic Web Environment for Building Autonomous Agents


OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments


Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risk of Language Models


AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories


Why Do Multi-Agent LLM Systems Fail? (MAST taxonomy)


LocAgent: Graph-Guided LLM Agents for Code Localization


WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models


SWE-bench Multimodal


SWE-bench Pro


SWE-Lancer


SWE-Gym


Multi-SWE-bench


SWE-rebench


RE-Bench


MLE-bench


PaperBench


Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge


Online-Mind2Web (An Illusion of Progress? Assessing the Current State of Web Agents)


WebGames: Challenging General-Purpose Web-Browsing AI Agents


GTA: A Benchmark for General Tool Agents


Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows


AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents


WindowsAgentArena: Evaluating Multi-Modal OS Agents at Scale


ST-WebAgentBench: Evaluating Safety and Trustworthiness in Web Agents


TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks


VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks


GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks


Remote Labor Index: Measuring AI Automation of Remote Work


Humanity's Last Exam


CORE-Bench: Computational Reproducibility Agent Benchmark


DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents


BixBench: A Comprehensive Benchmark for LLM-based Agents in Computational Biology


Gaia2 and ARE: Scaling Up Agent Environments and Evaluations


Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents


ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems


TRAIL: Trace Reasoning and Agentic Issue Localization


CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios


BenchJack: Systematically Auditing AI Agent Benchmarks


CyberGym


AIR-Bench 2024


RedCode


AgentPoison


Adding Error Bars to Evals (A Statistical Approach to LM Evaluations)


AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents


AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents


InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated LLM Agents


Defeating Prompt Injections by Design (CaMeL)


Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents