---
type: Research Source
title: BenchFlow Awesome Agent Evals Primary Source Ingestion — 2026
description: "Structured ingestion ledger for benchflow-ai/awesome-evals with local raw mirrors, deep notes, transcript queue markers, and golden JSONL exemplars."
tags: [benchmarks, evaluations, agents, source-ledger]
timestamp: "2026-06-24T11:42:39.376481+00:00"
---

# BenchFlow Awesome Agent Evals Primary Source Ingestion — 2026

**Repository:** https://github.com/benchflow-ai/awesome-evals
**README captured:** https://raw.githubusercontent.com/benchflow-ai/awesome-evals/main/README.md
**Citation rows:** 318
**Mirrored deep notes:** 143
**Video/talk/podcast rows queued for transcript handling:** 46
**Golden exemplars:** `sources/21-benchmarks/awesome-evals-golden-exemplars-2026.jsonl`
**Transcript state:** available in the local research workspace; not published as a public artifact.
**PDF/Chandra queue:** `sources/21-benchmarks/awesome-evals-pdf-chandra-queue-2026.jsonl`
**PDFs resolved through Chandra:** 83 / 83 paper-like rows
**Chandra extracted PDFs:** 83 rows, including 2 chunked extractions
**Media rows needing ASR:** 46 / 46

## Type Counts

| Type | Count |
|---|---:|
| paper_or_pdf | 83 |
| repository_or_docs | 70 |
| web_article | 60 |
| blog | 42 |
| talk_video | 23 |
| podcast_video | 18 |
| docs_or_book | 13 |
| lecture_video | 5 |
| newsletter | 4 |

## Section Counts

| Section | Count |
|---|---:|
| 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | 56 |
| 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming) | 22 |
| 🎤 Conference & individual talks | 22 |
| 10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing) | 21 |
| 8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable) | 21 |
| 5a · Eval frameworks & harnesses (code-first test-runners) | 20 |
| 🎙 Podcast episodes | 18 |
| 2 · "If you can eval it, you have built it" — eval ⇄ capability ⇄ RL environment | 17 |
| 5f · Observability + eval platforms (tracing · datasets · online/offline · CI) | 17 |
| 3 · The model / harness / skill decomposition | 16 |
| 1 · Why we need evals | 15 |
| 7 · Evals & RL environments (verifiers, reward design, difficulty calibration, lifecycle) | 12 |
| ⭐ Must-read starter set (read these first) | 12 |
| 4 · Observability & the output / eval space (the surfaces you can grade) | 11 |
| 5d · LLM-as-judge / reward / verifier libraries | 9 |
| 5e · RL-environment / verifiable-reward toolkits (eval ⇄ training) | 9 |
| 5c · RAG / retrieval evaluation | 5 |
| 5g · Tracing standards | 5 |
| 🎓 University lectures | 5 |
| 5b · TypeScript/JS-native eval runners | 4 |
| Companies & landscape (eval / RL-environment market) | 1 |

## Ingestion Boundary

This ledger treats BenchFlow's README as a curated citation graph and mirrors all visible BenchFlow deep notes as secondary reading notes. Raw web/blog/docs pages are opportunistically mirrored when the source is directly fetchable. YouTube talks, podcasts, and lectures were attempted through the transcript path; public captions were unavailable for all 46, so they are tracked as `needs_asr` for the podcast/video ASR pipeline. Paper-like rows were resolved to PDFs where possible: all 83 paper/PDF rows now have local Chandra text evidence, including 2 chunked extractions.

The JSONL exemplar file is not a claim that every source exposes an eval schema. It is a golden-quality classification example for how each citation should map into the enterprise eval contract: source bundle, artifact type, eval domain, failure code, evidence boundary, and required runtime metadata.

## Citation Ledger

| ID | Section | Type | Title | Status | Local evidence |
|---|---|---|---|---|---|
| `ae-001-the-second-half` | ⭐ Must-read starter set (read these first) | web_article | [The Second Half](https://ysymyth.github.io/The-Second-Half/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-001-the-second-half.raw.txt` |
| `ae-002-an-llm-as-judge-won-t-save-the-product-fixing-yo` | ⭐ Must-read starter set (read these first) | web_article | [An LLM-as-Judge Won't Save the Product, Fixing Your Process Will](https://eugeneyan.com/writing/eval-process/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-002-an-llm-as-judge-won-t-save-the-product-fixing-yo.raw.txt` |
| `ae-003-hidden-technical-debt-agent-evaluation-infrastru` | ⭐ Must-read starter set (read these first) | blog | [Hidden Technical Debt: Agent Evaluation Infrastructure](https://leehanchung.github.io/blogs/2026/06/13/hidden-technical-debt-agent-evaluation-infra/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-003-hidden-technical-debt-agent-evaluation-infrastru.raw.txt` |
| `ae-004-llm-evals-faq` | ⭐ Must-read starter set (read these first) | blog | [LLM Evals FAQ](https://hamel.dev/blog/posts/evals-faq/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-004-llm-evals-faq.raw.txt` |
| `ae-005-asymmetry-of-verification-and-verifier-s-law` | ⭐ Must-read starter set (read these first) | blog | [Asymmetry of Verification and Verifier's Law](https://www.jasonwei.net/blog/asymmetry-of-verification-and-verifiers-law) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-005-asymmetry-of-verification-and-verifier-s-law.raw.txt` |
| `ae-006-demystifying-evals-for-ai-agents` | ⭐ Must-read starter set (read these first) | web_article | [Demystifying Evals for AI Agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-006-demystifying-evals-for-ai-agents.raw.txt` |
| `ae-007-how-to-build-good-language-modeling-benchmarks` | ⭐ Must-read starter set (read these first) | web_article | [How to Build Good Language Modeling Benchmarks](https://ofir.io/How-to-Build-Good-Language-Modeling-Benchmarks/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-007-how-to-build-good-language-modeling-benchmarks.raw.txt` |
| `ae-008-ai-agents-that-matter` | ⭐ Must-read starter set (read these first) | paper_or_pdf | [AI Agents That Matter](https://arxiv.org/abs/2407.01502) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-008-ai-agents-that-matter.raw.txt` |
| `ae-009-building-on-evaluation-quicksand` | ⭐ Must-read starter set (read these first) | blog | [Building on Evaluation Quicksand](https://www.interconnects.ai/p/building-on-evaluation-quicksand) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-009-building-on-evaluation-quicksand.raw.txt` |
| `ae-010-who-validates-the-validators-evalgen` | ⭐ Must-read starter set (read these first) | paper_or_pdf | [Who Validates the Validators? (EvalGen)](https://arxiv.org/abs/2404.12272) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-010-who-validates-the-validators-evalgen.raw.txt` |
| `ae-011-benches-2026-llm-benchmarks-in-the-era-of-agents` | ⭐ Must-read starter set (read these first) | web_article | [Benches 2026 — "LLM benchmarks in the era of agents"](https://florianbrand.com/posts/benches-2026) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-011-benches-2026-llm-benchmarks-in-the-era-of-agents.raw.txt` |
| `ae-012-a-shared-playbook-for-trustworthy-third-party-ev` | ⭐ Must-read starter set (read these first) | docs_or_book | [A Shared Playbook for Trustworthy Third-Party Evaluations](https://openai.com/index/trustworthy-third-party-evaluations-foundations/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-012-a-shared-playbook-for-trustworthy-third-party-ev.raw.txt` |
| `ae-013-the-second-half` | 1 · Why we need evals | web_article | [The Second Half](https://ysymyth.github.io/The-Second-Half/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-013-the-second-half.raw.txt` |
| `ae-014-an-llm-as-judge-won-t-save-the-product-fixing-yo` | 1 · Why we need evals | web_article | [An LLM-as-Judge Won't Save the Product, Fixing Your Process Will](https://eugeneyan.com/writing/eval-process/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-014-an-llm-as-judge-won-t-save-the-product-fixing-yo.raw.txt` |
| `ae-015-your-ai-product-needs-evals` | 1 · Why we need evals | blog | [Your AI Product Needs Evals](https://hamel.dev/blog/posts/evals/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-015-your-ai-product-needs-evals.raw.txt` |
| `ae-016-a-field-guide-to-rapidly-improving-ai-products` | 1 · Why we need evals | blog | [A Field Guide to Rapidly Improving AI Products](https://hamel.dev/blog/posts/field-guide/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-016-a-field-guide-to-rapidly-improving-ai-products.raw.txt` |
| `ae-017-in-defense-of-ai-evals-for-everyone` | 1 · Why we need evals | blog | [In Defense of AI Evals, for Everyone](https://www.sh-reya.com/blog/in-defense-ai-evals/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-017-in-defense-of-ai-evals-for-everyone.raw.txt` |
| `ae-018-what-we-learned-from-a-year-of-building-with-llm` | 1 · Why we need evals | web_article | [What We Learned from a Year of Building with LLMs](https://applied-llms.org/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-018-what-we-learned-from-a-year-of-building-with-llm.raw.txt` |
| `ae-019-big-tech-s-llm-evals-are-just-marketing` | 1 · Why we need evals | blog | [Big Tech's LLM Evals Are Just Marketing](https://www.interconnects.ai/p/evals-are-marketing) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-019-big-tech-s-llm-evals-are-just-marketing.raw.txt` |
| `ae-020-ai-engineering-pitfalls` | 1 · Why we need evals | web_article | [AI Engineering pitfalls](https://huyenchip.com/2025/01/16/ai-engineering-pitfalls.html) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-020-ai-engineering-pitfalls.raw.txt` |
| `ae-021-evals-are-not-all-you-need` | 1 · Why we need evals | web_article | [Evals Are NOT All You Need](https://www.oreilly.com/radar/evals-are-not-all-you-need/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-021-evals-are-not-all-you-need.raw.txt` |
| `ae-022-why-ai-evals-are-the-hottest-new-skill-for-produ` | 1 · Why we need evals | newsletter | [Why AI evals are the hottest new skill for product builders](https://www.lennysnewsletter.com/p/why-ai-evals-are-the-hottest-new-skill) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-022-why-ai-evals-are-the-hottest-new-skill-for-produ.raw.txt` |
| `ae-023-how-evals-drive-the-next-chapter-in-ai-for-busin` | 1 · Why we need evals | web_article | [How evals drive the next chapter in AI for businesses](https://openai.com/index/evals-drive-next-chapter-of-ai/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-023-how-evals-drive-the-next-chapter-in-ai-for-busin.raw.txt` |
| `ae-024-beyond-vibe-checks-a-pm-s-complete-guide-to-eval` | 1 · Why we need evals | newsletter | [Beyond vibe checks: A PM's complete guide to evals](https://www.lennysnewsletter.com/p/beyond-vibe-checks-a-pms-complete) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-024-beyond-vibe-checks-a-pm-s-complete-guide-to-eval.raw.txt` |
| `ae-025-a-pragmatic-guide-to-llm-evals-for-devs` | 1 · Why we need evals | newsletter | [A pragmatic guide to LLM evals for devs](https://newsletter.pragmaticengineer.com/p/evals) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-025-a-pragmatic-guide-to-llm-evals-for-devs.raw.txt` |
| `ae-026-predicting-model-behavior-before-release-by-simu` | 1 · Why we need evals | web_article | [Predicting model behavior before release by simulating deployment (Deployment Simulation)](https://openai.com/index/deployment-simulation/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-026-predicting-model-behavior-before-release-by-simu.raw.txt` |
| `ae-027-evals-are-surprisingly-often-all-you-need` | 1 · Why we need evals | web_article | [evals are surprisingly often all you need](https://x.com/gdb/status/1733553161884127435) | metadata_only | `` |
| `ae-028-asymmetry-of-verification-and-verifier-s-law` | 2 · "If you can eval it, you have built it" — eval ⇄ capability ⇄ RL environment | blog | [Asymmetry of Verification and Verifier's Law](https://www.jasonwei.net/blog/asymmetry-of-verification-and-verifiers-law) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-028-asymmetry-of-verification-and-verifier-s-law.raw.txt` |
| `ae-029-a-taxonomy-of-rl-environments-for-llm-agents` | 2 · "If you can eval it, you have built it" — eval ⇄ capability ⇄ RL environment | blog | [A Taxonomy of RL Environments for LLM Agents](https://leehanchung.github.io/blogs/2026/03/21/rl-environments-for-llm-agents/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-029-a-taxonomy-of-rl-environments-for-llm-agents.raw.txt` |
| `ae-030-the-life-cycle-of-an-rl-environment` | 2 · "If you can eval it, you have built it" — eval ⇄ capability ⇄ RL environment | blog | [The Life Cycle of an RL Environment](https://muratbuffalo.blogspot.com/2026/06/acm-cais-conference-on-ai-and-agentic.html) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-030-the-life-cycle-of-an-rl-environment.raw.txt` |
| `ae-031-welcome-to-the-era-of-experience` | 2 · "If you can eval it, you have built it" — eval ⇄ capability ⇄ RL environment | paper_or_pdf | [Welcome to the Era of Experience](https://storage.googleapis.com/deepmind-media/Era-of-Experience%20/The%20Era%20of%20Experience%20Paper.pdf) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-031-welcome-to-the-era-of-experience.raw.txt` |
| `ae-032-rlhf-book-ch-16-evaluation` | 2 · "If you can eval it, you have built it" — eval ⇄ capability ⇄ RL environment | docs_or_book | [RLHF Book, Ch. 16 — Evaluation](https://rlhfbook.com/c/16-evaluation) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-032-rlhf-book-ch-16-evaluation.raw.txt` |
| `ae-033-what-comes-next-with-reinforcement-learning` | 2 · "If you can eval it, you have built it" — eval ⇄ capability ⇄ RL environment | blog | [What Comes Next with Reinforcement Learning](https://www.interconnects.ai/p/what-comes-next-with-reinforcement) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-033-what-comes-next-with-reinforcement-learning.raw.txt` |
| `ae-034-verifiers` | 2 · "If you can eval it, you have built it" — eval ⇄ capability ⇄ RL environment | repository_or_docs | [verifiers](https://github.com/PrimeIntellect-ai/verifiers) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-034-verifiers.raw.txt` |
| `ae-035-deepseek-r1-incentivizing-reasoning-capability-i` | 2 · "If you can eval it, you have built it" — eval ⇄ capability ⇄ RL environment | paper_or_pdf | [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://arxiv.org/abs/2501.12948) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-035-deepseek-r1-incentivizing-reasoning-capability-i.raw.txt` |
| `ae-036-t-lu-3-pushing-frontiers-in-open-language-model` | 2 · "If you can eval it, you have built it" — eval ⇄ capability ⇄ RL environment | paper_or_pdf | [Tülu 3: Pushing Frontiers in Open Language Model Post-Training](https://arxiv.org/abs/2411.15124) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-036-t-lu-3-pushing-frontiers-in-open-language-model.raw.txt` |
| `ae-037-natural-emergent-misalignment-from-reward-hackin` | 2 · "If you can eval it, you have built it" — eval ⇄ capability ⇄ RL environment | web_article | [Natural Emergent Misalignment from Reward Hacking in Production RL](https://www.anthropic.com/research/emergent-misalignment-reward-hacking) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-037-natural-emergent-misalignment-from-reward-hackin.raw.txt` |
| `ae-038-environments-hub-a-community-hub-to-scale-rl-to` | 2 · "If you can eval it, you have built it" — eval ⇄ capability ⇄ RL environment | blog | [Environments Hub: A Community Hub To Scale RL To Open AGI](https://www.primeintellect.ai/blog/environments) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-038-environments-hub-a-community-hub-to-scale-rl-to.raw.txt` |
| `ae-039-how-to-fully-automate-software-engineering` | 2 · "If you can eval it, you have built it" — eval ⇄ capability ⇄ RL environment | blog | [How to fully automate software engineering](https://www.mechanize.work/blog/how-to-fully-automate-software-engineering/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-039-how-to-fully-automate-software-engineering.raw.txt` |
| `ae-040-cheap-rl-tasks-will-waste-compute` | 2 · "If you can eval it, you have built it" — eval ⇄ capability ⇄ RL environment | blog | [Cheap RL tasks will waste compute](https://www.mechanize.work/blog/cheap-rl-tasks-will-waste-compute/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-040-cheap-rl-tasks-will-waste-compute.raw.txt` |
| `ae-041-an-faq-on-reinforcement-learning-environments` | 2 · "If you can eval it, you have built it" — eval ⇄ capability ⇄ RL environment | web_article | [An FAQ on Reinforcement Learning Environments](https://epoch.ai/gradient-updates/state-of-rl-envs) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-041-an-faq-on-reinforcement-learning-environments.raw.txt` |
| `ae-042-rl-environments-and-rl-for-science-data-foundrie` | 2 · "If you can eval it, you have built it" — eval ⇄ capability ⇄ RL environment | newsletter | [RL Environments and RL for Science: Data Foundries and Multi-Agent Architectures](https://newsletter.semianalysis.com/p/rl-environments-and-rl-for-science) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-042-rl-environments-and-rl-for-science-data-foundrie.raw.txt` |
| `ae-043-terminal-bench-benchmarking-agents-on-hard-reali` | 2 · "If you can eval it, you have built it" — eval ⇄ capability ⇄ RL environment | repository_or_docs | [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://github.com/harbor-framework/terminal-bench) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-043-terminal-bench-benchmarking-agents-on-hard-reali.raw.txt` |
| `ae-044-tau2-bench-bench-a-benchmark-for-tool-agent-user` | 2 · "If you can eval it, you have built it" — eval ⇄ capability ⇄ RL environment | repository_or_docs | [tau2-bench (τ²-Bench): A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://github.com/sierra-research/tau2-bench) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-044-tau2-bench-bench-a-benchmark-for-tool-agent-user.raw.txt` |
| `ae-045-hidden-technical-debt-agent-harness` | 3 · The model / harness / skill decomposition | blog | [Hidden Technical Debt: Agent Harness](https://leehanchung.github.io/blogs/2026/05/08/hidden-technical-debt-agent-harness/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-045-hidden-technical-debt-agent-harness.raw.txt` |
| `ae-046-hidden-technical-debt-series-index` | 3 · The model / harness / skill decomposition | blog | [Hidden Technical Debt series (index)](https://leehanchung.github.io/blogs/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-046-hidden-technical-debt-series-index.raw.txt` |
| `ae-047-measuring-ai-ability-to-complete-long-tasks` | 3 · The model / harness / skill decomposition | blog | [Measuring AI Ability to Complete Long Tasks](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-047-measuring-ai-ability-to-complete-long-tasks.raw.txt` |
| `ae-048-turing-post-interview-open-models-won-t-catch-up` | 3 · The model / harness / skill decomposition | blog | [Turing Post interview ("Open Models Won't Catch Up")](https://www.turingpost.com/p/nathanlambert) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-048-turing-post-interview-open-models-won-t-catch-up.raw.txt` |
| `ae-049-quo-vadis-llm-benchmarks` | 3 · The model / harness / skill decomposition | web_article | [Quo vadis, LLM benchmarks?](https://florianbrand.com/posts/benches-2026) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-049-quo-vadis-llm-benchmarks.raw.txt` |
| `ae-050-the-model-is-the-product` | 3 · The model / harness / skill decomposition | web_article | [The Model is the Product](https://leehanchung.github.io/talks/2025/04/23/the-model-is-the-product/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-050-the-model-is-the-product.raw.txt` |
| `ae-051-the-model-is-not-the-product` | 3 · The model / harness / skill decomposition | talk_video | [The Model is Not the Product](https://www.youtube.com/watch?v=EEw2PpL-_NM) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-lee-model-is-the-product.md` |
| `ae-052-agents-are-models-using-tools-in-a-loop` | 3 · The model / harness / skill decomposition | web_article | [Agents are models using tools in a loop](https://simonwillison.net/2025/May/22/tools-in-a-loop/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-052-agents-are-models-using-tools-in-a-loop.raw.txt` |
| `ae-053-harness-engineering-leveraging-codex-in-an-agent` | 3 · The model / harness / skill decomposition | web_article | [Harness engineering: leveraging Codex in an agent-first world](https://openai.com/index/harness-engineering/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-053-harness-engineering-leveraging-codex-in-an-agent.raw.txt` |
| `ae-054-equipping-agents-for-the-real-world-with-agent-s` | 3 · The model / harness / skill decomposition | web_article | [Equipping agents for the real world with Agent Skills](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-054-equipping-agents-for-the-real-world-with-agent-s.raw.txt` |
| `ae-055-effective-context-engineering-for-ai-agents` | 3 · The model / harness / skill decomposition | web_article | [Effective context engineering for AI agents](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-055-effective-context-engineering-for-ai-agents.raw.txt` |
| `ae-056-writing-effective-tools-for-agents-with-agents` | 3 · The model / harness / skill decomposition | web_article | [Writing effective tools for agents — with agents](https://www.anthropic.com/engineering/writing-tools-for-agents) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-056-writing-effective-tools-for-agents-with-agents.raw.txt` |
| `ae-057-same-model-different-results-why-coding-agents-a` | 3 · The model / harness / skill decomposition | blog | [Same Model, Different Results: Why Coding Agents Aren't Interchangeable](https://blog.thepete.net/blog/2025/12/10/same-model-different-results-why-coding-agents-arent-interchangeable/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-057-same-model-different-results-why-coding-agents-a.raw.txt` |
| `ae-058-holistic-agent-leaderboard-hal` | 3 · The model / harness / skill decomposition | web_article | [Holistic Agent Leaderboard (HAL)](https://hal.cs.princeton.edu/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-058-holistic-agent-leaderboard-hal.raw.txt` |
| `ae-059-agent-harness-engineering` | 3 · The model / harness / skill decomposition | web_article | [Agent Harness Engineering](https://www.oreilly.com/radar/agent-harness-engineering/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-059-agent-harness-engineering.raw.txt` |
| `ae-060-what-comes-next-with-open-models-weights-tools-h` | 3 · The model / harness / skill decomposition | blog | [What comes next with open models (weights / tools / harness decomposition)](https://www.interconnects.ai/p/the-next-phase-of-open-models) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-060-what-comes-next-with-open-models-weights-tools-h.raw.txt` |
| `ae-061-hidden-technical-debt-agent-evaluation-infrastru` | 4 · Observability & the output / eval space (the surfaces you can grade) | blog | [Hidden Technical Debt: Agent Evaluation Infrastructure](https://leehanchung.github.io/blogs/2026/06/13/hidden-technical-debt-agent-evaluation-infra/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-061-hidden-technical-debt-agent-evaluation-infrastru.raw.txt` |
| `ae-062-the-three-pillars-of-ai-observability` | 4 · Observability & the output / eval space (the surfaces you can grade) | blog | [The Three Pillars of AI Observability](https://www.braintrust.dev/blog/three-pillars-ai-observability) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-062-the-three-pillars-of-ai-observability.raw.txt` |
| `ae-063-agent-trajectory-evaluations` | 4 · Observability & the output / eval space (the surfaces you can grade) | docs_or_book | [Agent Trajectory Evaluations](https://arize.com/docs/ax/evaluate/evaluators/trace-and-session-evals/trace-level-evaluations/agent-trajectory-evaluations) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-063-agent-trajectory-evaluations.raw.txt` |
| `ae-064-ai-agent-metrics-how-elite-teams-evaluate` | 4 · Observability & the output / eval space (the surfaces you can grade) | blog | [AI Agent Metrics: How Elite Teams Evaluate](https://galileo.ai/blog/ai-agent-metrics) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-064-ai-agent-metrics-how-elite-teams-evaluate.raw.txt` |
| `ae-065-openinference-semantic-conventions` | 4 · Observability & the output / eval space (the surfaces you can grade) | repository_or_docs | [OpenInference semantic conventions](https://github.com/Arize-ai/openinference/blob/main/spec/semantic_conventions.md) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-065-openinference-semantic-conventions.md` |
| `ae-066-langsmith-evaluation-trajectory-evals` | 4 · Observability & the output / eval space (the surfaces you can grade) | docs_or_book | [LangSmith Evaluation / Trajectory evals](https://docs.langchain.com/langsmith/evaluation) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-066-langsmith-evaluation-trajectory-evals.raw.txt` |
| `ae-067-opentelemetry-genai-semantic-conventions-agent-f` | 4 · Observability & the output / eval space (the surfaces you can grade) | repository_or_docs | [OpenTelemetry GenAI Semantic Conventions (agent & framework spans)](https://github.com/open-telemetry/semantic-conventions-genai) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-067-opentelemetry-genai-semantic-conventions-agent-f.raw.txt` |
| `ae-068-semantic-conventions-for-genai-agent-and-framewo` | 4 · Observability & the output / eval space (the surfaces you can grade) | docs_or_book | [Semantic Conventions for GenAI agent and framework spans](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-068-semantic-conventions-for-genai-agent-and-framewo.raw.txt` |
| `ae-069-inside-the-llm-call-genai-observability-with-ope` | 4 · Observability & the output / eval space (the surfaces you can grade) | blog | [Inside the LLM Call: GenAI Observability with OpenTelemetry](https://opentelemetry.io/blog/2026/genai-observability/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-069-inside-the-llm-call-genai-observability-with-ope.raw.txt` |
| `ae-070-w-b-weave-tracing-evaluation-toolkit` | 4 · Observability & the output / eval space (the surfaces you can grade) | docs_or_book | [W&B Weave — tracing & evaluation toolkit](https://docs.wandb.ai/weave) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-070-w-b-weave-tracing-evaluation-toolkit.raw.txt` |
| `ae-071-laminar-open-source-observability-for-ai-agents` | 4 · Observability & the output / eval space (the surfaces you can grade) | web_article | [Laminar — open-source observability for AI agents](https://laminar.sh/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-071-laminar-open-source-observability-for-ai-agents.raw.txt` |
| `ae-072-inspect-ai` | 5a · Eval frameworks & harnesses (code-first test-runners) | repository_or_docs | [Inspect AI](https://github.com/UKGovernmentBEIS/inspect_ai) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-072-inspect-ai.raw.txt` |
| `ae-073-inspect-evals` | 5a · Eval frameworks & harnesses (code-first test-runners) | repository_or_docs | [inspect_evals](https://github.com/UKGovernmentBEIS/inspect_evals) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-073-inspect-evals.raw.txt` |
| `ae-074-lm-evaluation-harness` | 5a · Eval frameworks & harnesses (code-first test-runners) | repository_or_docs | [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-074-lm-evaluation-harness.raw.txt` |
| `ae-075-olmes` | 5a · Eval frameworks & harnesses (code-first test-runners) | repository_or_docs | [OLMES](https://github.com/allenai/olmes) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-075-olmes.raw.txt` |
| `ae-076-benchflow` | 5a · Eval frameworks & harnesses (code-first test-runners) | repository_or_docs | [BenchFlow](https://github.com/benchflow-ai/benchflow) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-076-benchflow.raw.txt` |
| `ae-077-lighteval` | 5a · Eval frameworks & harnesses (code-first test-runners) | repository_or_docs | [lighteval](https://github.com/huggingface/lighteval) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-077-lighteval.raw.txt` |
| `ae-078-openbench` | 5a · Eval frameworks & harnesses (code-first test-runners) | repository_or_docs | [OpenBench](https://github.com/groq/openbench) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-078-openbench.raw.txt` |
| `ae-079-simple-evals` | 5a · Eval frameworks & harnesses (code-first test-runners) | repository_or_docs | [simple-evals](https://github.com/openai/simple-evals) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-079-simple-evals.raw.txt` |
| `ae-080-openai-evals` | 5a · Eval frameworks & harnesses (code-first test-runners) | repository_or_docs | [OpenAI Evals](https://github.com/openai/evals) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-080-openai-evals.raw.txt` |
| `ae-081-promptfoo` | 5a · Eval frameworks & harnesses (code-first test-runners) | repository_or_docs | [promptfoo](https://github.com/promptfoo/promptfoo) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-081-promptfoo.raw.txt` |
| `ae-082-deepeval-confident-ai` | 5a · Eval frameworks & harnesses (code-first test-runners) | repository_or_docs | [DeepEval / Confident AI](https://github.com/confident-ai/deepeval) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-082-deepeval-confident-ai.raw.txt` |
| `ae-083-pydantic-evals` | 5a · Eval frameworks & harnesses (code-first test-runners) | repository_or_docs | [pydantic-evals](https://github.com/pydantic/pydantic-ai) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-083-pydantic-evals.raw.txt` |
| `ae-084-openevals` | 5a · Eval frameworks & harnesses (code-first test-runners) | repository_or_docs | [openevals](https://github.com/langchain-ai/openevals) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-084-openevals.raw.txt` |
| `ae-085-mlflow-genai-evaluate` | 5a · Eval frameworks & harnesses (code-first test-runners) | docs_or_book | [MLflow GenAI evaluate](https://mlflow.org/docs/latest/genai/eval-monitor/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-085-mlflow-genai-evaluate.raw.txt` |
| `ae-086-helm-crfm-helm` | 5a · Eval frameworks & harnesses (code-first test-runners) | repository_or_docs | [HELM (crfm-helm)](https://github.com/stanford-crfm/helm) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-086-helm-crfm-helm.raw.txt` |
| `ae-087-giskard` | 5a · Eval frameworks & harnesses (code-first test-runners) | repository_or_docs | [Giskard](https://github.com/Giskard-AI/giskard-oss) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-087-giskard.raw.txt` |
| `ae-088-deepchecks-llm` | 5a · Eval frameworks & harnesses (code-first test-runners) | repository_or_docs | [Deepchecks LLM](https://github.com/deepchecks/deepchecks) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-088-deepchecks-llm.raw.txt` |
| `ae-089-uptrain` | 5a · Eval frameworks & harnesses (code-first test-runners) | repository_or_docs | [UpTrain](https://github.com/uptrain-ai/uptrain) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-089-uptrain.raw.txt` |
| `ae-090-hf-evaluate` | 5a · Eval frameworks & harnesses (code-first test-runners) | repository_or_docs | [HF `evaluate`](https://github.com/huggingface/evaluate) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-090-hf-evaluate.raw.txt` |
| `ae-091-harbor` | 5a · Eval frameworks & harnesses (code-first test-runners) | repository_or_docs | [Harbor](https://github.com/harbor-framework/harbor) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-091-harbor.raw.txt` |
| `ae-092-evalite` | 5b · TypeScript/JS-native eval runners | repository_or_docs | [evalite](https://github.com/mattpocock/evalite) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-092-evalite.raw.txt` |
| `ae-093-mastra-scorers` | 5b · TypeScript/JS-native eval runners | repository_or_docs | [Mastra scorers](https://github.com/mastra-ai/mastra) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-093-mastra-scorers.raw.txt` |
| `ae-094-vercel-agent-eval` | 5b · TypeScript/JS-native eval runners | repository_or_docs | [Vercel agent-eval](https://github.com/vercel-labs/agent-eval) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-094-vercel-agent-eval.raw.txt` |
| `ae-095-autoevals` | 5b · TypeScript/JS-native eval runners | repository_or_docs | [Autoevals](https://github.com/braintrustdata/autoevals) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-095-autoevals.raw.txt` |
| `ae-096-trulens` | 5c · RAG / retrieval evaluation | repository_or_docs | [TruLens](https://github.com/truera/trulens) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-096-trulens.raw.txt` |
| `ae-097-ares` | 5c · RAG / retrieval evaluation | repository_or_docs | [ARES](https://github.com/stanford-futuredata/ARES) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-097-ares.raw.txt` |
| `ae-098-ragchecker` | 5c · RAG / retrieval evaluation | repository_or_docs | [RAGChecker](https://github.com/amazon-science/RAGChecker) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-098-ragchecker.raw.txt` |
| `ae-099-continuous-eval-relari` | 5c · RAG / retrieval evaluation | repository_or_docs | [continuous-eval (Relari)](https://github.com/relari-ai/continuous-eval) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-099-continuous-eval-relari.raw.txt` |
| `ae-100-tonic-validate` | 5c · RAG / retrieval evaluation | repository_or_docs | [Tonic Validate](https://github.com/TonicAI/tonic_validate) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-100-tonic-validate.raw.txt` |
| `ae-101-verdict` | 5d · LLM-as-judge / reward / verifier libraries | repository_or_docs | [verdict](https://github.com/haizelabs/verdict) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-101-verdict.raw.txt` |
| `ae-102-ruler` | 5d · LLM-as-judge / reward / verifier libraries | repository_or_docs | [RULER](https://github.com/OpenPipe/ART) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-102-ruler.raw.txt` |
| `ae-103-prometheus-2` | 5d · LLM-as-judge / reward / verifier libraries | repository_or_docs | [Prometheus 2](https://github.com/prometheus-eval/prometheus-eval) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-103-prometheus-2.raw.txt` |
| `ae-104-atla-selene` | 5d · LLM-as-judge / reward / verifier libraries | repository_or_docs | [Atla Selene](https://github.com/atla-ai/selene-mini) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-104-atla-selene.raw.txt` |
| `ae-105-patronus-lynx-glider` | 5d · LLM-as-judge / reward / verifier libraries | repository_or_docs | [Patronus Lynx / GLIDER](https://github.com/patronus-ai/Lynx-hallucination-detection) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-105-patronus-lynx-glider.raw.txt` |
| `ae-106-flow-judge` | 5d · LLM-as-judge / reward / verifier libraries | repository_or_docs | [Flow-Judge](https://github.com/flowaicom/flow-judge) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-106-flow-judge.raw.txt` |
| `ae-107-rewardbench` | 5d · LLM-as-judge / reward / verifier libraries | repository_or_docs | [RewardBench](https://github.com/allenai/reward-bench) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-107-rewardbench.raw.txt` |
| `ae-108-judgebench` | 5d · LLM-as-judge / reward / verifier libraries | repository_or_docs | [JudgeBench](https://github.com/ScalerLab/JudgeBench) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-108-judgebench.raw.txt` |
| `ae-109-reward-kit` | 5d · LLM-as-judge / reward / verifier libraries | repository_or_docs | [reward-kit](https://github.com/fw-ai-external/reward-kit) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-109-reward-kit.raw.txt` |
| `ae-110-verifiers` | 5e · RL-environment / verifiable-reward toolkits (eval ⇄ training) | repository_or_docs | [verifiers](https://github.com/PrimeIntellect-ai/verifiers) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-110-verifiers.raw.txt` |
| `ae-111-environments-hub` | 5e · RL-environment / verifiable-reward toolkits (eval ⇄ training) | repository_or_docs | [Environments Hub](https://github.com/PrimeIntellect-ai/community-environments) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-111-environments-hub.raw.txt` |
| `ae-112-prime-rl` | 5e · RL-environment / verifiable-reward toolkits (eval ⇄ training) | repository_or_docs | [prime-rl](https://github.com/PrimeIntellect-ai/prime-rl) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-112-prime-rl.raw.txt` |
| `ae-113-benchflow` | 5e · RL-environment / verifiable-reward toolkits (eval ⇄ training) | repository_or_docs | [BenchFlow](https://github.com/benchflow-ai/benchflow) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-113-benchflow.raw.txt` |
| `ae-114-hud` | 5e · RL-environment / verifiable-reward toolkits (eval ⇄ training) | repository_or_docs | [HUD](https://github.com/hud-evals/hud-python) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-114-hud.raw.txt` |
| `ae-115-atropos` | 5e · RL-environment / verifiable-reward toolkits (eval ⇄ training) | repository_or_docs | [Atropos](https://github.com/NousResearch/atropos) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-115-atropos.raw.txt` |
| `ae-116-verl` | 5e · RL-environment / verifiable-reward toolkits (eval ⇄ training) | repository_or_docs | [verl](https://github.com/volcengine/verl) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-116-verl.raw.txt` |
| `ae-117-openrlhf` | 5e · RL-environment / verifiable-reward toolkits (eval ⇄ training) | repository_or_docs | [OpenRLHF](https://github.com/OpenRLHF/OpenRLHF) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-117-openrlhf.raw.txt` |
| `ae-118-open-reward-standard-ors` | 5e · RL-environment / verifiable-reward toolkits (eval ⇄ training) | docs_or_book | [Open Reward Standard (ORS)](https://docs.openreward.ai/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-118-open-reward-standard-ors.raw.txt` |
| `ae-119-arize-phoenix` | 5f · Observability + eval platforms (tracing · datasets · online/offline · CI) | repository_or_docs | [Arize Phoenix](https://github.com/Arize-ai/phoenix) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-119-arize-phoenix.raw.txt` |
| `ae-120-langfuse` | 5f · Observability + eval platforms (tracing · datasets · online/offline · CI) | repository_or_docs | [Langfuse](https://github.com/langfuse/langfuse) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-120-langfuse.raw.txt` |
| `ae-121-opik` | 5f · Observability + eval platforms (tracing · datasets · online/offline · CI) | repository_or_docs | [Opik](https://github.com/comet-ml/opik) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-121-opik.raw.txt` |
| `ae-122-w-b-weave` | 5f · Observability + eval platforms (tracing · datasets · online/offline · CI) | repository_or_docs | [W&B Weave](https://github.com/wandb/weave) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-122-w-b-weave.raw.txt` |
| `ae-123-braintrust` | 5f · Observability + eval platforms (tracing · datasets · online/offline · CI) | docs_or_book | [Braintrust](https://www.braintrust.dev/docs/start/eval-sdk) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-123-braintrust.raw.txt` |
| `ae-124-patronus-ai` | 5f · Observability + eval platforms (tracing · datasets · online/offline · CI) | web_article | [Patronus AI](https://www.patronus.ai/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-124-patronus-ai.raw.txt` |
| `ae-125-maxim-ai` | 5f · Observability + eval platforms (tracing · datasets · online/offline · CI) | web_article | [Maxim AI](https://www.getmaxim.ai/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-125-maxim-ai.raw.txt` |
| `ae-126-galileo` | 5f · Observability + eval platforms (tracing · datasets · online/offline · CI) | web_article | [Galileo](https://galileo.ai/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-126-galileo.raw.txt` |
| `ae-127-vellum` | 5f · Observability + eval platforms (tracing · datasets · online/offline · CI) | web_article | [Vellum](https://www.vellum.ai/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-127-vellum.raw.txt` |
| `ae-128-helicone` | 5f · Observability + eval platforms (tracing · datasets · online/offline · CI) | repository_or_docs | [Helicone](https://github.com/helicone/helicone) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-128-helicone.raw.txt` |
| `ae-129-traceloop-openllmetry` | 5f · Observability + eval platforms (tracing · datasets · online/offline · CI) | repository_or_docs | [Traceloop / OpenLLMetry](https://github.com/traceloop/openllmetry) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-129-traceloop-openllmetry.raw.txt` |
| `ae-130-langtrace` | 5f · Observability + eval platforms (tracing · datasets · online/offline · CI) | repository_or_docs | [Langtrace](https://github.com/Scale3-Labs/langtrace) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-130-langtrace.raw.txt` |
| `ae-131-whylabs-langkit` | 5f · Observability + eval platforms (tracing · datasets · online/offline · CI) | repository_or_docs | [WhyLabs / LangKit](https://github.com/whylabs/langkit) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-131-whylabs-langkit.raw.txt` |
| `ae-132-portkey` | 5f · Observability + eval platforms (tracing · datasets · online/offline · CI) | repository_or_docs | [Portkey](https://github.com/portkey-ai/gateway) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-132-portkey.raw.txt` |
| `ae-133-datadog-llm-observability` | 5f · Observability + eval platforms (tracing · datasets · online/offline · CI) | web_article | [Datadog LLM Observability](https://www.datadoghq.com/product/ai/llm-observability/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-133-datadog-llm-observability.raw.txt` |
| `ae-134-fiddler-ai` | 5f · Observability + eval platforms (tracing · datasets · online/offline · CI) | web_article | [Fiddler AI](https://www.fiddler.ai/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-134-fiddler-ai.raw.txt` |
| `ae-135-promptlayer` | 5f · Observability + eval platforms (tracing · datasets · online/offline · CI) | web_article | [PromptLayer](https://www.promptlayer.com/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-135-promptlayer.raw.txt` |
| `ae-136-openinference` | 5g · Tracing standards | repository_or_docs | [OpenInference](https://github.com/Arize-ai/openinference) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-136-openinference.raw.txt` |
| `ae-137-opentelemetry-genai-semantic-conventions` | 5g · Tracing standards | docs_or_book | [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-137-opentelemetry-genai-semantic-conventions.raw.txt` |
| `ae-138-braintrust` | 5g · Tracing standards | web_article | [Braintrust](https://www.braintrust.dev/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-138-braintrust.raw.txt` |
| `ae-139-ragaai-catalyst` | 5g · Tracing standards | repository_or_docs | [RagaAI Catalyst](https://github.com/raga-ai-hub/RagaAI-Catalyst) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-139-ragaai-catalyst.raw.txt` |
| `ae-140-openai-cookbook-evals` | 5g · Tracing standards | docs_or_book | [OpenAI Cookbook — Evals](https://developers.openai.com/cookbook/topic/evals) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-140-openai-cookbook-evals.raw.txt` |
| `ae-141-how-to-build-good-language-modeling-benchmarks` | 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming) | web_article | [How to Build Good Language Modeling Benchmarks](https://ofir.io/How-to-Build-Good-Language-Modeling-Benchmarks/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-141-how-to-build-good-language-modeling-benchmarks.raw.txt` |
| `ae-142-ai-agents-that-matter` | 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming) | paper_or_pdf | [AI Agents That Matter](https://arxiv.org/abs/2407.01502) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-142-ai-agents-that-matter.raw.txt` |
| `ae-143-why-we-no-longer-evaluate-swe-bench-verified` | 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming) | web_article | [Why We No Longer Evaluate SWE-bench Verified](https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-143-why-we-no-longer-evaluate-swe-bench-verified.raw.txt` |
| `ae-144-the-leaderboard-illusion` | 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming) | paper_or_pdf | [The Leaderboard Illusion](https://arxiv.org/abs/2504.20879) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-144-the-leaderboard-illusion.raw.txt` |
| `ae-145-the-swe-bench-illusion-when-sota-llms-remember-i` | 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming) | paper_or_pdf | [The SWE-bench Illusion: When SOTA LLMs Remember Instead of Reason](https://arxiv.org/abs/2506.12286) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-145-the-swe-bench-illusion-when-sota-llms-remember-i.raw.txt` |
| `ae-146-establishing-best-practices-for-building-rigorou` | 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming) | paper_or_pdf | [Establishing Best Practices for Building Rigorous Agentic Benchmarks (ABC)](https://arxiv.org/abs/2507.02825) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-146-establishing-best-practices-for-building-rigorou.raw.txt` |
| `ae-147-frontiermath-tiers-1-3-v2-corrected` | 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming) | web_article | [FrontierMath Tiers 1–3 v2 (corrected)](https://epoch.ai/benchmarks/frontiermath-tiers-1-3-v2) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-147-frontiermath-tiers-1-3-v2-corrected.raw.txt` |
| `ae-148-about-30-of-humanity-s-last-exam-answers-are-wro` | 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming) | web_article | [About 30% of Humanity's Last Exam Answers Are Wrong](https://www.futurehouse.org/research-announcements/hle-exam) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-148-about-30-of-humanity-s-last-exam-answers-are-wro.raw.txt` |
| `ae-149-building-on-evaluation-quicksand` | 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming) | blog | [Building on Evaluation Quicksand](https://www.interconnects.ai/p/building-on-evaluation-quicksand) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-149-building-on-evaluation-quicksand.raw.txt` |
| `ae-150-lost-in-simulation` | 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming) | paper_or_pdf | [Lost in Simulation](https://arxiv.org/abs/2601.17087) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-150-lost-in-simulation.raw.txt` |
| `ae-151-swe-bench-can-lms-resolve-real-world-github-issu` | 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming) | paper_or_pdf | [SWE-bench: Can LMs Resolve Real-World GitHub Issues?](https://arxiv.org/abs/2310.06770) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-151-swe-bench-can-lms-resolve-real-world-github-issu.raw.txt` |
| `ae-152-task-specific-llm-evals-that-do-don-t-work` | 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming) | web_article | [Task-Specific LLM Evals that Do & Don't Work](https://eugeneyan.com/writing/evals/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-152-task-specific-llm-evals-that-do-don-t-work.raw.txt` |
| `ae-153-andrej-karpathy-on-evals` | 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming) | web_article | [Andrej Karpathy on evals](https://x.com/karpathy/status/1896266683301659068) | metadata_only | `` |
| `ae-154-a-careful-examination-of-llm-performance-on-grad` | 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming) | paper_or_pdf | [A Careful Examination of LLM Performance on Grade School Arithmetic (GSM1k)](https://arxiv.org/abs/2405.00332) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-154-a-careful-examination-of-llm-performance-on-grad.raw.txt` |
| `ae-155-pervasive-label-errors-in-test-sets-destabilize` | 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming) | paper_or_pdf | [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://arxiv.org/abs/2103.14749) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-155-pervasive-label-errors-in-test-sets-destabilize.raw.txt` |
| `ae-156-are-we-done-with-mmlu-mmlu-redux` | 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming) | paper_or_pdf | [Are We Done with MMLU? (MMLU-Redux)](https://arxiv.org/abs/2406.04127) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-156-are-we-done-with-mmlu-mmlu-redux.raw.txt` |
| `ae-157-livecodebench-holistic-and-contamination-free-ev` | 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming) | paper_or_pdf | [LiveCodeBench: Holistic and Contamination-Free Evaluation of LLMs for Code](https://arxiv.org/abs/2403.07974) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-157-livecodebench-holistic-and-contamination-free-ev.raw.txt` |
| `ae-158-livebench-a-challenging-contamination-limited-ll` | 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming) | repository_or_docs | [LiveBench: A Challenging, Contamination-Limited LLM Benchmark](https://github.com/LiveBench/LiveBench) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-158-livebench-a-challenging-contamination-limited-ll.raw.txt` |
| `ae-159-the-llm-evaluation-guidebook-open-llm-leaderboar` | 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming) | repository_or_docs | [The LLM Evaluation Guidebook (Open LLM Leaderboard team)](https://github.com/huggingface/evaluation-guidebook) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-159-the-llm-evaluation-guidebook-open-llm-leaderboar.raw.txt` |
| `ae-160-holistic-agent-leaderboard-the-missing-infrastru` | 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming) | paper_or_pdf | [Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation](https://arxiv.org/abs/2510.11977) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-160-holistic-agent-leaderboard-the-missing-infrastru.raw.txt` |
| `ae-161-gaming-the-system-goodhart-s-law-exemplified-in` | 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming) | blog | [Gaming the System: Goodhart's Law Exemplified in the AI Leaderboard Controversy](https://blog.collinear.ai/p/gaming-the-system-goodharts-law-exemplified-in-ai-leaderboard-controversy) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-161-gaming-the-system-goodhart-s-law-exemplified-in.raw.txt` |
| `ae-162-a-shared-playbook-for-trustworthy-third-party-ev` | 6 · Benchmark vs. eval (and benchmark integrity: contamination, saturation, label errors, leaderboard gaming) | docs_or_book | [A Shared Playbook for Trustworthy Third-Party Evaluations](https://openai.com/index/trustworthy-third-party-evaluations-foundations/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-162-a-shared-playbook-for-trustworthy-third-party-ev.raw.txt` |
| `ae-163-rewardbench` | 7 · Evals & RL environments (verifiers, reward design, difficulty calibration, lifecycle) | paper_or_pdf | [RewardBench](https://arxiv.org/abs/2403.13787) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-163-rewardbench.raw.txt` |
| `ae-164-the-new-rl-scaling-laws` | 7 · Evals & RL environments (verifiers, reward design, difficulty calibration, lifecycle) | blog | [The New RL Scaling Laws](https://www.interconnects.ai/p/the-new-rl-scaling-laws) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-164-the-new-rl-scaling-laws.raw.txt` |
| `ae-165-spurious-rewards-rethinking-training-signals-in` | 7 · Evals & RL environments (verifiers, reward design, difficulty calibration, lifecycle) | paper_or_pdf | [Spurious Rewards: Rethinking Training Signals in RLVR](https://arxiv.org/abs/2506.10947) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-165-spurious-rewards-rethinking-training-signals-in.raw.txt` |
| `ae-166-the-state-of-post-training-2025` | 7 · Evals & RL environments (verifiers, reward design, difficulty calibration, lifecycle) | blog | [The State of Post-Training 2025](https://www.interconnects.ai/p/the-state-of-post-training-2025) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-166-the-state-of-post-training-2025.raw.txt` |
| `ae-167-reward-hacking-in-reinforcement-learning` | 7 · Evals & RL environments (verifiers, reward design, difficulty calibration, lifecycle) | web_article | [Reward Hacking in Reinforcement Learning](https://lilianweng.github.io/posts/2024-11-28-reward-hacking/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-167-reward-hacking-in-reinforcement-learning.raw.txt` |
| `ae-168-specification-gaming-the-flip-side-of-ai-ingenui` | 7 · Evals & RL environments (verifiers, reward design, difficulty calibration, lifecycle) | blog | [Specification gaming: the flip side of AI ingenuity](https://deepmind.google/blog/specification-gaming-the-flip-side-of-ai-ingenuity/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-168-specification-gaming-the-flip-side-of-ai-ingenui.raw.txt` |
| `ae-169-multi-turn-rl-for-multi-hour-agents-with-will-br` | 7 · Evals & RL environments (verifiers, reward design, difficulty calibration, lifecycle) | blog | [Multi-Turn RL for Multi-Hour Agents — with Will Brown (Prime Intellect)](https://www.latent.space/p/willccbb) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-169-multi-turn-rl-for-multi-hour-agents-with-will-br.raw.txt` |
| `ae-170-position-the-hidden-costs-and-measurement-gaps-o` | 7 · Evals & RL environments (verifiers, reward design, difficulty calibration, lifecycle) | paper_or_pdf | [Position: The Hidden Costs and Measurement Gaps of RLVR](https://arxiv.org/abs/2509.21882) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-170-position-the-hidden-costs-and-measurement-gaps-o.raw.txt` |
| `ae-171-rewardbench-2-advancing-reward-model-evaluation` | 7 · Evals & RL environments (verifiers, reward design, difficulty calibration, lifecycle) | paper_or_pdf | [RewardBench 2: Advancing Reward Model Evaluation](https://arxiv.org/abs/2506.01937) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-171-rewardbench-2-advancing-reward-model-evaluation.raw.txt` |
| `ae-172-reward-modeling-rlhf-book-ch-5` | 7 · Evals & RL environments (verifiers, reward design, difficulty calibration, lifecycle) | docs_or_book | [Reward Modeling (RLHF Book, ch. 5)](https://rlhfbook.com/c/05-reward-models) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-172-reward-modeling-rlhf-book-ch-5.raw.txt` |
| `ae-173-curriculum-rl-from-easy-to-hard-tasks-improves-l` | 7 · Evals & RL environments (verifiers, reward design, difficulty calibration, lifecycle) | paper_or_pdf | [Curriculum RL from Easy to Hard Tasks Improves LLM Reasoning (E2H Reasoner)](https://arxiv.org/abs/2506.06632) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-173-curriculum-rl-from-easy-to-hard-tasks-improves-l.raw.txt` |
| `ae-174-genenv-difficulty-aligned-co-evolution-between-l` | 7 · Evals & RL environments (verifiers, reward design, difficulty calibration, lifecycle) | paper_or_pdf | [GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulators](https://arxiv.org/abs/2512.19682) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-174-genenv-difficulty-aligned-co-evolution-between-l.raw.txt` |
| `ae-175-evaluating-the-effectiveness-of-llm-evaluators` | 8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable) | web_article | [Evaluating the Effectiveness of LLM-Evaluators](https://eugeneyan.com/writing/llm-evaluators/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-175-evaluating-the-effectiveness-of-llm-evaluators.raw.txt` |
| `ae-176-creating-an-llm-as-a-judge-that-drives-business` | 8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable) | blog | [Creating an LLM-as-a-Judge That Drives Business Results](https://hamel.dev/blog/posts/llm-judge/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-176-creating-an-llm-as-a-judge-that-drives-business.raw.txt` |
| `ae-177-who-validates-the-validators-evalgen` | 8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable) | paper_or_pdf | [Who Validates the Validators? (EvalGen)](https://arxiv.org/abs/2404.12272) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-177-who-validates-the-validators-evalgen.raw.txt` |
| `ae-178-llm-evals-faq` | 8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable) | blog | [LLM Evals FAQ](https://hamel.dev/blog/posts/evals-faq/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-178-llm-evals-faq.raw.txt` |
| `ae-179-llm-as-a-judge-rethinking-model-based-evaluation` | 8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable) | blog | [LLM-as-a-Judge: Rethinking Model-Based Evaluations](https://leehanchung.github.io/blogs/2024/08/11/llm-as-a-judge/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-179-llm-as-a-judge-rethinking-model-based-evaluation.raw.txt` |
| `ae-180-judging-llm-as-a-judge-with-mt-bench-and-chatbot` | 8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable) | paper_or_pdf | [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://arxiv.org/abs/2306.05685) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-180-judging-llm-as-a-judge-with-mt-bench-and-chatbot.raw.txt` |
| `ae-181-llms-instead-of-human-judges-a-large-scale-study` | 8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable) | paper_or_pdf | [LLMs Instead of Human Judges? A Large-Scale Study](https://arxiv.org/abs/2406.18403) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-181-llms-instead-of-human-judges-a-large-scale-study.raw.txt` |
| `ae-182-aligneval` | 8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable) | web_article | [AlignEval](https://eugeneyan.com/writing/aligneval/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-182-aligneval.raw.txt` |
| `ae-183-product-evals-in-three-simple-steps` | 8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable) | web_article | [Product Evals in Three Simple Steps](https://eugeneyan.com/writing/product-evals/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-183-product-evals-in-three-simple-steps.raw.txt` |
| `ae-184-statistics-for-ai-ml-part-3-cohen-s-kappa` | 8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable) | blog | [Statistics for AI/ML, Part 3 — Cohen's Kappa](https://leehanchung.github.io/blogs/2025/03/03/cohen-kappa/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-184-statistics-for-ai-ml-part-3-cohen-s-kappa.raw.txt` |
| `ae-185-data-flywheels-for-llm-applications` | 8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable) | blog | [Data Flywheels for LLM Applications](https://www.sh-reya.com/blog/ai-engineering-flywheel/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-185-data-flywheels-for-llm-applications.raw.txt` |
| `ae-186-spade` | 8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable) | paper_or_pdf | [SPADE](https://arxiv.org/html/2401.03038v1) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-186-spade.raw.txt` |
| `ae-187-llm-evaluators-recognize-and-favor-their-own-gen` | 8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable) | paper_or_pdf | [LLM Evaluators Recognize and Favor Their Own Generations](https://arxiv.org/abs/2404.13076) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-187-llm-evaluators-recognize-and-favor-their-own-gen.raw.txt` |
| `ae-188-g-eval-nlg-evaluation-using-gpt-4-with-better-hu` | 8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable) | paper_or_pdf | [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://arxiv.org/abs/2303.16634) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-188-g-eval-nlg-evaluation-using-gpt-4-with-better-hu.raw.txt` |
| `ae-189-a-survey-on-llm-as-a-judge` | 8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable) | paper_or_pdf | [A Survey on LLM-as-a-Judge](https://arxiv.org/abs/2411.15594) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-189-a-survey-on-llm-as-a-judge.raw.txt` |
| `ae-190-one-token-to-fool-llm-as-a-judge` | 8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable) | paper_or_pdf | [One Token to Fool LLM-as-a-Judge](https://arxiv.org/abs/2507.08794) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-190-one-token-to-fool-llm-as-a-judge.raw.txt` |
| `ae-191-weaver-closing-the-generation-verification-gap-w` | 8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable) | blog | [Weaver: Closing the Generation-Verification Gap with Weak Verifiers](https://hazyresearch.stanford.edu/blog/2025-06-18-weaver) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-191-weaver-closing-the-generation-verification-gap-w.raw.txt` |
| `ae-192-agent-as-a-judge-evaluate-agents-with-agents` | 8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable) | paper_or_pdf | [Agent-as-a-Judge: Evaluate Agents with Agents](https://arxiv.org/abs/2410.10934) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-192-agent-as-a-judge-evaluate-agents-with-agents.raw.txt` |
| `ae-193-verifybench-a-systematic-benchmark-for-evaluatin` | 8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable) | paper_or_pdf | [VerifyBench: A Systematic Benchmark for Evaluating Reasoning Verifiers Across Domains](https://arxiv.org/abs/2507.09884) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-193-verifybench-a-systematic-benchmark-for-evaluatin.raw.txt` |
| `ae-194-enhancing-llm-as-a-judge-with-grading-notes-from` | 8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable) | blog | [Enhancing LLM-as-a-Judge with Grading Notes / From Pilot to Production with Custom Judges](https://www.databricks.com/blog/pilot-production-custom-judges) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-194-enhancing-llm-as-a-judge-with-grading-notes-from.raw.txt` |
| `ae-195-justice-or-prejudice-quantifying-biases-in-llm-a` | 8 · LLM-as-judge & verifiers (alignment, biases, verifiable vs judgeable) | paper_or_pdf | [Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (CALM framework)](https://arxiv.org/abs/2410.02736) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-195-justice-or-prejudice-quantifying-biases-in-llm-a.raw.txt` |
| `ae-196-demystifying-evals-for-ai-agents` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | web_article | [Demystifying Evals for AI Agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-196-demystifying-evals-for-ai-agents.raw.txt` |
| `ae-197-bench-bench` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [τ-bench / τ²-bench](https://arxiv.org/abs/2406.12045) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-197-bench-bench.raw.txt` |
| `ae-198-benchmarking-ai-agents` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | blog | [Benchmarking AI Agents](https://sierra.ai/blog/benchmarking-ai-agents) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-198-benchmarking-ai-agents.raw.txt` |
| `ae-199-gaia-a-benchmark-for-general-ai-assistants` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [GAIA: A Benchmark for General AI Assistants](https://arxiv.org/abs/2311.12983) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-199-gaia-a-benchmark-for-general-ai-assistants.raw.txt` |
| `ae-200-patterns-for-building-cybersecurity-evals` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | web_article | [Patterns for Building Cybersecurity Evals](https://eugeneyan.com/writing/cybersecurity-evals/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-200-patterns-for-building-cybersecurity-evals.raw.txt` |
| `ae-201-statistics-for-ai-ml-part-4-pass-k-and-unbiased` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | blog | [Statistics for AI/ML, Part 4 — pass@k and Unbiased Estimator](https://leehanchung.github.io/blogs/2025/09/08/pass-at-k/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-201-statistics-for-ai-ml-part-4-pass-k-and-unbiased.raw.txt` |
| `ae-202-first-principles-eval` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | blog | [First-Principles Eval](https://leehanchung.github.io/blogs/2024/05/22/first-principles-eval/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-202-first-principles-eval.raw.txt` |
| `ae-203-swe-bench-grading-harness` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | repository_or_docs | [SWE-bench grading harness](https://github.com/SWE-bench/SWE-bench/blob/main/swebench/harness/grading.py) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-203-swe-bench-grading-harness.raw.txt` |
| `ae-204-human-eval-pass-k-estimator` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | repository_or_docs | [human-eval (pass@k estimator)](https://github.com/openai/human-eval/blob/master/human_eval/evaluation.py) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-204-human-eval-pass-k-estimator.raw.txt` |
| `ae-205-webarena-a-realistic-web-environment-for-buildin` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://arxiv.org/abs/2307.13854) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-205-webarena-a-realistic-web-environment-for-buildin.raw.txt` |
| `ae-206-osworld-benchmarking-multimodal-agents-for-open` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://arxiv.org/abs/2404.07972) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-206-osworld-benchmarking-multimodal-agents-for-open.raw.txt` |
| `ae-207-terminal-bench-benchmarking-agents-on-hard-reali` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | web_article | [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command-Line Interfaces](https://www.tbench.ai/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-207-terminal-bench-benchmarking-agents-on-hard-reali.raw.txt` |
| `ae-208-cybench-a-framework-for-evaluating-cybersecurity` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risk of Language Models](https://arxiv.org/abs/2408.08926) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-208-cybench-a-framework-for-evaluating-cybersecurity.raw.txt` |
| `ae-209-agentrewardbench-evaluating-automatic-evaluation` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories](https://arxiv.org/abs/2504.08942) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-209-agentrewardbench-evaluating-automatic-evaluation.raw.txt` |
| `ae-210-why-do-multi-agent-llm-systems-fail-mast-taxonom` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [Why Do Multi-Agent LLM Systems Fail? (MAST taxonomy)](https://arxiv.org/abs/2503.13657) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-210-why-do-multi-agent-llm-systems-fail-mast-taxonom.raw.txt` |
| `ae-211-appworld-a-controllable-world-of-apps-and-people` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | web_article | [AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents](https://aclanthology.org/2024.acl-long.850/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-211-appworld-a-controllable-world-of-apps-and-people.raw.txt` |
| `ae-212-browsecomp-a-simple-yet-challenging-benchmark-fo` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | web_article | [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://openai.com/index/browsecomp/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-212-browsecomp-a-simple-yet-challenging-benchmark-fo.raw.txt` |
| `ae-213-locagent-graph-guided-llm-agents-for-code-locali` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [LocAgent: Graph-Guided LLM Agents for Code Localization](https://arxiv.org/abs/2503.09089) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-213-locagent-graph-guided-llm-agents-for-code-locali.raw.txt` |
| `ae-214-webvoyager-building-an-end-to-end-web-agent-with` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://arxiv.org/abs/2401.13919) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-214-webvoyager-building-an-end-to-end-web-agent-with.raw.txt` |
| `ae-215-skillsbench` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | repository_or_docs | [SkillsBench](https://github.com/benchflow-ai/skillsbench) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-215-skillsbench.raw.txt` |
| `ae-216-clawsbench` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | repository_or_docs | [ClawsBench](https://github.com/benchflow-ai/ClawsBench) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-216-clawsbench.raw.txt` |
| `ae-217-swe-bench-verified` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | web_article | [SWE-bench Verified](https://openai.com/index/introducing-swe-bench-verified/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-217-swe-bench-verified.raw.txt` |
| `ae-218-swe-bench-multimodal` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [SWE-bench Multimodal](https://arxiv.org/abs/2410.03859) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-218-swe-bench-multimodal.raw.txt` |
| `ae-219-swe-bench-pro` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [SWE-bench Pro](https://arxiv.org/abs/2509.16941) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-219-swe-bench-pro.raw.txt` |
| `ae-220-swe-lancer` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [SWE-Lancer](https://arxiv.org/abs/2502.12115) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-220-swe-lancer.raw.txt` |
| `ae-221-swe-gym` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [SWE-Gym](https://arxiv.org/abs/2412.21139) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-221-swe-gym.raw.txt` |
| `ae-222-multi-swe-bench` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [Multi-SWE-bench](https://arxiv.org/abs/2504.02605) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-222-multi-swe-bench.raw.txt` |
| `ae-223-swe-rebench` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [SWE-rebench](https://arxiv.org/abs/2505.20411) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-223-swe-rebench.raw.txt` |
| `ae-224-re-bench` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [RE-Bench](https://arxiv.org/abs/2411.15114) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-224-re-bench.raw.txt` |
| `ae-225-mle-bench` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [MLE-bench](https://arxiv.org/abs/2410.07095) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-225-mle-bench.raw.txt` |
| `ae-226-paperbench` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [PaperBench](https://arxiv.org/abs/2504.01848) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-226-paperbench.raw.txt` |
| `ae-227-konwinski-prize-k-prize` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | web_article | [Konwinski Prize (K Prize)](https://www.kaggle.com/competitions/konwinski-prize) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-227-konwinski-prize-k-prize.raw.txt` |
| `ae-228-mind2web-2-evaluating-agentic-search-with-agent` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge](https://arxiv.org/abs/2506.21506) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-228-mind2web-2-evaluating-agentic-search-with-agent.raw.txt` |
| `ae-229-online-mind2web-an-illusion-of-progress-assessin` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [Online-Mind2Web (An Illusion of Progress? Assessing the Current State of Web Agents)](https://arxiv.org/abs/2504.01382) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-229-online-mind2web-an-illusion-of-progress-assessin.raw.txt` |
| `ae-230-real-benchmarking-autonomous-agents-on-determini` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | repository_or_docs | [REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites](https://github.com/agi-inc/REAL) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-230-real-benchmarking-autonomous-agents-on-determini.raw.txt` |
| `ae-231-webgames-challenging-general-purpose-web-browsin` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [WebGames: Challenging General-Purpose Web-Browsing AI Agents](https://arxiv.org/abs/2502.18356) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-231-webgames-challenging-general-purpose-web-browsin.raw.txt` |
| `ae-232-berkeley-function-calling-leaderboard-bfcl-v4` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | web_article | [Berkeley Function Calling Leaderboard (BFCL) V4](https://gorilla.cs.berkeley.edu/leaderboard.html) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-232-berkeley-function-calling-leaderboard-bfcl-v4.raw.txt` |
| `ae-233-gta-a-benchmark-for-general-tool-agents` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [GTA: A Benchmark for General Tool Agents](https://arxiv.org/abs/2407.08713) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-233-gta-a-benchmark-for-general-tool-agents.raw.txt` |
| `ae-234-spider-2-0-evaluating-language-models-on-real-wo` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows](https://arxiv.org/abs/2411.07763) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-234-spider-2-0-evaluating-language-models-on-real-wo.raw.txt` |
| `ae-235-androidworld-a-dynamic-benchmarking-environment` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents](https://arxiv.org/abs/2405.14573) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-235-androidworld-a-dynamic-benchmarking-environment.raw.txt` |
| `ae-236-windowsagentarena-evaluating-multi-modal-os-agen` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [WindowsAgentArena: Evaluating Multi-Modal OS Agents at Scale](https://arxiv.org/abs/2409.08264) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-236-windowsagentarena-evaluating-multi-modal-os-agen.raw.txt` |
| `ae-237-st-webagentbench-evaluating-safety-and-trustwort` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [ST-WebAgentBench: Evaluating Safety and Trustworthiness in Web Agents](https://arxiv.org/abs/2410.06703) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-237-st-webagentbench-evaluating-safety-and-trustwort.raw.txt` |
| `ae-238-theagentcompany-benchmarking-llm-agents-on-conse` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks](https://arxiv.org/abs/2412.14161) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-238-theagentcompany-benchmarking-llm-agents-on-conse.raw.txt` |
| `ae-239-visualwebarena-evaluating-multimodal-agents-on-r` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks](https://arxiv.org/abs/2401.13649) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-239-visualwebarena-evaluating-multimodal-agents-on-r.raw.txt` |
| `ae-240-gdpval-evaluating-ai-model-performance-on-real-w` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks](https://arxiv.org/abs/2510.04374) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-240-gdpval-evaluating-ai-model-performance-on-real-w.raw.txt` |
| `ae-241-remote-labor-index-measuring-ai-automation-of-re` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [Remote Labor Index: Measuring AI Automation of Remote Work](https://arxiv.org/abs/2510.26787) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-241-remote-labor-index-measuring-ai-automation-of-re.raw.txt` |
| `ae-242-humanity-s-last-exam` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [Humanity's Last Exam](https://arxiv.org/abs/2501.14249) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-242-humanity-s-last-exam.raw.txt` |
| `ae-243-scienceagentbench-toward-rigorous-assessment-of` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | repository_or_docs | [ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery](https://github.com/OSU-NLP-Group/ScienceAgentBench) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-243-scienceagentbench-toward-rigorous-assessment-of.raw.txt` |
| `ae-244-core-bench-computational-reproducibility-agent-b` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [CORE-Bench: Computational Reproducibility Agent Benchmark](https://arxiv.org/abs/2409.11363) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-244-core-bench-computational-reproducibility-agent-b.raw.txt` |
| `ae-245-deepresearch-bench-a-comprehensive-benchmark-for` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents](https://arxiv.org/abs/2506.11763) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-245-deepresearch-bench-a-comprehensive-benchmark-for.raw.txt` |
| `ae-246-bixbench-a-comprehensive-benchmark-for-llm-based` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [BixBench: A Comprehensive Benchmark for LLM-based Agents in Computational Biology](https://arxiv.org/abs/2503.00096) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-246-bixbench-a-comprehensive-benchmark-for-llm-based.raw.txt` |
| `ae-247-gaia2-and-are-scaling-up-agent-environments-and` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [Gaia2 and ARE: Scaling Up Agent Environments and Evaluations](https://arxiv.org/abs/2509.17158) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-247-gaia2-and-are-scaling-up-agent-environments-and.raw.txt` |
| `ae-248-vending-bench-a-benchmark-for-long-term-coherenc` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents](https://arxiv.org/abs/2502.15840) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-248-vending-bench-a-benchmark-for-long-term-coherenc.raw.txt` |
| `ae-249-arc-agi-2-a-new-challenge-for-frontier-ai-reason` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems](https://arxiv.org/abs/2505.11831) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-249-arc-agi-2-a-new-challenge-for-frontier-ai-reason.raw.txt` |
| `ae-250-trail-trace-reasoning-and-agentic-issue-localiza` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [TRAIL: Trace Reasoning and Agentic Issue Localization](https://arxiv.org/abs/2505.08638) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-250-trail-trace-reasoning-and-agentic-issue-localiza.raw.txt` |
| `ae-251-crmarena-pro-holistic-assessment-of-llm-agents-a` | 9 · Agent-specific evaluation (trajectories, tool use, multi-turn, world state, multi-agent, localization) | paper_or_pdf | [CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios](https://arxiv.org/abs/2505.18878) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-251-crmarena-pro-holistic-assessment-of-llm-agents-a.raw.txt` |
| `ae-252-benchjack-systematically-auditing-ai-agent-bench` | 10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing) | paper_or_pdf | [BenchJack: Systematically Auditing AI Agent Benchmarks](https://arxiv.org/abs/2605.12673) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-252-benchjack-systematically-auditing-ai-agent-bench.raw.txt` |
| `ae-253-towards-building-safe-secure-agentic-ai` | 10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing) | web_article | [Towards Building Safe & Secure Agentic AI](https://rdi.berkeley.edu/adv-llm-agents/slides/dawn-agentic-ai.pdf) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-253-towards-building-safe-secure-agentic-ai.raw.txt` |
| `ae-254-dawn-song-iclr-2025-keynote-on-llm-safety` | 10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing) | web_article | [Dawn Song — ICLR 2025 keynote on LLM safety](https://iclr.cc/virtual/2025/invited-talk/36783) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-254-dawn-song-iclr-2025-keynote-on-llm-safety.raw.txt` |
| `ae-255-cybergym` | 10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing) | paper_or_pdf | [CyberGym](https://arxiv.org/html/2506.02548v2) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-255-cybergym.raw.txt` |
| `ae-256-air-bench-2024` | 10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing) | paper_or_pdf | [AIR-Bench 2024](https://arxiv.org/abs/2407.17436v2) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-256-air-bench-2024.raw.txt` |
| `ae-257-decodingtrust` | 10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing) | web_article | [DecodingTrust](https://decodingtrust.github.io) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-257-decodingtrust.raw.txt` |
| `ae-258-redcode` | 10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing) | paper_or_pdf | [RedCode](https://arxiv.org/abs/2411.07781) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-258-redcode.raw.txt` |
| `ae-259-agentpoison` | 10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing) | paper_or_pdf | [AgentPoison](https://arxiv.org/abs/2407.12784) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-259-agentpoison.raw.txt` |
| `ae-260-adding-error-bars-to-evals-a-statistical-approac` | 10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing) | paper_or_pdf | [Adding Error Bars to Evals (A Statistical Approach to LM Evaluations)](https://arxiv.org/abs/2411.00640) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-260-adding-error-bars-to-evals-a-statistical-approac.raw.txt` |
| `ae-261-agentdojo-a-dynamic-environment-to-evaluate-prom` | 10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing) | paper_or_pdf | [AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents](https://arxiv.org/abs/2406.13352) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-261-agentdojo-a-dynamic-environment-to-evaluate-prom.raw.txt` |
| `ae-262-agentharm-a-benchmark-for-measuring-harmfulness` | 10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing) | paper_or_pdf | [AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents](https://arxiv.org/abs/2410.09024) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-262-agentharm-a-benchmark-for-measuring-harmfulness.raw.txt` |
| `ae-263-injecagent-benchmarking-indirect-prompt-injectio` | 10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing) | paper_or_pdf | [InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated LLM Agents](https://arxiv.org/abs/2403.02691) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-263-injecagent-benchmarking-indirect-prompt-injectio.raw.txt` |
| `ae-264-defeating-prompt-injections-by-design-camel` | 10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing) | paper_or_pdf | [Defeating Prompt Injections by Design (CaMeL)](https://arxiv.org/abs/2503.18813) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-264-defeating-prompt-injections-by-design-camel.raw.txt` |
| `ae-265-the-lethal-trifecta-for-ai-agents-private-data-u` | 10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing) | web_article | [The lethal trifecta for AI agents: private data, untrusted content, and external communication](https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-265-the-lethal-trifecta-for-ai-agents-private-data-u.raw.txt` |
| `ae-266-shade-arena-evaluating-sabotage-and-monitoring-i` | 10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing) | web_article | [SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents](https://www.anthropic.com/research/shade-arena-sabotage-monitoring) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-266-shade-arena-evaluating-sabotage-and-monitoring-i.raw.txt` |
| `ae-267-agentic-misalignment-how-llms-could-be-insider-t` | 10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing) | web_article | [Agentic Misalignment: How LLMs Could Be Insider Threats](https://www.anthropic.com/research/agentic-misalignment) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-267-agentic-misalignment-how-llms-could-be-insider-t.raw.txt` |
| `ae-268-pyrit-python-risk-identification-tool-for-genera` | 10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing) | repository_or_docs | [PyRIT — Python Risk Identification Tool for generative AI](https://github.com/Azure/PyRIT) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-268-pyrit-python-risk-identification-tool-for-genera.raw.txt` |
| `ae-269-owasp-top-10-for-agentic-applications-2026-llm-a` | 10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing) | web_article | [OWASP Top 10 for Agentic Applications (2026) + LLM Applications (2025)](https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-269-owasp-top-10-for-agentic-applications-2026-llm-a.raw.txt` |
| `ae-270-mitre-atlas-adversarial-threat-landscape-for-ai` | 10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing) | web_article | [MITRE ATLAS — Adversarial Threat Landscape for AI Systems](https://atlas.mitre.org/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-270-mitre-atlas-adversarial-threat-landscape-for-ai.raw.txt` |
| `ae-271-agent-security-bench-asb-formalizing-and-benchma` | 10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing) | paper_or_pdf | [Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents](https://proceedings.iclr.cc/paper_files/paper/2025/file/5750f91d8fb9d5c02bd8ad2c3b44456b-Paper-Conference.pdf) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-271-agent-security-bench-asb-formalizing-and-benchma.raw.txt` |
| `ae-272-gray-swan-x-uk-aisi-agent-red-teaming-challenge` | 10 · Safety / adversarial evaluation (prompt injection, jailbreaks, action-authorization, benchmark auditing) | blog | [Gray Swan x UK AISI Agent Red-Teaming Challenge](https://app.grayswan.ai/arena/blog/agent-red-teaming-the-ai-jailbreak-showdown) | metadata_only | `` |
| `ae-273-how-to-construct-domain-specific-llm-evaluation` | 🎤 Conference & individual talks | talk_video | [How to Construct Domain Specific LLM Evaluation Systems](https://www.youtube.com/watch?v=eLXF0VojuSs) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-hamel-sedgh-domain-eval-systems.md` |
| `ae-274-how-to-look-at-your-data` | 🎤 Conference & individual talks | talk_video | [How to look at your data](https://www.youtube.com/watch?v=jryZvCuA0Uc) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-huber-liu-look-at-your-data.md` |
| `ae-275-failure-is-a-funnel` | 🎤 Conference & individual talks | talk_video | [Failure is a Funnel](https://www.youtube.com/watch?v=k98gDjYbSaU) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-bischof-failure-is-a-funnel.md` |
| `ae-276-using-llms-as-judges-insights-challenges-best-pr` | 🎤 Conference & individual talks | talk_video | [Using LLMs as Judges: Insights, Challenges, Best Practices](https://www.youtube.com/watch?v=7EGF0Mc0_os) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-yan-llms-as-judges.md` |
| `ae-277-scaling-up-vibe-checks-for-llms` | 🎤 Conference & individual talks | talk_video | [Scaling Up Vibe Checks for LLMs](https://www.youtube.com/watch?v=eGVDKegRdgM) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-shankar-scaling-vibe-checks.md` |
| `ae-278-why-llm-data-processing-pipelines-fail` | 🎤 Conference & individual talks | talk_video | [Why LLM Data Processing Pipelines Fail](https://www.youtube.com/watch?v=H-1QaLPnGsg) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-shankar-why-pipelines-fail.md` |
| `ae-279-evals-are-not-unit-tests` | 🎤 Conference & individual talks | talk_video | [Evals Are Not Unit Tests](https://www.youtube.com/watch?v=L8OoYeDI_ls) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-pesok-evals-not-unit-tests.md` |
| `ae-280-building-metrics-that-actually-work-workshop` | 🎤 Conference & individual talks | talk_video | [Building Metrics that actually work (workshop)](https://www.youtube.com/watch?v=jxrGodnopHo) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-karam-metrics-that-work.md` |
| `ae-281-from-self-driving-to-autonomous-voice-agents` | 🎤 Conference & individual talks | talk_video | [From Self-driving to Autonomous Voice Agents](https://www.youtube.com/watch?v=kDczF4wBh8s) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-hopkins-self-driving-voice-agents.md` |
| `ae-282-fuzzing-in-the-genai-era` | 🎤 Conference & individual talks | talk_video | [Fuzzing in the GenAI Era](https://www.youtube.com/watch?v=OMGPvW8TBHc) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-tang-fuzzing-genai.md` |
| `ae-283-on-engineering-ai-systems-that-endure-the-bitter` | 🎤 Conference & individual talks | talk_video | [On Engineering AI Systems that Endure the Bitter Lesson](https://www.youtube.com/watch?v=qdmxApz3EJI) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-khattab-systems-that-endure.md` |
| `ae-284-strategies-for-llm-evals-harnesses-workshop` | 🎤 Conference & individual talks | talk_video | [Strategies for LLM Evals (harnesses workshop)](https://www.youtube.com/watch?v=89NuzmKokIk) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-smith-strategies-for-llm-evals.md` |
| `ae-285-the-future-of-evals` | 🎤 Conference & individual talks | talk_video | [The Future of Evals](https://www.youtube.com/watch?v=MC55hdWLq4o) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-goyal-future-of-evals.md` |
| `ae-286-3-key-ideas-in-ai-in-2025-verifier-s-law` | 🎤 Conference & individual talks | talk_video | [3 Key Ideas in AI in 2025 (Verifier's Law)](https://www.youtube.com/watch?v=b6Doq2fz81U) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-wei-3-key-ideas-2025.md` |
| `ae-287-some-intuitions-about-large-language-models` | 🎤 Conference & individual talks | talk_video | [Some Intuitions About Large Language Models](https://www.youtube.com/watch?v=l898fqkjdFc) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/papers/evaluating-large-language-models-trained-on-code.md` |
| `ae-288-deep-dive-into-llms-like-chatgpt` | 🎤 Conference & individual talks | talk_video | [Deep Dive into LLMs like ChatGPT](https://www.youtube.com/watch?v=7xTGNNLPyMI) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-karpathy-deep-dive-llms.md` |
| `ae-289-rlhf-progress-and-challenges` | 🎤 Conference & individual talks | talk_video | [RLHF: Progress and Challenges](https://www.youtube.com/watch?v=hhiLw5Q_UFg) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-schulman-rlhf-progress-challenges.md` |
| `ae-290-aligning-open-language-models` | 🎤 Conference & individual talks | talk_video | [Aligning Open Language Models](https://www.youtube.com/watch?v=AdLgPmcrXwQ) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/papers/evaluating-large-language-models-trained-on-code.md` |
| `ae-291-building-llm-applications-for-production` | 🎤 Conference & individual talks | talk_video | [Building LLM Applications for Production](https://www.youtube.com/watch?v=spamOhG7BOA) | transcript_queued | `` |
| `ae-292-the-model-is-the-product` | 🎤 Conference & individual talks | talk_video | [The Model is the Product](https://www.youtube.com/watch?v=4dUFIRj-BWo) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-lee-model-is-the-product.md` |
| `ae-293-rl-environments-at-scale` | 🎤 Conference & individual talks | talk_video | [RL Environments at Scale](https://www.youtube.com/watch?v=_IzZWeuTx7I) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-brown-rl-environments-at-scale.md` |
| `ae-294-llm-benchmarks-in-the-time-of-agents` | 🎤 Conference & individual talks | talk_video | [LLM benchmarks in the time of agents](https://www.youtube.com/watch?v=kmTMc-fVSXw) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-brand-benchmarks-time-of-agents.md` |
| `ae-295-evals-error-analysis-and-better-prompts` | 🎙 Podcast episodes | podcast_video | [Evals, error analysis, and better prompts](https://www.youtube.com/watch?v=PgzOBNse2EA) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/articles/paul-iusztin-ai-evals-dataset-error-analysis.md` |
| `ae-296-evals-are-the-new-prd-for-ai-products` | 🎙 Podcast episodes | podcast_video | [Evals are the new PRD for AI products](https://www.youtube.com/watch?v=QE_1hRLsehM) | transcript_queued | `` |
| `ae-297-ep-60-10-things-i-hate-about-ai-evals` | 🎙 Podcast episodes | podcast_video | [Ep 60: 10 Things I Hate About AI Evals](https://www.youtube.com/watch?v=QEk-XwrkqhI) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-pod-vg60-10-things-i-hate.md` |
| `ae-298-ep-50-a-field-guide-to-rapidly-improving-ai-prod` | 🎙 Podcast episodes | podcast_video | [Ep 50: A Field Guide to Rapidly Improving AI Products](https://www.youtube.com/watch?v=rWToRi2_SeY) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-pod-vg50-field-guide.md` |
| `ae-299-five-hard-earned-lessons-about-evals` | 🎙 Podcast episodes | podcast_video | [Five Hard-Earned Lessons About Evals](https://www.youtube.com/watch?v=a4BV0gGmXgA) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-pod-ls-goyal-five-lessons.md` |
| `ae-300-artificial-analysis-independent-llm-evals` | 🎙 Podcast episodes | podcast_video | [Artificial Analysis: Independent LLM Evals](https://www.youtube.com/watch?v=v5mBjeX4TJ8) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/articles/artificial-analysis-independent-llm-evals-latent-space.md` |
| `ae-301-reality-the-final-eval-vending-bench` | 🎙 Podcast episodes | podcast_video | [Reality: The Final Eval (Vending-Bench)](https://www.youtube.com/watch?v=ZAimcoJXUBo) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/articles/andon-labs-reality-final-eval-latent-space.md` |
| `ae-302-5-designing-evals` | 🎙 Podcast episodes | podcast_video | [#5 Designing Evals](https://www.youtube.com/watch?v=-N6MajRfqYw) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-pod-aitw5-designing-evals.md` |
| `ae-303-16-evaluating-prompts-across-models` | 🎙 Podcast episodes | podcast_video | [#16 Evaluating Prompts Across Models](https://www.youtube.com/watch?v=OawyQOrlubM) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/papers/evaluating-large-language-models-trained-on-code.md` |
| `ae-304-24-evals-for-classification` | 🎙 Podcast episodes | podcast_video | [#24 Evals for Classification](https://www.youtube.com/watch?v=5Fy0hBzyduU) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-pod-aitw24-classification-evals.md` |
| `ae-305-34-multimodal-evals` | 🎙 Podcast episodes | podcast_video | [#34 Multimodal Evals](https://www.youtube.com/watch?v=jzhVo0iAX_I) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-pod-aitw34-multimodal-evals.md` |
| `ae-306-372-it-s-2026-and-we-re-still-talking-evals` | 🎙 Podcast episodes | podcast_video | [#372 It's 2026 and We're Still Talking Evals](https://www.youtube.com/watch?v=9EjWR3QpJYk) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-pod-mlops372-still-talking-evals.md` |
| `ae-307-728-generative-benchmarking` | 🎙 Podcast episodes | podcast_video | [#728 Generative Benchmarking](https://www.youtube.com/watch?v=3kbiGPn0cOo) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-pod-twiml728-generative-benchmarking.md` |
| `ae-308-shaping-ai-benchmarks-helm` | 🎙 Podcast episodes | podcast_video | [Shaping AI Benchmarks (HELM)](https://www.youtube.com/watch?v=kwkdKirqi6s) | transcript_queued | `` |
| `ae-309-evaluating-llms-with-chatbot-arena` | 🎙 Podcast episodes | podcast_video | [Evaluating LLMs with Chatbot Arena](https://www.youtube.com/watch?v=okHMaczHPXc) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/papers/bertscore-evaluating-text-generation-with-bert.md` |
| `ae-310-evaluating-ai-designing-for-non-determinism` | 🎙 Podcast episodes | podcast_video | [Evaluating AI, Designing for Non-Determinism](https://www.youtube.com/watch?v=v0eTTn7ZPEc) | transcript_queued | `` |
| `ae-311-karpathy-rl-is-terrible-why-benchmarks-mislead` | 🎙 Podcast episodes | podcast_video | [Karpathy: RL is terrible, why benchmarks mislead](https://www.youtube.com/watch?v=-lRBpyPt79c) | transcript_queued | `` |
| `ae-312-how-to-build-ai-evals-in-2026-step-by-step` | 🎙 Podcast episodes | podcast_video | [How to Build AI Evals in 2026 (Step-by-Step)](https://www.youtube.com/watch?v=J7N9FMouSKg) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-talk-hamel-shreya-build-evals-2026.md` |
| `ae-313-towards-building-safe-trustworthy-ai-agents` | 🎓 University lectures | lecture_video | [Towards Building Safe & Trustworthy AI Agents](https://www.youtube.com/watch?v=QAgR4uQ15rc) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-song-safe-trustworthy-agents.md` |
| `ae-314-towards-building-safe-and-secure-agentic-ai` | 🎓 University lectures | lecture_video | [Towards Building Safe and Secure Agentic AI](https://www.youtube.com/watch?v=ti6yPE2VPZc) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-song-safe-secure-agentic.md` |
| `ae-315-measuring-agent-capabilities-and-anthropic-s-rsp` | 🎓 University lectures | lecture_video | [Measuring Agent Capabilities and Anthropic's RSP](https://www.youtube.com/watch?v=6y2AnWol7oo) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-mann-measuring-capabilities-rsp.md` |
| `ae-316-open-source-and-science-in-the-era-of-foundation` | 🎓 University lectures | lecture_video | [Open-Source and Science in the Era of Foundation Models](https://www.youtube.com/watch?v=f3KKx9LWntQ) | transcript_queued | `` |
| `ae-317-cs336-lecture-12-evaluation` | 🎓 University lectures | lecture_video | [CS336 Lecture 12: Evaluation](https://www.youtube.com/watch?v=x-R5l2HsXqM) | transcript_queued | `sources/21-benchmarks/awesome-evals-primary-sources/notes/talks/talk-cs336-lec12-evaluation.md` |
| `ae-318-pavlovslist-com` | Companies & landscape (eval / RL-environment market) | web_article | [pavlovslist.com](https://pavlovslist.com/) | mirrored | `sources/21-benchmarks/awesome-evals-primary-sources/raw/ae-318-pavlovslist-com.raw.txt` |