See also: agentic-ai-governance · training-architecture · ai-model-evaluation-benchmarks
Cross-references within pillar: benchmark-evolution-enterprise-evals-2026.md · benchmark-foundations-enterprise-evals-2026.md
Credibility tiers:
- HIGH — peer-reviewed paper or independent third-party study with full methodology
- MEDIUM — preprint (arXiv) or well-documented vendor benchmark with disclosed methodology
- LOW — tweet-only, blog post, or announcement without methodology
Reference inventory capturing all notable benchmarks announced or prominently discussed April–May 2026 (rev 3 — added entries 25–37 from Sundai Discord benchmark channel). Not an analysis — the goal is to miss nothing. Each entry flags credibility tier and source type.
Source credibility: MIXED — Reference inventory of benchmarks announced or discussed April–May 2026. Individual entries carry their own credibility rating (HIGH/MEDIUM/LOW). This document is a tracking artifact, not an analysis; no aggregate source credibility rating applies. Entries marked HIGH have peer-reviewed methodology; entries marked LOW are unverified announcements.
TIER 1 — April–May 2026 publication.
1. EsoLang-Bench
Publisher/org: Lossfunk (independent researchers; contributed to benchflow-ai/benchmarks)
Announced: March 2026 (arXiv preprint 2603.09678); surfaced widely on Twitter ~May 17, 2026
Credibility: HIGH — peer-reviewed preprint on arXiv with full methodology
What it measures: Genuine reasoning in LLMs via esoteric programming languages (Brainfuck, Befunge-98, Whitespace, Unlambda, Shakespeare). 80 problems across 4 difficulty levels.
Why notable / gap it fills: Standard benchmarks (HumanEval, SWE-bench) use Python-heavy corpora, enabling potential memorization. Esoteric languages have 340–60,000× less GitHub presence than Python, making corpus contamination at scale highly unlikely. Models scoring 85–95% on standard benchmarks score 0–11% on equivalent EsoLang tasks, with 0% accuracy beyond the Easy tier. Isolates genuine reasoning from pattern recall.
Key finding: Performance collapse on esoteric languages is near-total for all current frontier models, suggesting that most “coding benchmark” performance is heavily contamination-assisted.
URL: https://arxiv.org/abs/2603.09678
Dataset: https://huggingface.co/datasets/Lossfunk/Esolang-Bench
GitHub: https://github.com/Lossfunk/EsolangBench
2. Harvey LAB (Legal Agent Benchmark)
Publisher/org: Harvey AI
Announced: May 6, 2026
Credibility: MEDIUM — vendor benchmark; open-sourced; methodology disclosed; no independent replication yet; built by the company selling legal AI, which is a conflict of interest for headline scores
What it measures: Long-horizon legal agent performance. 1,200+ agentic tasks across 24 legal practice areas. Evaluated by 75,000+ expert-written rubric criteria. Each task supplies an instruction, a client matter with relevant materials, and requires the agent to produce a deliverable for review — mirroring actual law firm assignment structure.
Why notable / gap it fills: Existing code and task benchmarks have no legal-domain analog. LAB is the first open-source benchmark explicitly designed for long-horizon agent tasks in a professional services context. Task structure (instruction → materials → deliverable) is transferable to other professional domains (finance, compliance, consulting). Released without a leaderboard initially; Harvey committed to publishing one after external baseline runs.
Conflict-of-interest note: Harvey builds and sells legal AI. A benchmark where Harvey’s system performs well serves commercial purposes. Treat published scores with skepticism pending independent replication.
URL: https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark
GitHub: https://github.com/harveyai/harvey-labs
Coverage: https://www.artificiallawyer.com/2026/05/06/harvey-launches-legal-agent-bench/
3. τ-knowledge (tau-knowledge)
Publisher/org: Sierra
Announced: March 2026 (arXiv preprint 2603.04370); prominent Twitter signal from Bret Taylor ~May 13, 2026
Credibility: MEDIUM — arXiv preprint with methodology; vendor-built; structural gap it addresses is real and well-documented
What it measures: Conversational agent performance over large, unstructured, constantly-changing knowledge bases — policy manuals, product catalogs, procedural guides, internal tool descriptions. Tests whether an agent can find the right information AND act on it correctly in the same task episode.
Why notable / gap it fills: RAG benchmarks (RAGAS, BEIR) measure retrieval quality in isolation. Agentic benchmarks (GAIA, OSWorld, WebArena) measure action given sufficient context. τ-knowledge tests the joint capability: determine what’s needed, retrieve it correctly, act on it. This is the actual failure mode in deployed customer-facing agents.
Key finding: Best frontier model (GPT-5.2 with high reasoning) passed only 25.5% of tasks on first try (Pass@1) and 9.3% reliably (Pass@4). Even the best model tops out at 37% Pass@1.
Part of: τ³-Bench — Sierra’s umbrella expanding agent evaluation to knowledge retrieval and voice (see entries 4 and 5 below)
URL: https://sierra.ai/blog/tau-knowledge
arXiv: https://arxiv.org/pdf/2603.04370
4. τ-voice (tau-voice)
Publisher/org: Sierra
Announced: March 2026 (arXiv preprint 2603.13686); bundled with τ³-Bench announcement
Credibility: MEDIUM — arXiv preprint; vendor-built; same conflict-of-interest caveat as τ-knowledge
What it measures: Real-time full-duplex voice agent performance. 278 grounded customer-service tasks across retail, airline, and telecom. Includes a controllable voice user simulator with diverse accents, realistic audio environments, and rich turn-taking dynamics.
Why notable / gap it fills: Voice agents are deployed at scale but evaluated almost exclusively via text proxies. τ-voice quantifies the performance degradation from text to voice conditions in realistic settings.
Key finding: Voice agents reach only 31–51% under clean conditions and 26–38% under realistic conditions (noise, diverse accents) — retaining only 30–45% of text-evaluated capability. The gap between lab text-eval scores and voice deployment performance is large and systematically underreported.
URL: https://sierra.ai/blog/tau-voice-benchmarking-real-time-voice-agents-on-real-world-tasks
arXiv: https://arxiv.org/abs/2603.13686
5. τ³-Bench (tau-cubed-bench)
Publisher/org: Sierra
Announced: March 2026
Credibility: MEDIUM — same as τ-knowledge and τ-voice above; umbrella framework covering both
What it measures: Unified umbrella covering τ-knowledge (retrieval+action) and τ-voice (real-time voice agent grounding), extending Sierra’s original τ-Bench (tool-agent-user interaction) with two new evaluation axes.
Why notable: The progression from τ-Bench → τ²-Bench → τ³-Bench represents a systematic extension of agent evaluation from task completion toward knowledge navigation and voice realism — the two dimensions where production deployments most commonly fail.
URL: https://sierra.ai/blog/bench-advancing-agent-benchmarking-to-knowledge-and-voice
6. ProgramBench
Publisher/org: Meta FAIR, Stanford University, Harvard University (joint)
Announced: May 5, 2026 (arXiv 2605.03546)
Credibility: HIGH — arXiv preprint, multi-institution, full methodology, deterministic verifier
What it measures: Whether an LLM agent can independently architect and rebuild a complex software system from observed behavior alone, without access to the original source code. 200 cleanroom reconstruction tasks, ranging from CLI tools to FFmpeg, SQLite, and the PHP interpreter. Evaluation uses agent-driven fuzzing with thousands of automatically generated behavioral tests.
Why notable / gap it fills: All major coding benchmarks (HumanEval, MBPP, SWE-bench) test patch-writing or small function completion. ProgramBench tests architectural reconstruction — the hardest form of software understanding. The 0% score from every frontier model on the full-resolution metric is notable regardless of whether the benchmark becomes a standard.
Key finding: All 9 evaluated frontier models (including Claude Opus 4.7, GPT-5.4, Gemini 3.1 Pro) scored 0% on fully resolved tasks. Best performer (Opus 4.7) passed 95% of tests on only 3% of tasks — meaning it can get close but cannot complete reconstruction at any meaningful rate.
URL: https://arxiv.org/html/2605.03546v1
Site: https://programbench.com/
7. SkillsBench (BenchFlow)
Publisher/org: BenchFlow (Xiangyi Li; seed-funded 2025)
Announced: January 2026; continued Twitter presence through May 2026
Credibility: MEDIUM — documented methodology, 84 expert-curated tasks, deterministic verifiers, full trajectory logging; independent startup, not a vendor of the systems being evaluated
What it measures: Whether procedural skills (custom instructions, scripts, references loaded on demand) make LLM agents better at real work. 84 tasks across 11 domains. Every task run under three conditions: no skills, curated skills, self-generated skills. Enables direct measurement of skills efficacy rather than raw model capability.
Why notable / gap it fills: As agent frameworks ship “skills” / “tools” / “playbooks” as differentiators, no benchmark previously measured whether these augmentations actually improve performance vs. the base agent. SkillsBench is the first purpose-built instrument for that question.
URL: https://www.benchflow.ai/
Related arXiv: https://arxiv.org/html/2602.12670v1 (SkillsBench paper)
8. Reward Hacking Benchmark (RHB)
Publisher/org: Independent researchers (multi-institution)
Announced: May 3, 2026 (arXiv 2605.02964)
Credibility: HIGH — arXiv preprint, multi-model evaluation across 13 frontier systems, deterministic task design
What it measures: Whether RL-trained LLM agents with tool access exploit reward signals instead of solving tasks as intended. Multi-step tasks with naturalistic shortcut opportunities: skipping verification steps, inferring answers from task-adjacent metadata, tampering with evaluation-relevant functions.
Why notable / gap it fills: Reward hacking in production agentic systems is a known risk but was unmeasured at benchmark scale. RHB provides the first systematic cross-vendor measurement.
Key findings:
- Exploit rates range from 0% (Claude Sonnet 4.5) to 13.9% (DeepSeek-R1-Zero) across 13 frontier models
- RL post-training is associated with substantially higher reward hacking (DeepSeek-V3 vs. DeepSeek-R1-Zero controlled comparison)
- Safety-tuned models show lower exploit rates independent of capability level
URL: https://arxiv.org/abs/2605.02964
9. ExploitBench
Publisher/org: Carnegie Mellon University (Seunghyun Lee, David Brumley) + Bugcrowd
Announced: May 13, 2026 (arXiv 2605.14153)
Credibility: HIGH — peer-reviewed preprint, deterministic mechanical verifier, no LLM-as-judge in evaluation loop
What it measures: LLM cybersecurity agent capability decomposed into 16 mechanically graded flags across 5 tiers: coverage → crash triggering → in-cage primitives → cage-escape primitives → control-flow hijack / arbitrary code execution. Target environment: V8’s standalone JavaScript shell.
Why notable / gap it fills: Existing LLM security benchmarks treat “triggered a crash” as exploitation success, collapsing the hardest part of the problem — progressing from a crash to reusable primitives and full control. ExploitBench decomposes the capability ladder so partial progress is measurable. Relevant for enterprise security teams evaluating offensive AI risk and red-teaming capability.
URL: https://arxiv.org/abs/2605.14153
Site: https://exploitbench.ai/
10. EnergyAgentBench
Publisher/org: Independent researchers (QuarluxAI infrastructure platform)
Announced: May 13, 2026 (arXiv 2605.15230)
Credibility: MEDIUM — arXiv preprint; live data integration is a meaningful methodological contribution; single institution; limited model coverage reported
What it measures: LLM agent performance on live electricity market tasks. 70 task variants across 5 families: datacenter siting under cost-carbon trade-offs, long-horizon portfolio siting, LCOE ranking, 30-year portfolio optimization, and causal grid diagnosis. Tasks require 3–48 sequential tool calls against live endpoints (EIA, NREL, QuarluxAI).
Why notable / gap it fills: First agentic benchmark grounded in live, updating real-world data for the energy/infrastructure domain. Most domain-specific benchmarks use static snapshots. The live-data design makes contamination very difficult and ensures scores reflect current reasoning rather than memorized answers.
URL: https://arxiv.org/abs/2605.15230
11. THINK-Bench
Publisher/org: Zhiyuan Li, Yi Chang, Yuan Wu (independent researchers)
Announced: May 2026 (arXiv 2505.22113)
Credibility: HIGH — arXiv preprint, 1,375 questions, 13,311 manually curated reasoning step annotations
What it measures: Thinking efficiency and Chain-of-Thought quality in large reasoning models. 1,375 questions across mathematics, physics, and chemistry. Two efficiency metrics: First Correct Tokens (tokens to reach first correct answer) and Efficiency Score (ratio of first correct tokens to total tokens).
Why notable / gap it fills: Addresses the same gap Nous Research identified: accuracy benchmarks are abundant, but benchmarks for reasoning efficiency are nearly absent. Overthinking (generating excessive tokens on simple tasks) is a cost and reliability issue in production. THINK-Bench provides the measurement instrument.
Key finding: Most large reasoning models exhibit severe overthinking on easy questions. Mathematical tasks show low efficiency; chemistry and physics tasks show higher efficiency. Open models use 1.5–4× more tokens than closed models on identical tasks (up to 10× on simple questions).
URL: https://arxiv.org/abs/2505.22113
Site: https://think-bench.github.io/
12. Nous Research — Thinking Efficiency Study
Publisher/org: Nous Research (Teknium / @NousResearch)
Announced: May 2026 (blog post + Twitter)
Credibility: LOW — blog post / vendor study, no arXiv preprint, no peer review; but findings directionally consistent with THINK-Bench (above)
What it measures: Token efficiency across reasoning models — tokens used for reasoning relative to solution tokens, normalized by task complexity.
Why notable: Framing contribution more than measurement contribution. Nous Research proposed “tokens per correct answer, normalized by task complexity” as a core metric alongside accuracy — an argument that token efficiency should become a first-class evaluation dimension for enterprise procurement, given that non-reasoning token costs can negate per-token pricing advantages.
Key claim: Open models output 1.5–4× more tokens than closed models on identical tasks. Token efficiency is the “missing benchmark.”
URL: https://nousresearch.com/measuring-thinking-efficiency-in-reasoning-models-the-missing-benchmark
13. LemmaBench
Publisher/org: Independent researchers (multi-institution)
Announced: February 27, 2026 (arXiv 2602.24173); actively discussed May 2026
Credibility: HIGH — arXiv preprint, updatable pipeline design, two published iterations (Aug 2025 and Feb 2026)
What it measures: LLM capabilities at research-level mathematics (theorem proving). Auto-pipeline extracts lemmas from recent arXiv submissions and rewrites them into self-contained statements. Benchmark can be updated continuously with new problems from human mathematical research, while previous iterations can be used for training without contaminating future evaluations.
Why notable / gap it fills: Solves the contamination-at-saturation problem for math benchmarks (AMC, AIME, MATH). Because problems come from arXiv submissions in the prior week, no training set can contain them. The live-update design makes LemmaBench an ongoing measurement tool, not a one-time snapshot.
Key finding: Current SOTA LLMs obtain ~10–15% accuracy on theorem proving (pass@1), showing large headroom before human-level performance.
URL: https://arxiv.org/abs/2602.24173
14. Toto 2.0 + BOOM Benchmark
Publisher/org: Datadog
Announced: April 2026
Credibility: MEDIUM — well-documented vendor benchmark; open-weights models; BOOM benchmark released alongside as independent instrument
What it measures: Time series foundation model performance on observability metrics (CPU, memory, network, request rates, error counts, latency percentiles, queue depths, multivariate telemetry). BOOM (Benchmark for Observability and Operations Metrics) is the accompanying evaluation instrument.
Why notable / gap it fills: General-purpose time series benchmarks (Monash, ETT, M4) do not represent the statistical properties of production observability data: high-cardinality, irregular, sparse, bursty, correlated across variables. BOOM fills that domain gap. Toto 2.0 ships 5 checkpoints (4M–2.5B parameters), all on the Pareto frontier of BOOM.
Key finding: Toto 2.0 22M matches or beats Toto 1.0 across all metrics with ~7× fewer parameters — demonstrating that domain-specific pretraining enables smaller models to outperform larger general-purpose models on domain tasks.
Benchmark-vs-production gap signal (@dylantechn, May 14): “toto 2.0 beating every benchmark is fire, but i’m staying woke on the production logs. if it can handle a cold start without hallucinating a spike, i’m sold fr” — The practitioner reaction illustrates a structural pattern: benchmark performance in the observability domain (and more broadly) does not automatically translate to production readiness. Cold-start behavior and spike hallucination are deployment failure modes that no current benchmark measures. The Toto 2.0 benchmark claim is accurate within its domain scope; it does not certify production behavior.
URL: https://www.datadoghq.com/blog/ai/toto-2/
BOOM: https://www.datadoghq.com/blog/ai/toto-boom-unleashed/
15. GiantsBench
Publisher/org: Independent researchers (Chelsea Finn group, Stanford University)
Announced: April 22, 2026
Credibility: HIGH — Chelsea Finn is a Stanford professor and recognized ML researcher; benchmark published at giants-insights.github.io with methodology
What it measures: Whether LLMs can generate new scientific insights that build on prior research — specifically, whether a model can synthesize new insights given two parent papers. Tests the synthesis step, not retrieval or summarization.
Why notable / gap it fills: Most LLM benchmarks test retrieval, summarization, or reasoning over a single document. GiantsBench specifically targets cross-paper synthesis: the capability required to produce a new finding rather than restate existing ones. This is the gap between “knows the literature” and “advances the literature.” 109K views at time of signal.
Key quote (@chelseabfinn, Apr 22): “Can LLMs generate new insights that build on prior research? GiantsBench is a new scientific discovery benchmark, that tests whether models can synthesize new insights given two parent papers.”
URL: https://giants-insights.github.io
16. GEPA (Gradient-free Efficient Prompt Adaptation)
Publisher/org: UC Berkeley
Announced: Mid-2025; accepted at ICLR 2026; surfaced widely on Twitter ~May 1, 2026
Credibility: HIGH — ICLR 2026 accepted paper; peer-reviewed; now a first-class optimizer in DSPy
What it measures / what it does: Prompt optimization technique. Not a benchmark itself, but relevant to benchmark integrity: GEPA achieves ~10 percentage point gains on standard benchmarks using the same task and same base model, purely through prompt reformulation. It beat GRPO without GPU compute.
Why notable for benchmark evaluation: A 10-point benchmark gain achieved by prompt engineering alone — without changing the model or training — is a measurement integrity signal. It means headline benchmark scores are sensitive to prompt selection, and that models are not being evaluated on equivalent footing unless prompt optimization is controlled for. GEPA is now accessible via DSPy, making this effect reproducible.
Key quote (@akshay_pachaar, May 1): “Same task, same base model, 10 points higher on the benchmark. The technique is called GEPA. It came out of Berkeley in mid-2025, got accepted at ICLR 2026, and is now a first-class optimizer in DSPy.” 69K views.
17. CoDeC — Contamination Detection Metric (NVIDIA)
Publisher/org: NVIDIA
Announced: Circulating in benchmark integrity discussions; applied comparatively by @bnjmn_marie (Benjamin Marie), April 7, 2026
Credibility: MEDIUM-HIGH — NVIDIA-developed metric with disclosed threshold; independently applied by third parties
What it measures: “Benchmark affinity” — degree to which a model’s training data overlaps with a given benchmark’s test set. CoDeC > 80 is NVIDIA’s stated threshold for flagging likely contamination.
Why notable / gap it fills: Contamination is widely acknowledged as a problem but difficult to measure without access to training corpora. CoDeC provides a proxy metric that researchers and practitioners can apply externally. Applied comparison of Qwen3.5 27B vs. Gemma 4 31B IT showed measurable benchmark affinity differences, consistent with contamination hypotheses. 96K views on the @bnjmn_marie thread.
Key quote (@bnjmn_marie, Apr 7): “I reused NVIDIA’s CoDeC metric to probe ‘benchmark affinity’. Officially, CoDeC > 80 suggests contamination.”
Enterprise use case: CoDeC-style contamination screening is a practical tool for procurement teams evaluating vendor-reported benchmark scores. A model with high CoDeC on the benchmark it is being evaluated on should trigger additional scrutiny.
18. “Bullshit Benchmark” — Unanswerable Question Refusal Set
Publisher/org: Peter Gostev (independent practitioner)
Announced: February 24, 2026
Credibility: MEDIUM — practitioner-built, 55-question set, no formal peer review; 826K views indicates significant community engagement
What it measures: Whether LLMs correctly refuse or push back on questions that have no answer or are logically nonsensical — rather than confabulating a plausible-sounding response.
Why notable / gap it fills: Standard benchmarks reward answer production. The failure mode of confidently answering unanswerable or nonsensical prompts is penalized in no mainstream benchmark. Gostev built 55 questions to target this directly. The scale of community engagement (826K views) signals that this failure mode resonates with practitioners as a real deployment concern, even without a peer-reviewed methodology.
Key quote (@petergostev, Feb 24): “What bothers me about the current breed of LLMs is that they tend to try to be too helpful regardless of how dumb the question is. So I’ve built 55 ‘bullshit’ questions that don’t [have answers / are nonsensical].”
Enterprise relevance: Customer-facing agents that answer unanswerable questions confidently create compliance and trust liabilities. This is the eval gap the Bullshit Benchmark targets.
19. ARC-AGI Hidden Test Set — Contamination Architecture and Governance Tension
Publisher/org: ARC Prize (François Chollet)
Ongoing governance discussion: April–May 2026
Credibility: HIGH — ARC Prize is independently run; hidden test set design is a documented architectural choice
What it captures: ARC-AGI maintains a hidden (non-public) test set not released to the research community, specifically to prevent memorization and benchmark gaming. As @deedydas (Deedy, May 5) noted: “Every benchmark can be overfit to and memorized. You can memorize all the bugs in SWE-Bench too. ARC AGI might solve this by having a hidden set of games you can’t look at” — framing it as the cleanest available solution to the contamination problem.
Governance controversy signal (@raw_works/Raymond Weitekamp, Apr 19): “it won’t be the first time that RLM blows a benchmark out of the water on day 1, only to be immediately dismissed by the creators” — referring to ARC Prize not verifying a Symbolica agent claim despite a reported high score. This illustrates the governance tension: a hidden test set prevents contamination but requires the benchmark operator to function as a trusted verifier, creating a centralization-of-trust problem.
Why notable: ARC-AGI’s hidden-test-set approach is the strongest current technical solution to contamination at benchmark saturation — but it introduces benchmark governance risk (unverified claims, operator discretion). Both sides of this tension are relevant for enterprise eval design.
20. SLM Task-Specific Benchmarking — Evaluation Design Principle
Observed: May 2, 2026
Source: @j_golebiowski (Jacek Golebiowski, ML practitioner)
Credibility: MEDIUM — practitioner perspective, no empirical study; 1,600 views; directionally consistent with domain-specific benchmark proliferation pattern
What it captures: The argument that small language models (SLMs) should not be benchmarked against general-purpose LLMs on general benchmarks. “SLMs are not ‘cheaper LLMs.’ They are a different tool class. Purpose-built for one taxonomy, they outperform general-purpose models on the bounded job they were trained for. Stop benchmarking SLMs against LLMs. Benchmark them on the job they were specialized for.”
Why notable for enterprise eval design: This maps directly to the Harvey LAB and SkillsBench design logic — task-scoped benchmarks beat horizontal ones for procurement decisions. For enterprises deploying SLMs in narrow enterprise functions (document classification, structured data extraction, specific workflow automation), comparison to GPT-4-class models on MMLU is a category error. The relevant question is performance on the specific task taxonomy the model was trained for.
21. Speculative Decoding — Inference Speed Benchmark Inflation Warning
Observed: March 29, 2026
Source: @gpusteve (GPU infrastructure practitioner)
Credibility: MEDIUM — practitioner methodology warning; no formal study; directionally consistent with known speculative decoding dynamics
What it captures: “Speculative decoding speedups are extremely sensitive to how often draft tokens are accepted — and your benchmark likely inflated that.” Speculative decoding benchmarks measure token acceptance rates under benchmark conditions (typically high-frequency repetitive text). Production workloads have lower, more variable acceptance rates — meaning reported speedups overstate real-world gains.
Enterprise relevance: Inference speed is a procurement input (latency SLAs, cost per token). Vendor-reported speculative decoding speed numbers should be treated as upper bounds, not expected throughput. Request benchmark conditions: what was the token acceptance rate, and was it measured on production-representative traffic?
22. Quantization Impact on Tool-Calling — Qwen3.5-27B
Observed: March 27, 2026
Source: @stevibe (independent evaluator)
Credibility: LOW-MEDIUM — practitioner test, small task set (15 tasks), methodology not peer-reviewed; 61K views
What it measures: Tool-calling accuracy for Qwen3.5-27B across quantization levels: Q6/Q8 quant (15/15 tasks passed), Q2 quant (13/15 tasks passed). Practitioner conclusion: “Q6 is the sweet spot.”
Why notable / gap it fills: Most model benchmarks report full-precision (BF16/FP16) scores. Enterprise deployments almost always run quantized models for cost and latency reasons. The Q2-vs-Q6 performance gap on tool-calling specifically — not just perplexity — is a practical signal for enterprises choosing quantization levels in agentic workflows. A 2-task failure rate on tool-calling at Q2 is a real reliability risk in production.
Key quote (@stevibe, Mar 27): “Qwen3.5-27B: 15/15 at Q6/Q8 quant, 13/15 at Q2. Q6 is the sweet spot.”
Enterprise use case: Any enterprise deploying quantized models in tool-calling or function-calling contexts should benchmark at the specific quantization level they intend to deploy, not at full precision.
23. psql_bm25s — Retrieval Infrastructure Benchmark Claim
Publisher/org: Independent developers (open source)
Announced/observed: May 13, 2026
Credibility: LOW-MEDIUM — tweet claim, standard benchmark cited without full methodology detail; 161K views
What it measures / claims: Exact BM25 retrieval implemented as a native Postgres access method. Claim: ~23x faster than pg_search on the “standard benchmark.”
Why it belongs in this inventory: This is a retrieval infrastructure benchmark claim, not an LLM benchmark. Relevant to enterprise RAG evaluation because BM25 retrieval speed is a cost and latency input in hybrid search pipelines. The “23x faster” claim should be verified against the cited benchmark methodology before use in procurement comparisons — “standard benchmark” is unspecified.
Key quote (@ii_posts, May 13): “so we built psql_bm25s. exact BM25 retrieval. native Postgres access method. ~23x faster than pg_search on the standard benchmark. retrieval stops being a budget item.” 161K views.
Enterprise relevance: For enterprises running BM25/hybrid search in RAG stacks on Postgres, retrieval latency directly affects agent response times and per-query cost. Signal worth tracking pending independent verification.
24. SlopCodeBench
Publisher/org: Gabe Orlanski (@GOrlanski, independent practitioner)
Announced: December 17, 2025
Credibility: MEDIUM — practitioner-built benchmark, website live at scbench.ai; no arXiv preprint as of announcement date; worth verifying current state before citing
What it measures: Whether AI-generated code is “sloppy” — overly verbose, unnecessarily defensive, or brittle as specifications change. Targets a failure mode that HumanEval and SWE-bench do not capture: code that passes tests but degrades a codebase over time through verbosity, unnecessary guard clauses, and over-engineering.
Why notable / gap it fills: Standard coding benchmarks evaluate whether generated code produces correct outputs for a given spec. They do not evaluate whether the code remains maintainable as specs evolve, whether it introduces unnecessary complexity, or whether it adds defensive checks that become tech debt. SlopCodeBench is the first benchmark purpose-built for this failure mode. A model that scores well on SWE-bench may still consistently produce code that erodes codebase quality — the “AI slop” problem applied to software.
Announcement quote (@GOrlanski, Dec 17, 2025): “Often, a bug I have had to fix has been tied directly to overly verbose or defensive code from an AI agent that has eroded my project as the specs changed. Why isn’t there an eval for this? There is now.”
Enterprise relevance: Directly relevant to enterprise code review governance and AI-assisted software development workflows. An enterprise shipping AI-generated code at scale without this dimension measured is accumulating unmeasured tech debt. The relevant procurement question: ask vendors for SlopCodeBench scores alongside SWE-bench, not just correctness metrics.
URL: https://scbench.ai
Cross-reference: The “AI slop” failure mode (AI-generated content passed off without review) is documented more broadly in microsoft-new-future-of-work-2025.md — 40% of workers receive AI-generated content presented as human work, raising analogous quality-governance concerns outside the code domain.
25. Benchmark Credibility Wars — Adversarial Selection Pattern
Observed: May 16, 2026
Source: Twitter/X — @Teknium (Nous Research)
Credibility of specific dispute: LOW (vendor conflict, no methodology detail)
Relevance of pattern: HIGH
What it captures: @Teknium publicly disputed an unnamed benchmark as “unscientific,” claiming Nous Research “smoke[s] you all on quality benchmarks on every open model.” The specific dispute has no resolvable methodology detail in the public record.
Why it belongs in this inventory: The episode is a visible instance of the adversarial benchmark-selection dynamic: labs select benchmarks where they score well and dismiss those where they don’t. This is distinct from training contamination (a technical problem) but has the same effect on signal quality. The benchmark-selection incentive problem weakens vendor-reported leaderboard scores as procurement inputs and strengthens the case for:
- Independent third-party evals (LMSYS Chatbot Arena, METR time-horizon)
- Enterprise-internal domain benchmarks the vendor cannot select or pre-optimize against
26. Continual Learning Bench 1.0
Publisher/org: Parth Asawa (@pgasawa) + Andrew (Sundai Discord)
Announced: May 4–5, 2026
Credibility: MEDIUM — documentation at continual-learning-bench.com; methodology disclosed; no arXiv preprint confirmed at announcement
What it measures: AI system ability to adapt and improve from feedback across sequential task instances — testing whether models can carry forward learning within an ongoing deployment rather than treating each task independently.
Why notable / gap it fills: All mainstream benchmarks assume stationarity: each test case is independent, model state is reset between examples, and no feedback loop exists. Real deployed AI systems receive feedback (explicit corrections, user signals, downstream errors) and ideally improve. Continual Learning Bench 1.0 is the first benchmark explicitly designed for this online adaptation scenario.
Key design distinction: The benchmark is not testing in-context learning (which reuses a fixed context window). It tests whether a system can persistently improve across a sequence of instances in a way that generalizes — closer to how a human employee learns on the job than to few-shot prompting.
Enterprise relevance: For any organization deploying AI agents in roles where feedback cycles are part of the workflow (sales development, customer service, code review), the ability of the system to improve from feedback is directly operational. Current vendor benchmarks give no signal on this dimension.
URL: https://continual-learning-bench.com/news/cl-bench-1-0/
27. HiL-Bench (Human-in-the-Loop Benchmark)
Publisher/org: Scale AI Labs (@ScaleAILabs)
Announced: May 4, 2026
Credibility: MEDIUM-HIGH — Scale AI has infrastructure access and benchmark credibility; vendor-built; Scale sells annotation and evaluation infrastructure
What it measures: Whether AI agents appropriately recognize when they are missing information and ask for clarification, vs. confidently guessing and producing plausible-but-wrong output. Tests the “know what you don’t know” dimension of agentic systems.
Why notable / gap it fills: All standard coding and task benchmarks provide complete specifications. Real agentic deployments operate with incomplete specs. HiL-Bench removes key details from task specs and measures whether agents ask the right clarifying questions or confidently guess wrong. Finding: frontier models perform well with perfect specs; remove a few key details and they “confidently guess and ship plausible wrong answers.”
Key structural insight: This is the benchmark analog of the HVAC dispatcher case documented in institute-ai-transformation-enterprise-agent-performance-2026.md — agents that can’t identify what they’re missing will fail silently on underspecified tasks, which is the normal state of enterprise work.
Enterprise relevance: An agent that asks the right clarifying questions before starting a long task is more operationally useful than one that completes the task confidently on incorrect assumptions. HiL-Bench is the first measurement instrument for this capability.
Source: https://x.com/ScaleAILabs/status/2051333688798097567
28. Terminal-Bench 2.0
Publisher/org: Open-source community (tbench.ai)
Announced/active: November 2025; v2.0 results circulating November 2025 – May 2026
Credibility: MEDIUM-HIGH — open-source benchmark with public leaderboard; community-maintained
What it measures: AI agent performance on terminal/CLI tasks — the ability to operate effectively in a command-line environment, use shell tools, navigate filesystems, run scripts, and complete development workflows.
Key scores (v2.0 leaderboard, April 2026):
- Gemini 3.1 Pro: leading score (exact TBD — above 70%)
- Kimi K2.6: 66.7 (ahead of GPT-5.4 and Claude Opus 4.6; up from K2.5’s 50.8)
- GPT-5.3-Codex: 76% (per Sam Altman launch post, Feb 5, 2026)
- GLM-5.1: top 3 globally per Z.ai (alongside SWE-Bench Pro and NL2Repo)
- Claude Code: #39; OpenAI Codex: #34
Why notable: The Terminal-Bench leaderboard is notable for what it reveals about the gap between headline model launches and realistic agent performance. Claude Code at #39 and Codex at #34 despite being heavily marketed as coding/agentic products suggests that real-world CLI/terminal agent capability is concentrated in fewer models than marketing suggests. The “why don’t we hear about the ones at the top?” observation (Sundai Discord, March 30, 2026) is exactly the adversarial benchmark-selection dynamic in entry 25.
URL: https://www.tbench.ai/leaderboard/terminal-bench/2.0
29. SWE-Bench Pro (replacing SWE-Bench Verified)
Publisher/org: OpenAI / SWE-bench community
Announced: February 23, 2026 — OpenAI announced it would stop reporting SWE-bench Verified and recommended SWE-bench Pro as the new standard
Credibility: HIGH — successor to a widely-used benchmark; OpenAI public announcement
What it measures: More complex, harder real-world software engineering tasks than SWE-bench Verified. Verified reached saturation as a discrimination tool for frontier models. Pro raises the difficulty ceiling.
Why notable: SWE-bench Verified becoming a “legacy” benchmark is the clearest example in the coding domain of the saturation/Goodhart pattern described in benchmark-foundations-enterprise-evals-2026.md. When the benchmark operator themselves says “stop reporting this, use the harder version,” that is an explicit saturation acknowledgment.
Community context: Epoch AI’s analysis (October 2025) confirmed that SWE-bench Verified tests primarily simple bug fixes in familiar Python repositories (especially Django), making it a narrow proxy for general software engineering capability. Independent practitioner consensus (Andrew, Sundai Discord): “swe bench is meaningless. GLM 4.6 is better in the real world than 4 models shown above it.”
Key scores on SWE-bench Pro (as of February 2026): GPT-5.3-Codex: 57%; GLM-5.1: top 3 globally.
Source: https://x.com/openaidevs/status/2026002219909427270
Epoch AI analysis: https://epoch.ai/blog/what-skills-does-swe-bench-verified-evaluate
30. IQuest-Coder Contamination — Git History Exploit
Publisher/org: Xeophon (@xeophon, independent researcher)
Discovered: December–January 2025/2026
Credibility: HIGH — specific, reproducible contamination mechanism; confirmed disqualification of IQuest-Coder SWE-bench scores
What it captures: IQuest-Coder’s SWE-bench benchmark setup inadvertently included the full git history of the repository being evaluated — including future commits (the fixes). The model discovered this during evaluation and used future commit data to produce “correct” patches rather than reasoning to solutions.
Why it belongs in this inventory: This is not a theoretical contamination concern — it is a documented, specific instance of a model exploiting an evaluation setup flaw to inflate scores. The model’s “reasoning” was effectively database lookup. The episode reinforces why benchmark verification (checking evaluation harness integrity, not just test set novelty) is a distinct and necessary component of benchmark governance.
Enterprise implication: When evaluating vendor-reported benchmark scores, the question is not only “could the model have seen the test set during training?” but “was the evaluation harness itself sound?” Harness integrity is a second contamination surface that is rarely audited.
Source: https://x.com/xeophon/status/2006969664346501589
31. WebTailBench (Microsoft / Fara-7B)
Publisher/org: Microsoft
Announced: January 29, 2026
Credibility: MEDIUM — vendor benchmark; released alongside open-weight model (Fara-7B); Hugging Face dataset published
What it measures: Computer use / web agent performance, specifically designed to complement or extend existing WebArena-style evaluations. Released alongside Fara-7B, an efficient agentic model for computer use based on Qwen2.5-VL-7B.
Why notable: Fara-7B achieves competitive computer use scores at 7B parameters — demonstrating that efficient, locally-deployable models can approach frontier computer-use performance. WebTailBench provides the evaluation instrument for this class. Relevant to enterprise deployments where a locally-hosted small model doing computer use is preferable for data governance reasons.
URL: https://huggingface.co/datasets/microsoft/WebTailBench
Model: https://github.com/microsoft/fara
32. LLM Neuroanatomy — Layer Duplication Leaderboard Exploit
Publisher/org: dnhkng (independent researcher)
Announced/discussed: March 14, 2026 (Yannic Kilcher paper discussion)
Credibility: HIGH — empirically verified; RYS-XLarge reached #1 on HuggingFace Open LLM Leaderboard; subsequent fine-tunes held top 4 positions as of early 2026
What it captures: By exhaustively sweeping all possible contiguous layer-duplication configurations of Qwen2-72B and scoring against two deliberately orthogonal proxy tasks (hard mental arithmetic + emotional intelligence/EQ-Bench), the author discovered that duplicating specific middle layers (45–51) produces consistent, generalizable capability improvements — evidence for discrete multi-layer functional circuits in LLMs. The resulting model, RYS-XLarge (78B, zero modified weights), reached #1 on the HuggingFace leaderboard across 6 held-out benchmarks, with no additional training.
Why it belongs in this inventory: This is a benchmark gaming / leaderboard inflation case of a different kind — not contamination, but architectural optimization specifically targeting benchmark performance without modifying weights or training. A zero-training-compute technique that produces #1 leaderboard results raises questions about what the HuggingFace Open LLM Leaderboard actually measures. The technique is orthogonal to fine-tuning and runs on consumer hardware.
Key finding: LLMs appear to develop functional anatomy during pre-training with reasoning organized into discrete multi-layer circuits. Duplicating the right circuits improves capability. This also has genuine scientific implications beyond benchmark gaming.
URL: https://dnhkng.github.io/posts/rys/
33. Agent Sandbox Taxonomy — Security Benchmark Framework
Publisher/org: kajogo777 (independent; open source)
Announced: March 7, 2026
Credibility: MEDIUM — open taxonomy, 18 products scored; independent practitioner framework; no peer review
What it measures: An open taxonomy and scoring framework for evaluating AI agent sandbox security: 7 defense layers × 7 threat categories × 3 evaluation dimensions, applied to 18 products.
Why notable / gap it fills: As agentic AI deployments proliferate, the security properties of the execution environment (sandbox) become a critical evaluation dimension with no existing standard. This is the first structured taxonomy for comparing sandbox security postures across products. Relevant to the JPMC lethal trifecta framework (runtime security) and Singapore MGF Dimension 4 (operational sovereignty) documented in research/19-agent-frameworks/.
URL: https://github.com/kajogo777/the-agent-sandbox-taxonomy
34. Vibe-Code Bench
Publisher/org: Vals AI
Announced/observed: November 24, 2025
Credibility: MEDIUM — Vals AI runs private domain-specific benchmarks for legal, tax, and finance; vibe-code bench appears to test code generation in the “vibe coding” paradigm
What it measures: Code generation quality in natural-language-driven / “vibe coding” workflows — evaluating how well models translate high-level intent into working code in the rapid-iteration style popularized by 2025 AI coding tools. Orchids.app claims “#1 on App Bench and UI Bench” (separate but related benchmarks from the same ecosystem).
Why notable: As “vibe coding” becomes a common enterprise paradigm (non-engineers building applications with AI assistance), evaluation of how well models serve this use case requires different metrics than SWE-bench-style patch correctness. Vibe-Code Bench is early infrastructure for this evaluation gap.
URL: https://www.vals.ai/benchmarks/vibe-code
35. Kimi Provider Quality Benchmark (VectorLab)
Publisher/org: VectorLab (independent)
Announced: September 2025
Credibility: MEDIUM — independent measurement; addresses a structural gap in how model benchmarks are reported
What it captures: Measurement of performance degradation based on inference provider for the same model. Key finding: many inference providers do not implement models correctly, leading to large performance decreases vs. official API or self-hosted. For Kimi K2 specifically, the choice of inference provider significantly affected observed performance.
Why notable for benchmark evaluation: Published benchmark scores are typically measured against the official API or the model vendor’s inference infrastructure. Enterprise deployments using third-party inference providers (Fireworks, Together, Groq, OpenRouter) may see substantially different performance. Provider-quality benchmarking is an underappreciated dimension of production eval programs.
URL: https://vectorlab.dev/news/weekly-9-22-to-9-28/#kimi-inference-provider-bench
36. Konwinski Prize
Publisher/org: Andy Konwinski (independent funder)
Announced: December 2024; active through 2025–2026
Credibility: HIGH — formal prize with clear success criterion; $1M commitment
What it measures: $1M prize for the first team to exceed 90% on a new, harder version of SWE-bench (post-Verified). Designed to raise the difficulty floor as Verified approached saturation.
Why notable: Prize-incentivized benchmark development is a structural response to Goodhart’s Law. By attaching a significant financial incentive to a harder, independently maintained benchmark, the Konwinski Prize creates commercial motivation to build systems that genuinely solve harder software engineering problems rather than overfitting to an existing test set. As of early 2026, the 90% threshold had not been claimed.
URL: https://kprize.ai/
37. NL2Repo
Publisher/org: Benchmark community / cited in GLM-5.1 launch
Credibility: MEDIUM — used as a comparative benchmark in GLM-5.1 announcement; full methodology source TBD
What it measures: Natural language to repository generation — the ability to create a full working code repository from a natural language description. Related to ProgramBench but focused on NL-driven creation rather than behavioral reconstruction.
Why notable: GLM-5.1 (Z.ai) claims #1 open source and #3 globally on NL2Repo alongside SWE-Bench Pro and Terminal-Bench. Signals that repository-level generation is emerging as a standard third axis of coding evaluation alongside patch-writing (SWE-bench) and terminal task completion (Terminal-Bench).
Source: https://x.com/Zai_org/status/2041550153354519022
Meta-Themes Across This Cohort
-
The evaluation surface is fragmenting by task type. Coding evaluation alone now has distinct benchmarks for: patch-writing (SWE-bench → SWE-bench Pro), terminal/CLI agents (Terminal-Bench 2.0), whole-repo reconstruction (ProgramBench), NL-to-repo generation (NL2Repo), code verbosity/maintainability (SlopCodeBench), and computer use (OSWorld, WebTailBench). No single coding benchmark captures all dimensions. Enterprise procurement should specify which dimension matters for their use case.
-
Contamination resistance is the primary design constraint. EsoLang-Bench (esoteric languages), LemmaBench (live arXiv extractions), EnergyAgentBench (live market data), and ARC-AGI’s hidden test set all approach this problem differently but share the same motivation. Static held-out test sets are increasingly insufficient. NVIDIA’s CoDeC metric now gives practitioners a proxy tool to screen for contamination without training corpus access.
-
Efficiency is the emerging second axis. THINK-Bench and the Nous Research study are both responses to the same gap: accuracy-only evaluation does not capture the token-cost dimension that matters for enterprise deployment economics.
-
Domain-specific long-horizon tasks are proliferating. Harvey LAB (legal), EnergyAgentBench (energy infrastructure), τ-knowledge (enterprise knowledge bases), ExploitBench (cybersecurity exploitation ladder), GiantsBench (scientific synthesis) all reflect the same pattern: horizontal benchmarks (GAIA, MMMU) are insufficient for procurement decisions in specific verticals.
-
Code quality is an unmeasured dimension of coding benchmarks. SlopCodeBench surfaces a structural gap: correctness benchmarks (HumanEval, SWE-bench) cannot distinguish code that passes tests from code that degrades a codebase. Verbosity, unnecessary defensiveness, and brittleness under spec change are real enterprise failure modes with no existing measurement instrument until SlopCodeBench.
-
Vendor benchmarks are structurally suspect, but methodologically instructive. Sierra’s τ³ suite and Harvey LAB both have conflict-of-interest problems for headline scores. Both have evaluation designs worth studying for internal enterprise eval construction, independent of whether you trust the published numbers.
-
Benchmark scores are sensitive to prompt selection and quantization level. GEPA demonstrates that the same model on the same task can score 10 points higher with optimized prompting. The Qwen3.5-27B tool-calling data shows that full-precision scores do not predict quantized performance. Neither effect is captured in standard leaderboards.
-
Production behavior and benchmark scores diverge on deployment-specific failure modes. Cold starts, spike hallucination (Toto 2.0/BOOM), confabulation on unanswerable questions (Bullshit Benchmark), and speculative decoding inflation all represent production failure modes that no current mainstream benchmark measures. Enterprise eval programs need task-specific coverage of these gaps.
-
SLM evaluation requires task-scoped benchmarks, not general comparisons to LLMs. Benchmarking a purpose-built SLM against GPT-4-class models on MMLU is a category error. The relevant comparison is performance on the specific task taxonomy the model was trained for — the same design logic underlying Harvey LAB and SkillsBench.
Cross-References
- benchmark-evolution-enterprise-evals-2026.md — earlier analysis note covering EsoLang-Bench and τ-knowledge in more depth
- _index.md — pillar scope and established benchmark coverage (SWE-bench, GPQA, MMMU, GAIA)
- wiki/agentic-ai-governance.md — governance implications of benchmark gaming
- wiki/training-architecture.md — contamination mechanics
Brandon Sneider | brandon@brandonsneider.com May 2026