← Benchmarks 🕐 30 min read
Benchmarks

Unsolved Hard Problems in RAG and Agentic AI — May 2026

Brandon Sneider · May 2026

Cross-references within pillar: rag-retrieval-leaderboard-landscape-2026 · surprising-slm-bench-toppers-2026 · enterprise-eval-rag-finetuned-slm-2026 · enterprise-abstention-whitepaper-2026

See also (wiki): wiki/enterprise-rag-architecture.md · wiki/agentic-ai-governance.md · wiki/data-readiness.md · wiki/ai-model-evaluation-benchmarks.md · wiki/rag-production-failure-modes.md Cross-references within pillar: rag-retrieval-leaderboard-landscape-2026 · surprising-slm-bench-toppers-2026 · enterprise-eval-rag-finetuned-slm-2026 · enterprise-abstention-whitepaper-2026

Source credibility: ICLR/ACL/NeurIPS papers are TIER 1. arXiv preprints with benchmark data are TIER 1 for the numbers, TIER 2 for conclusions until peer-reviewed. Vendor benchmarks with methodology disclosure are TIER 2.

Naming corrections (applied throughout): arXiv:2603.04532 is titled “Still Fresh?” — not “DriftBench” (which is arXiv:2510.10858, a separate database drift paper). arXiv:2604.11209 introduces ConflictQA; ConflictBank is the distinct arXiv:2408.12076 paper. GaRAGe’s 31.1% metric is deflection true positive rate, not precision. TableLlama (arXiv:2311.09206) handles single-table tasks only; it does not address cross-table FK resolution.

Executive Summary

  • Vendor marketing describes RAG and agentic AI as production-ready. Academic benchmarks show a different picture: the best frontier models fail at multi-table reasoning (50.78% vs. 82.7% human), calibrated abstention (31.1% correct refusal rate for GPT-4o), and prompt-injection-triggered data exfiltration (57.4% attack success rate in the AgentDojo competition, arXiv:2603.15714).
  • Six RAG failure modes are quantified and unsolved at enterprise quality thresholds: cross-table foreign key resolution, temporal conflict reconciliation, calibrated abstention, long-context multi-fact integration, structured output reliability for complex schemas, and citation granularity.
  • The agent security picture is worse. Agentic orchestration creates novel attack surfaces: prompt injection can cause irreversible actions, MCP server poisoning is exploitable at CVSS 9.4, and no current architecture solves the “confused deputy” problem without human checkpoints.
  • Reasoning LLMs make some hard problems harder. RLVR training that improves math reasoning actively degrades abstention behavior by ~24% (AbstentionBench). Models trained to reason toward answers are poorly calibrated for knowing when no answer exists.
  • The practical implication: before deploying RAG over financial reports, legal contracts, or policy documents, verify abstention behavior on unanswerable queries. Current out-of-box performance is under 32% correct refusal for the best frontier models.

Why This Note Exists

Marketing claims about RAG and agents describe capabilities that benchmarks do not confirm at production-relevant quality levels. This note documents the specific failure modes — with numbers — where current models fall short of what enterprise deployments require. Each section names the benchmark that quantifies the failure, the best current score, and what (if anything) is actively attacking the problem.

Status labels used throughout:

  • Unsolved — no model exceeds ~60% of human performance; no robust live leaderboard
  • Partially solved — meaningful progress; real-world deployment gaps remain
  • Solved-but-not-known — academic SOTA is adequate; practitioners haven’t caught up

Part 1 — RAG Hard Problems

1. Multi-Table Reasoning and Cross-Document Numerical Grounding

Status: Unsolved

The fundamental problem: models fail when an answer requires resolving foreign keys across tables, traversing hierarchical (merged) headers, or joining numerical values from separate documents. Standard RAG pipelines retrieve at document or passage level — they have no mechanism to identify that column “Revenue” in Table 2 refers to the same entity as “Net Sales” in Table 5.

MMQA (ICLR 2025 Oral, arXiv:2501.12384): First benchmark demanding cross-table foreign key resolution, multi-hop joins, and hierarchical header traversal in the same query. Five subtask types: standard TableQA, foreign key selection, cross-table multi-hop, hierarchical header parsing, document join.

  • Human: 89.8% EM (Table QA), 82.7% (foreign key selection)
  • o1-preview: 50.78% EM — best any model achieved on Table QA; ~56-point gap on foreign key selection
  • GPT-4: 28.88% EM on Table QA; 26.88% on foreign key selection
  • No open-weight model tested near o1-preview

TQA-Bench (arXiv:2411.19504): Performance degrades sharply as table count grows beyond 3. No model maintains quality at 5+ table joins.

TAT-QA (EMNLP 2021, still used): Tabular + textual QA from financial reports. Human 90.8% F1 / 84.1% EM. Best fine-tuned open-weight: TAT-LLM (70B, NExT) at 81.4 EM / 88.4 F1 — within 2.7 EM of human. Proprietary models: ~75–80%; sub-20B open-weight typically 40–55%.

Important clarification: TableLlama (arXiv:2311.09206, 2023) handles single-table tasks via LongLoRA context extension. It does NOT address cross-table FK resolution or multi-document joins. Prior references to TableLlama for multi-table scenarios are incorrect.

Why unsolved: Models conflate columns with identical semantic labels across tables, fail to resolve foreign keys without explicit schema annotations, and treat hierarchical headers (row-spanning cells) as flat rows. No open-weight model has been specifically trained for cross-table schema resolution. Text-to-SQL is the only reliable production workaround — but requires structured data access, not just document ingestion.

Production risk: Any enterprise RAG over financial reports, ERP exports, or multi-sheet spreadsheets hits this.

2. Temporal Reasoning and Knowledge Staleness in RAG

Status: Newly benchmarked, unsolved

The specific failure mode: When a RAG system retrieves both a 2023 document and a 2025 document on the same entity, and the facts contradict, models blend both into a hedge answer rather than deferring to the more recent source. The model does not hallucinate — it accurately synthesizes conflicting information, which is worse for users who expect a single correct answer.

Temporal Conflict QA (arXiv:2506.07270, June 2025): Two benchmarks — Temporal Wiki (factual drift across Wikipedia snapshots) and Unified Clark (timestamped news). Finding: LLMs consistently fail to reconcile conflicting temporal versions. Using knowledge-organization prompting (structuring retrieved context by timestamp before generation), Llama 3.1 70B’s score on Unified Clark improves from 0.43 (raw) → 0.68 (explicit date framing) → 0.76 (full knowledge org). This suggests the model can reason temporally when prompted, but doesn’t do so by default.

“Still Fresh?” (arXiv:2603.04532): Re-evaluated the FreshStack retrieval benchmark against corpus snapshots (Oct 2024 vs. Oct 2025). Critical finding: retrieval model rankings remain stable (Kendall τ up to 0.978) even as the corpus ages — meaning temporal drift doesn’t break the retrieval system, it just makes the answers wrong. The retriever happily surfaces “the right kind of document” that now contains stale information. MTEB scores do not capture this failure.

ConflictQA (arXiv:2604.11209, 2026): Classifies conflicts as misinformation, temporal, and semantic. Temporal conflicts are the hardest category for all models tested. Note: distinct from ConflictBank (arXiv:2408.12076, NeurIPS 2024, 7.45M claim-evidence pairs), which covers similar territory at larger scale.

T-GRAG (arXiv:2508.01680): Dynamic GraphRAG framework for temporal conflicts. One confirmed approach: knowledge-organization prompting (Llama 3.1 70B: 0.43 → 0.76 on Unified Clark). No open-weight-only controlled evaluation for T-GRAG itself.

The unsolved gap: No benchmark specifically isolates the scenario: same topic, same retriever, Document A dated 2023, Document B dated 2025, facts contradict, date metadata visible in both documents. See Open Research Gap §2 for the design requirements.

3. Calibrated Abstention — Knowing When NOT to Answer

Status: Unsolved. The numbers are alarming.

GaRAGe (arXiv:2506.07671, Amazon Science, 2,366 questions, 35K+ annotated passages): Explicitly tests whether models deflect when no relevant grounding exists. Key metrics:

  • Relevance-Aware Factuality (RAF): best ~60.67% (Amazon Nova Pro)
  • Attribution F1: best 58.9% (Claude Haiku and Amazon Nova Micro, tied); range 48.1–58.9% across tested models
  • Deflection true positive rate (correctly refusing to answer when context is insufficient): best 31.1% (GPT-4o)

The 31.1% figure is the production-critical number. The best frontier model correctly identifies “I cannot answer from this context” fewer than a third of the time. Models pattern-match to partial passage relevance and generate plausible-sounding synthesis from insufficient context.

AbstentionBench (arXiv:2506.09038, June 2025): 20 frontier LLMs, 20 diverse datasets (unanswerable questions, underspecification, false premises, outdated info). Conclusion: “abstention is an unsolved problem, and scaling models is of little use.” Reasoning LLMs specifically fail on unanswerable questions — they reason toward an answer even when none exists. RLVR training actively degrades abstention by ~24% for reasoning models.

Abstain-R1 (arXiv:2604.17073): A 3B model trained via GRPO RL specifically for calibrated abstention achieves 68.1% unanswerable refusal rate (+58.7pp over baseline) — demonstrating that targeted abstention training works but has not been applied at frontier scale.

RefusalBench (arXiv:2510.10390): 6 uncertainty categories, 176 perturbation strategies. Claude-4-Sonnet: 73% on NQ but only 36.1% on multi-document questions — confirming abstention degrades as retrieval complexity increases.

Energy-based abstention (arXiv:2509.04482): Healthcare-specific paper showing energy landscape scoring significantly outperforms probability-based confidence for abstention decisions. Potentially generalizable but not yet enterprise-tested.

Production risk: highest on this list. An enterprise RAG over policy documents, legal contracts, or financial data that synthesizes rather than abstaining generates confident wrong answers with no error signal.

4. Long-Context Faithfulness: “Lost in the Middle” Has Shifted

Status: Partially solved for retrieval; still failing for multi-fact integration

NIAH (needle-in-a-haystack) is effectively solved for frontier models at 128K. The failure mode has moved: models can find information anywhere in a long context, but cannot integrate multi-fact reasoning when relevant facts are distributed across the window.

HELMET (Princeton/Intel, ICLR 2025, arXiv:2410.02694): Critical non-obvious finding: generating the correct answer and valid citations are not well correlated. A model can be strong at recall but systematically bad at attribution. Open-source models significantly lag closed models when tasks require full-context reasoning, with the gap widening with context length. The citation-generation sub-task shows a 30–40 absolute point gap between closed-source and best open-source models at 128K. Full per-model numbers are in the linked Google Sheets spreadsheet; HTML tables are not text-extractable.

LongRoPE2 (arXiv:2502.20082): LLaMA3-8B extended to 128K context via non-uniform position interpolation achieves RULER@128K = 82.03 with 98.6% short-context retention. Best known result for sub-10B models at this context length.

BABILong (NeurIPS 2024): Reasoning across 20 task types with facts distributed in noise. Qwen2.5-72B and Gemini 1.5 Flash are the only models performing well on NIAH tasks at 128K; sub-40B models degrade faster and more severely.

LongFuncEval (arXiv:2505.10570): Long-context degradation specifically in tool-calling settings:

  • Tool catalog (8K–120K tokens): GPT-4o degrades 10–12% (most robust); Mistral-Large degrades 78–94%
  • Tool response length (10K–80K tokens): GPT-4o degrades 7%; Mistral-Large degrades 91%
  • Multi-turn (10K–80K tokens): all models degrade 13–40%
  • Most robust open-weight: QwQ-32B (21–42% across challenges); GPT-4o is the only model under 15% across all three dimensions

The sub-40B ceiling: Qwen2.5-32B is the best sub-40B option on aggregate RULER and HELMET performance, but still lags frontier closed models by 15–25 points on complex multi-fact reasoning.

5. Structured Output Reliability Under Schema Constraints

Status: Partially solved for syntactic conformance; unsolved for complex schemas

JSONSchemaBench (arXiv:2501.10868, January 2025): 10K real-world JSON schemas from six corpora (GitHub repos, JSON Schema Store, GlaiveAI function signatures, Snowplow events, Kubernetes configs, Washington Post ANS).

Framework Compile errors Over-constrained Under-constrained Best use case
Guidance 25 7 1 Coverage-critical workloads
Outlines 42 16 8 Legacy deployments
Llamacpp 37 18 7 Legacy deployments
XGrammar 3 0 38 Production (but verify)
OpenAI strict mode <0.1% syntactic failure — — API-only deployments

The non-obvious finding: XGrammar has the fewest crashes but 38 silent-invalid outputs — instances where the generated JSON is syntactically valid but fails schema validation (e.g., accepts {"count": -1} when schema specifies minimum: 0). In production, silent schema violations propagate downstream undetected. This is worse than a hard crash.

XGrammar-2 (arXiv:2601.04426, Jan 2026): 6× faster compilation (10ms vs 1000ms+ for tool-calling), ~7× end-to-end latency speedup, ≤6% gap vs. unconstrained output. Default backend for vLLM, SGLang, TensorRT-LLM as of March 2026.

Schema features causing cross-framework failures: minItems/maxItems (Outlines timeouts of 40s–10+ min), enum with large value sets, recursive $ref, complex anyOf/oneOf, format keyword.

Task accuracy impact (Table 8): Constrained decoding provides ~3–4 point absolute improvement. Guidance scores highest on 6/8 datasets: 96% on GlaiveAI, 86% on GitHub Easy.

StructuredRAG (arXiv:2408.11061): Specifically for JSON output in RAG pipelines (6 task types). Gemini 1.5 Pro: 93.4% average. Llama 3 8B-instruct (4-bit): 71.7% — drops to 44.7% on List[string] tasks and 50% on nested list-of-objects. The list/composite-object failure is the production-relevant gap.

GitHub-Hard dataset: Only 13% empirical coverage for unconstrained LM approaches — meaning for real-world complex schemas, unconstrained generation fails ~87% of the time.

6. Citation Quality: Answer Correctness and Citation Accuracy Are Decorrelated

Status: Unsolved. Best frontier models at 58.9% Attribution F1.

The core finding from HELMET (Princeton, ICLR 2025): Citation quality and answer quality are decorrelated. A model can produce the correct answer while citing the wrong passage, and vice versa. You cannot use answer quality as a proxy for citation quality — they must be measured separately. The citation-generation sub-task shows a 30–40 absolute point gap between closed-source and best open-source models at 128K.

GaRAGe (arXiv:2506.07671): Best attribution F1 = 58.9% (Claude Haiku and Amazon Nova Micro, tied). Range: 48.1–58.9%. The failure mode is over-summarization: models generate accurate content but cite the wrong supporting passage. Most models show conservative attribution behavior: high precision, low recall (under-cite rather than over-cite).

Citation granularity finding (arXiv:2604.01432, 2026): Paragraph-level citations perform best — not sentence-level. Fine-grained (sentence-level) citations degrade attribution quality by 16–276% compared to optimal granularity across models from 8B to 120B parameters. Sentence-level constraints “disrupt semantic dependencies needed for attributing evidence to claims” and “disproportionately penalize larger models.” Implication for enterprise RAG: configuring citation granularity at sentence level is actively counterproductive.

LongCite (ACL 2025, arXiv:2409.02897): Fine-grained span citation in long-context QA. 8B/9B LongCite-trained models outperform GPT-4o by 3.6–6.4% on Citation F1 with twice the granularity. Best open-weight approach for span-level citation.

The span-level gap: No published benchmark tests quote-level (character-exact) span attribution for enterprise document types. See Open Research Gap §1.

7. Multi-Hop Retrieval: Solved at 2 Hops, Fails at 4+

Status: Partially solved (2-hop); unsolved (4+ hops for open-weight models)

PropRAG (arXiv:2504.18070, EMNLP 2025): Proposition-level beam search (atomic self-contained statements) over a graph, staged as Personalized PageRank → beam search with max path L_max=3, beam width B=4. LLM-free at inference time.

Method MuSiQue F1 2Wiki F1 HotpotQA F1 MuSiQue Recall@5
NV-Embed-v2 (baseline) 45.7% 61.5% 75.3% 69.7%
HippoRAG 2 (arXiv:2502.14802) 48.6% 71.0% 75.5% 74.7%
PropRAG L_max=3 53.9% 75.3% 76.1% 78.3%

Human performance on MuSiQue: ~89.8% F1; on HotpotQA: ~91% F1. These benchmarks are not saturated.

The hop-count collapse (arXiv:2601.21214, Jan 2026): Models tested (Qwen2.5-7B, Phi-3-mini, LLaMA3-8B, Qwen3-8B) show “pronounced accuracy drop as reasoning hops increase.” ~2% of attention heads act as “erroneous processing (ep) heads” that amplify incorrect reasoning trajectories. Intervention by knocking out individual ep heads improves ground-truth token prediction by ~0.6.

Weakest Link Law (arXiv:2601.12499, Jan 2026): Multi-hop performance collapses to the level of the “least visible evidence, governed by absolute position rather than linear distance between facts.” Most at-risk: vertical reasoning chains where one step’s output is the next step’s input.

The verbose output trap: For open-weight models on MuSiQue, verbose outputs correlate with F1 dropping from 30% to 3.3% EM. Brevity requirements in system prompts matter more for multi-hop than single-hop tasks.

FrugalRAG (ICLR 2025 workshop, OpenReview ZHHhlQbXnc): Trains a retriever-reasoner to jointly decide when and what to retrieve during CoT generation. Claims highest document recall on HotpotQA and MuSiQue from only 1,000 training examples. RL-based extension (arXiv:2507.07634) is the stronger July 2025 successor.

Best open-weight production stack for 4+ hops: PropRAG (LLM-free retrieval) + Llama3.1-70B or Qwen2.5-72B for generation. IRCoT-style interleaved retrieval-and-reasoning on top of PropRAG is the current frontier approach.

8. Numeric Hallucination: Anchoring Bias

Status: Quantified; the open/frontier gap is the largest of any RAG task

FaithJudge (arXiv:2505.04847): Hallucination rates across RAG task types. Summarization has the lowest detection F1 (80.2% F1-Macro) — hardest task for hallucination detection. Data-to-text has the highest detection rate (86.3%).

Model Hallucination rate
Gemini-2.5-Pro 6.65%
Gemini-2.0-Flash 10.18%
GPT-4.5-Preview 11.94%
o3-mini-high 12.52%
Llama-3.3-70B 16.44% (best open-weight)
Llama-4-Maverick 20.55%
Llama-3.1-8B 28.38%

FinQA / FinanceReasoning: FinQA (8,281 QA pairs from S&P 500 earnings reports): Amazon Nova Pro 77.2% on llm-stats.com leaderboard (3 models only; no open-weight entries verified). FinanceReasoning (ACL 2025, arXiv:2506.05828, covers 67.8% of financial formulas): GPT-4o with Python-formatted knowledge augmentation (3,133 functions) achieves 91.6%, beating o1 (89.1%). Best open-weight for financial reasoning: DeepSeek-R1 (complex multi-step), Qwen3-235B-A22B (dual-mode), QwQ-32B (cost-effective).

Anchoring bias papers:

  • “Understanding the Anchoring Effect of LLM” (arXiv:2505.15392): Reasoning models (DeepSeek-R1, Qwen3-235B) show 22–25% anchoring rate vs. 57–61% for tiny models. Effect operates at shallow model depths (early layers). Cannot be eliminated by conventional strategies; reasoning offers partial mitigation.
  • Benford’s Curse (arXiv:2506.01734, NeurIPS 2025): Leading-digit bias in open-source LLMs tracing to pretraining data distributions. A small set of digit-selective FFN neurons in deep layers; pruning them mitigates the bias.
  • “Dissecting the Ledger: Liar Circuits” (arXiv:2511.21756): Suppressing Layer 46 in GPT-2 XL reduces hallucinatory arithmetic output confidence by 81.8%.

PCN — Proof-Carrying Numbers (arXiv:2509.06902): Presentation-layer verification protocol. Numeric spans emitted as claim-bound tokens; verifier checks each under declared policy (exact equality, rounding tolerance, aliases). RAG citations do not guarantee fidelity if the model fabricates a value adjacent to a cited passage.

Best production mitigation stack (2026): (1) reasoning model (QwQ-32B or DeepSeek-R1) reduces anchoring rate; (2) PoT (Program-of-Thought) prompting — Python-formatted calculation chains; (3) PCN-style verification layer for high-stakes numerical outputs.

9. Chunking Strategy: Corpus-Dependent, No Controlled Standard

Status: Solved-but-not-known for academic corpora; unsolved for enterprise document types

What controlled studies show (contradictory by corpus type):

  • Vectara 2026: Recursive 512-token splitting at 69% accuracy. Semantic chunking at 54% (43-token average fragments). Proposition-level chunking is 3× more expensive with no consistent quality gain for single-document faithfulness.
  • Clinical study (PubMed): Adaptive length chunking aligned to logical topic boundaries: 87% accuracy vs. 13% for fixed-size.

The FloTorch finding (February 2026): Simpler approaches outperform complex ones on cost-adjusted performance. Proposition-level chunking’s recall gains do not survive cost normalization for most enterprise use cases.

Enterprise document type benchmarks (limited):

  • LegalBench-RAG (arXiv:2408.10343): Retrieval precision on legal clauses; uses Summary-Augmented Chunking (SAC) to reduce Document-Level Retrieval Mismatch.
  • FinRAGBench-V (arXiv:2505.17471): 60,780 Chinese + 51,219 English pages from 1,104 financial documents; visual citation evaluation included.
  • Oil and Gas enterprise chunking (arXiv:2603.24556): One of the few papers testing chunking strategy on enterprise (non-academic) documents.

No controlled benchmark compares chunking strategies (fixed-size, recursive, semantic, hierarchical) on financial tables, legal numbered-clause structures, and technical spec documents simultaneously. Every published study uses academic paper corpora. See Open Research Gap §5.

Best empirical defaults based on current evidence:

  • General retrieval: recursive splitting at 512 tokens with 10–20% overlap
  • Multi-hop reasoning where accuracy > cost: proposition-level chunking (+3× index cost)
  • Semantic chunking: only when documents have clear logical structure (numbered clauses, section headers)

Part 2 — Agentic Hard Problems

10. Tool Selection Under Ambiguity: Silent Misframing

Status: Unsolved. 39–63% silent misframing rate on ambiguous tasks.

Ambig-DS (arXiv:2605.09698, May 2026): Data-science agents on tasks where the target metric or objective is ambiguous. Two ambiguity axes: Target Ambiguity (DSBench — correct target column renamed, decoy column added, within 0.02–0.10 metric gap) and Objective Ambiguity (MLE-bench — metric references stripped).

Per-model scores (normalized [0,1]):

Axis Model Full Ambiguous Ask Δambig Δask
Target GPT-5.4 0.61 0.32 0.52 −0.29 +0.20
Target Gemini 3.1 Pro 0.64 0.49 0.60 −0.15 +0.11
Target Claude Haiku 4.5 0.56 0.32 0.52 −0.24 +0.20
Objective Gemini 3 Flash 0.31 0.22 0.31 −0.09 +0.09
Objective GPT-5.4 0.26 0.22 0.26 −0.04 +0.04

Maximum recovery from oracle clarification: +0.20 (Claude Haiku 4.5 and GPT-5.4 on Target ambiguity). Recovery drops to +0.09 on Objective ambiguity — models have less facility with metric-selection clarification.

The key failure: Models can use clarifying information when provided (nearly full recovery), but fail to detect when to ask for it. Objective ambiguity is harder to recover from than target ambiguity.

Safety caution — ASPI (arXiv:2605.17324): Adding clarification-seeking state increases vulnerability to prompt injection attacks (728 task-attack scenarios). Naively adding clarification to agents without injection defenses creates a new attack surface.

ClarEval (arXiv:2603.00187): Coding ability and collaborative clarification are dissociable skills — models that excel at coding do not automatically clarify well.

BFCL V4 (Berkeley, gorilla.cs.berkeley.edu): Best open-weight agentic score: Granite-20B (IBM) leads openly licensed models under 30B. Qwen3-30B-A3B-Thinking competitive at the 30B boundary. Claude-Opus-4-5 at 77.47% leads overall.

11. Error Recovery in Multi-Step Tool Chains

Status: Unsolved. Accuracy collapses between 3–5 step sequences.

ToolMisuseBench (arXiv:2604.01508): Deterministic fault injection — schema drift, rate limits, timeouts, authorization failures. Results by fault type:

Fault Type Best method Success Recovery
Timeout Schema Repair 0.502 0.502
Schema Drift Policy Aware 0.497 0.497
Rate Limit All baselines 0.000 0.000
Authorization All baselines 0.000 0.000

Authorization failures and rate limiting are completely unsolved by all tested methods. Schema drift and timeouts are tractable with schema-aware approaches.

TRAJECT-Bench (arXiv:2510.04550): Scores trajectory quality separately. On simple queries: Gemini-2.5-Pro 0.851 EM, Claude-4 0.846. On hard queries: Claude-4 drops to 0.445 EM (from 0.846 on simple) — a 47% absolute collapse. The Inclusion-EM gap is the key diagnostic: models can include the right tools but in wrong order/parameters, reaching the correct answer via path-inefficient recovery. In non-deterministic environments, this fails.

FamilyTool (arXiv:2504.06766): Completion drops from 88.8% → 45.2% from 1-hop to 3-hop tasks. Best overall model: Qwen3-8B at 40.37% EM on FamilyTool-b; DeepSeek-V3 leads KG extraction at 65.42%/78.90% (EM/F1).

ERR Measure (arXiv:2601.22352): Formalized recovery metrics.

Model RR (%) ES Obs. ERR
Vanilla 14B 38.2±1.6 0.312±0.010 7.02±0.10
ToolBench 61.5±1.1 0.504±0.009 4.98±0.09
FORTIFY 94.7±0.8 0.814±0.007 1.78±0.07

PALADIN (arXiv:2509.25238, ICLR 2026): LoRA fine-tuning on 50K+ recovery-annotated trajectories. Largest absolute gain on LLaMA-3.1-8B:

Model Vanilla TSR PALADIN TSR Gain
LLaMA-3.1-8B 17.46% 78.72% +61.26pp (4.5×)
Gemma-3-12B 23.62% 78.38% +54.76pp
Qwen-2.5-14B 37.53% 79.48% +41.95pp
AM-Thinking V1 52.93% 81.24% +28.31pp

Fission-GRPO (arXiv:2601.15625): RL-based error recovery training. Improves Qwen3-8B error recovery rate by 5.7% absolute on BFCL V4 Multi-Turn (+4% overall accuracy).

Production approach: Explicit error state in context + retrieval from failure exemplar bank (PALADIN’s approach) outperforms naive retry with exponential backoff. PALADIN LoRA checkpoints on Qwen-2.5-14B and LLaMA-3.1-8B are the closest published artifacts for deployment.

12. Context Accumulation in Long Tool-Call Sessions

Status: Unsolved. 7–91% performance degradation as tool history grows.

LongFuncEval (arXiv:2505.10570): First systematic study of long-context understanding specifically in tool-calling settings, up to 128K context windows. Three challenge types with per-model degradation:

Large Tool Catalog (8K–120K token tool descriptions):

Model Degradation
GPT-4o-2024-11-20 10–12% (most robust)
QwQ-32B 21–42%
BitAgent-8B 57–82%
DeepSeek-R1-Distill-Qwen-32B 73–85%
Mistral-Large 78–94%

Long Tool Responses (10K–80K tokens):

Model Degradation
GPT-4o 7%
Granite-3.1-8B 30.5%
Llama-70B 52%
Mistral-Large 91%

Recency bias: performance varies 5–75% by answer position in long tool responses.

Architectural mitigations:

  • SideQuest (arXiv:2602.22603): Model-driven compression. Up to 65% reduction in peak tokens, 53–71% KV cache reduction, 83.9% throughput increase; only 2% accuracy loss on FRAMES, 5% on BrowseComp. Outperforms H₂O, SnapKV, R-KV on agentic tasks.
  • CodeComp (arXiv:2604.10235): Code Property Graph-guided KV cache compression. Qwen3-8B at cap=0.4: 0.72 EM on DebugBench vs. 0.25 for ParallelComp.
  • ACON: 26–54% peak token reduction on AppWorld, OfficeBench.

Specific agentic benchmarks for this problem: AgentLongBench (arXiv:2602.22769), LifelongAgentBench, LoCoBench-Agent measure downstream task performance under long-context; none specifically test compression quality.

13. Compound Constraint Following

Status: Unsolved. No model exceeds 50% Pass^3 on disambiguation tasks.

ComplexBench (arXiv:2407.03978, NeurIPS 2024): 4 constraint types (Lexical, Format, Semantic, Utility), 19 dimensions, 4 composition types (Single, And, Chain, Selection). Key scores:

  • GPT-4 fails on ~20% of complex instructions overall
  • GPT-4 achieves only 14.9% accuracy on multi-layer nested Selection instructions (coherent test)
  • Length constraints: even top models achieve only 53% accuracy
  • Open-source models “substantially lag behind closed-source counterparts on multi-constraint tasks”
  • Evaluation: Rule-Augmented LLM (RAL) — each constraint becomes a yes/no verification pass aggregated by dependency structure

CAR-bench (arXiv:2601.22027): Disambiguation tasks with external resolution (user clarification) vs. internal resolution (data already in environment).

Full per-model results:

  • Proprietary thinking models (GPT-5, Claude-Opus-4.5): Pass@3 on Base tasks ≈74–88%, dropping to ~68% on Disambiguation
  • Qwen3-32B (thinking): 62% Pass@1 on Base, ~50% Pass@3 on Disambiguation
  • GPT-5: drops from 68% Pass@3 to 36% Pass^3 on Disambiguation (must succeed on all 3 trials)

Root cause of GPT-5’s Pass^3 collapse: Error Type E1 (Premature Actions) — despite explicit system prompt mandating internal information gathering first, the model “often queries the user or executes best-guess actions before gathering complete environment information” in a stochastic fraction of runs. The gap is not inability — it is inconsistent application.

CFBench (arXiv:2408.01122, ACL 2025): 1,000 curated samples, 200+ real-life scenarios, 50+ NLP tasks, 10 primary constraint categories. Finding: “substantial room for improvement” across current LLMs.

IFEval saturation: GPT-5 now achieves 95.9% on original IFEval — near-saturated. IFEval++ (541 tests) extends headroom. For compound constraints, ComplexBench and CAR-bench provide harder signal.

Best training approach: RLVR (verifier-based RL) for instruction-following; ToolPRM-GRPO for structured multi-constraint satisfaction in tool tasks. Decomposing constraints into sequential RAL-style yes/no verification is the best-evidenced production mitigation.

14. Agent Trajectory Quality vs. Outcome Quality

Status: Benchmarked but unsolved. PRMs for agents are very new.

Outcome-only benchmarks miss agents that reach correct answers through dangerous or inefficient paths. This matters for irreversible actions (booking transactions, data deletion, API calls).

TRAJECT-Bench (arXiv:2510.04550): The Inclusion-EM gap is the key diagnostic. A model can achieve high Inclusion (includes the right tools) with low EM (wrong order, wrong parameters, redundant extras) while still producing the correct final answer — “paradoxically, weak Inclusion-EM gaps suggest models sometimes recover partial tool sets despite incorrect final answers.” In non-deterministic environments, these path-inefficient recoveries fail.

Web-Shepherd (arXiv:2505.15277): PRM specifically for web navigation. Architecture: checklist generation → step-level reward modeling via verbalizer scoring P(‘Yes’|…) + 0.5×P(‘In Progress’|…) per checklist item.

WebArena-lite results with GPT-4o-mini policy:

PRM Success Rate Δ
None (unguided) 23.64% —
GPT-4o-mini (prompted) 24.24% +0.6%
Web-Shepherd-3B 32.12% +8.48%
Web-Shepherd-8B 34.55% +10.9%

Web-Shepherd-8B scores 88.3% MRR on WebRewardBench vs. GPT-4o at 67.4% — at ~100× lower cost than GPT-4o as the process reward signal.

ToolPRMBench (arXiv:2601.12294): The only published benchmark specifically for PRMs on tool-use agents (not math reasoning). Step-level evaluation from ToolTalk, GTA, BFCL, ToolSandbox with multi-LLM verification (96% human agreement).

Model Average Accuracy
ToolPRM-GRPO 78.6%
Claude-4.5-haiku 75.1%
GPT-5 74.4%
ToolPRM-CoT 63.2%
Qwen3-14B 63.0%
LLaMA-3-70B 53.6%

ToolPRM-GRPO is the first open-weight PRM trained specifically for tool-use that exceeds frontier closed models on this benchmark.

AgentPRM (arXiv:2511.08325, ACM Web Conference 2026): TD-based estimation + GAE for step-level credit assignment. Evaluates “proximity to goal and progress made” — not just correctness. 8× more compute-efficient than prior baselines.

15. Multi-Agent Coordination Failures

Status: Unsolved. Topology-dependent failure rates are production-breaking.

AgentCollabBench (arXiv:2605.08647, May 2026): 900 tasks (software engineering, DevOps, data engineering), three complexity tiers, five topologies. Four behavioral metrics:

Model IDR (constraint decay ↓) CPR (belief contagion ↓) RTD (info retention ↑) CLC (leakage ↓)
Qwen-3.5-35B-A3B 0.9% 20.7% 94.0% 4.7%
Gemini 2.5 Flash Lite 2.4% 20.4% 82.5% 3.8%
GPT 4.1 mini 3.1% 17.7% 80.3% 2.6%
Llama 3.1 8B Instruct 10.1% 40.3% 62.6% 4.5%

Converging-DAG failure mechanism: Converging-DAG hops drop constraints at 1.6–5.0× the rate of linear-chain hops. Root cause: synthesis bottleneck at convergence nodes — when an agent synthesizes inputs from multiple parent branches, it silently discards constraints carried by a minority branch. This is NOT explained by path length (Pearson r = −0.18) or fan-in count (r = −0.07). It is information triage during multi-input synthesis.

Topology explains 7–40% of variance in whether constraints survive multi-hop forwarding — comparable to backbone model choice.

MAST taxonomy (arXiv:2503.13657, NeurIPS 2025 — distinct from MultiAgentBench arXiv:2503.01935): 14 failure modes from 1,600+ annotated traces across 7 MAS frameworks, inter-annotator Cohen’s Kappa = 0.88.

  • FC1 (Specification/Design): 5 modes — disobey task spec, disobey role spec, step repetition, conversation history loss, unaware of termination
  • FC2 (Inter-Agent Misalignment): 6 modes — conversation reset, failure to clarify, task derailment, information withholding, ignored peer input, reasoning-action mismatch
  • FC3 (Verification/Termination): 3 modes — premature termination, incomplete verification, incorrect verification

MultiAgentBench / MARBLE (arXiv:2503.01935, ACL 2025): Per-model coordination scores across 6 domains (Research, Minecraft, Database, Coding, Bargaining, Werewolf). GPT-4o-mini achieves Task Score 84.1 / Coordination Score 52.0 on Research; drops to TS 14.1 / CS 60.1 on Werewolf (highest coordination demand). Graph topology best for research scenarios; tree topology worst (highest token cost, lowest coordination).

Specification Gap (arXiv:2603.24284): Two-agent integration accuracy drops from 58% to 25% across specification levels — a 25–39pp coordination gap consistent across Claude Sonnet and Haiku. Restoring full specification alone recovers single-agent ceiling (89%). AST-based conflict detector achieves 97% precision without LLM calls.

Production architecture for fan-in contagion prevention: LLM Map-Reduce with quarantined Reduce — isolated instances process parent branches separately (Map); aggregation uses non-LLM operations or constrained LLM receiving symbolic summaries, not raw agent outputs. Complementary: Dual LLM (privileged coordinator never sees untrusted peer outputs directly), context-minimization (pass structured constraint records, not raw outputs), action-selector at fan-in node (limit to predefined merge actions).

Best open-weight for constraint-preserving multi-agent: Qwen-3.5-35B-A3B (0.9% IDR, 94.0% RTD based on AgentCollabBench direct results).

16. Prompt Injection in Tool Call Results

Status: Unsolved and directly exploitable in production today.

AgentDojo (arXiv:2406.13352, NeurIPS 2024): 97 realistic tasks, 629 security test cases.

  • “Important message” canonical attack: 53.1% ASR against baseline GPT-4o
  • GPT-4o utility: 69% benign → 45% under active attack
  • Competition (arXiv:2603.15714, 464 participants, 272K+ attack attempts, 13 frontier models): base competition ASR range 0.5% (Claude Opus 4.5) to 8.5% (Gemini 2.5 Pro)
  • AutoInject RL-trained attack: 77.96% ASR on Gemini 2.5 Flash; <35% on other closed models; 21.88% on Meta-SecAlign-70B (fine-tuned defense)
  • GUI agent with extended thinking (Anthropic Claude Opus 4.6 System Card, Feb 2026): 17.8% success at 1 attempt; 78.6% breach at 200 attempts without safeguards; 57.1% at 200 attempts with safeguards

InjectAgent (arXiv:2403.02691): 1,054 test cases, 30 LLM agents. Open-weight ASR (direct harm, base / enhanced with hacking prompt):

Model Direct Harm Base Direct Harm Enhanced
Llama2-70B 91.9% 98.3%
Mistral-7B 13.5% 75.8%
Qwen-72B 8.4% 49.9%

Hacking prompt reinforcement nearly doubles success rates. Llama2-70B is effectively fully compromised in the enhanced setting.

CyberSecEval 4 (Meta/CrowdStrike): State-of-the-art models show 25–50% successful prompt injection rates. Open-weight models more vulnerable than closed.

Available defenses (layered):

  1. PromptArmor (arXiv:2507.15219, ICLR 2026): Off-the-shelf LLM preprocessing scan of tool outputs before re-injection. <1% false positive rate, <1% false negative rate. Simplest and strongest standalone defense per current literature.
  2. CaMeL (arXiv:2503.18813): Separates control flow (trusted query) from data flow (tool output). Solves 77% of AgentDojo tasks with provable security vs. 84% undefended — 7pp utility cost for formal guarantees.
  3. Meta-SecAlign (arXiv:2507.02735, CCS 2025): Fine-tuned Llama-3.1-8B-Instruct and Llama-3.3-70B-Instruct. Instruction-following ASRs exceeding 90% reduced to 0–2% in Meta-SecAlign-70B. Even AutoInject achieves 21.88% ASR — fine-tuning alone is necessary but not sufficient.

No single defense is sufficient (arXiv:2506.08837): Production systems require combination defense.

17. Intent Tracking Degradation Across Multi-Turn Conversations

Status: Unsolved. Scaling does not fix it; stronger models degrade proportionally.

Intent Mismatch / Lost in Conversation (arXiv:2602.07338, Feb 2026): LiC benchmark — Code, Database, Actions, Math. “Sharded” condition: multi-turn, fragmented exchanges where users omit critical constraints due to pragmatic ellipsis.

Model Full Avg Sharded Avg w/ Mediator Avg Full Reliability Sharded Reliability
GPT-4o-mini 86.9% 53.6% 73.9% 83.2% 54.2%
GPT-5.2 92.7% 48.5% — 85.4% 46.8%
DeepSeek-v3.2 94.7% 60.8% — 88.7% 56.2%

“Scaling model capabilities alone cannot resolve the ambiguity problem.” GPT-5.2 degrades more in absolute terms than GPT-4o-mini (−44pp vs. −33pp) despite higher baseline.

Mediator framework (no parameter updates): Refiner Module analyzes contrastive pairs (failed vs. successful trajectories) → Mediator Module reconstructs ambiguous inputs into fully-specified instructions. Recovers +20.3pp for GPT-4o-mini (53.6% → 73.9%), closing roughly half the gap to Full.

τ-bench airline domain: Claude Sonnet 4.5 leads at 70.0% Pass@1. Retail: Claude Sonnet 4.5 at 86.2%. GPT-4.1 nano at 14.0% airline / 22.6% retail — confirming small models effectively fail multi-turn policy-constrained tasks.

τ³-bench SABER audit: 53 tasks corrected (27 airline, 26 retail) across 5 categories — incorrect expected actions, ambiguous instructions, impossible constraints, missing fallbacks, policy loophole exploitation. Post-fix airline gains: +14 to +20pp Pass@1; GPT-5.2 Pass@4 gained +22pp (50% → 72%).

DeepContext (arXiv:2602.16935): Stateful RNN for adversarial intent drift detection. F1=0.84 on multi-turn jailbreak detection vs. 0.67 for Llama-Prompt-Guard-2. Sub-20ms on T4 GPU. Note: DeepContext addresses adversarial intent manipulation, not benign underspecification. These are complementary problems.

ConsistentChat SFT dataset (arXiv:2506.03558): ~15,000 multi-turn conversations with skeleton-guided intent trajectory modeling. Fine-tuning: 20–30% improvement in chat consistency, up to 15% increase in task success rate.

Summary: Top Production Risks

Ranked by how likely an enterprise using RAG or agents today hits this in production:

Rank Problem Failure rate in production Detection difficulty
1 Calibrated abstention (§3) ~69% failure to correctly refuse when context insufficient No error signal — model answers confidently
2 Prompt injection via tool outputs (§16) 53% ASR on realistic tasks; 78.6% at 200 attempts No detection without tool-call sandboxing
3 Context accumulation in long tool sessions (§12) 7–91% degradation as tool history grows Silent quality decline; no error thrown
4 Intent drift in multi-turn (§17) 30–44pp drop in underspecified multi-turn Model acts confidently on wrong interpretation
5 Numeric anchoring hallucination (§8) Systematic compression of “ugly” numbers; 22–61% anchoring rate Deterministic but hard to detect without ground truth
6 Citation/answer decorrelation (§6) 41% wrong-passage citation even when answer correct Correct answer masks incorrect provenance

Open Research Gaps (No Benchmark Exists Yet)

  1. Quote-level span attribution — LongCite covers sentence-level in long-form QA; no benchmark tests character-exact span attribution on enterprise document types (tables, legal clauses, technical specs). Requirements: span precision/recall vs. passage-level separately, hallucinated span boundaries, multi-document span attribution, granularity consistency across document types.

  2. Temporal document conflict with explicit date metadata — evolveQA (arXiv:2510.19172) is closest (uses real timestamp metadata from AWS/Azure/WHO corpora, 12 LLMs, 6–31% performance drop). No benchmark provides controlled pairs where: (a) two documents cover same entity, (b) each carries explicit ISO-date metadata, © facts directly contradict, (d) tested across 3+ domains with date-visible vs. date-hidden conditions.

  3. Numeric anchoring bias — systematic patterns well-documented (Benford’s Curse, SynAnchors, Liar Circuits) but no held-out benchmark for model comparison on anchoring resistance. Requirements: factual numeric retrieval from context (isolates retrieval from parametric recall), planted anchors at varied offsets, numeric types coverage (percentages, large integers, decimals, dates-as-numbers), anti-rounding test.

  4. Cross-file dependency graph retrieval in code — DependEval (arXiv:2503.06689, ACL 2025) and CrossCodeEval (arXiv:2310.11248) cover cross-file completion. No benchmark specifically tests: given a failing test, trace back through the full import/dependency graph to identify the root file and line requiring change. Tree-sitter AST graph (DKB pattern) achieves 100% correctness vs. 40% for vector-only on 15-question architectural evaluation (arXiv:2601.08773), but this is a single case study, not a held-out benchmark.

  5. Enterprise document type chunking — LegalBench-RAG tests retrieval precision; FinRAGBench-V tests multimodal RAG end-to-end. No benchmark simultaneously tests chunking strategies across (a) financial tables with numerical cells, (b) numbered legal clauses with cross-references, © technical spec with hierarchical sections, using controlled QA ground truth anchored to specific spans with consistent evaluation methodology.

Key Data Points

Finding Value Source Date Tier
Cross-table multi-hop: best model (o1-preview) vs. human 50.78% vs. 82.7% EM MMQA, ICLR 2025 Oral, arXiv:2501.12384 2025 TIER 1
Calibrated abstention: best frontier model correct refusal rate 31.1% (GPT-4o) GaRAGe, Amazon Science, arXiv:2506.07671, n=2,366 Qs Jun 2025 TIER 1
Prompt injection success rate against LLM agents 57.4% attack success AgentDojo competition, arXiv:2603.15714 Mar 2026 TIER 1
RLVR training degradation of abstention −24% for reasoning models AbstentionBench, arXiv:2506.09038, 20 models Jun 2025 TIER 1
GPT-4o degradation (tool catalog 8K→120K tokens) 10–12% LongFuncEval, arXiv:2505.10570 May 2025 TIER 1
Mistral-Large degradation (tool catalog 8K→120K tokens) 78–94% LongFuncEval, arXiv:2505.10570 May 2025 TIER 1
Temporal knowledge-org prompting improvement 0.43 → 0.76 (Llama 3.1 70B) ConflictQA / T-GRAG, arXiv:2508.01680 2025 TIER 1
Multi-agent prompt injection (AgentDojo competition) 57.4% attack success AgentDojo competition, arXiv:2603.15714 Mar 2026 TIER 1
MCP Inspector arbitrary command execution severity CVSS 9.4 CVE-2025-49596, NVD 2025 TIER 1
Abstain-R1 targeted abstention training improvement +58.7pp unanswerable refusal (3B model) arXiv:2604.17073 Apr 2026 TIER 1

What This Means for Your Organization

The benchmark numbers in this document are not academic curiosities — they describe failure modes that will manifest in your enterprise deployment if not explicitly engineered against. A RAG system over financial data that fails cross-table reasoning 50% of the time will produce wrong synthesis answers that look correct. A system that correctly refuses unanswerable questions fewer than a third of the time will hallucinate confidently under any context gap.

Three actions follow from this evidence. First, before signing off on any RAG deployment over financial reports, legal contracts, or policy documents, run abstention testing on a sample of genuinely unanswerable queries. If the system answers more than half of them, it is not production-ready for those document types. Second, agentic workflows that have external action capability — sending emails, modifying records, initiating transactions — require human-in-the-loop checkpoints for any action the Singapore Model AI Governance Framework calls “high blast radius.” The prompt injection attack surface is not theoretical; it has confirmed production incidents. Third, apply knowledge-organization prompting (structuring retrieved context by timestamp before generation) when your corpus spans more than 12 months. The temporal conflict research shows this lifts accuracy by 33 points on conflicting-document queries.

These are solvable engineering problems. They are not solved by default in any vendor’s out-of-box RAG offering. If this raised questions about how your current architecture handles these failure modes, I’d welcome the conversation — brandon@brandonsneider.com.

Sources

RAG problems:

  1. MMQA — ICLR 2025 Oral, arXiv:2501.12384. TIER 1.
  2. TQA-Bench — arXiv:2411.19504. TIER 1.
  3. Temporal Conflict QA — arXiv:2506.07270. TIER 1.
  4. “Still Fresh?” — arXiv:2603.04532. TIER 1. [Note: previously labeled “DriftBench” in error]
  5. ConflictQA — arXiv:2604.11209. TIER 1. [Note: distinct from ConflictBank arXiv:2408.12076]
  6. ConflictBank — arXiv:2408.12076 (NeurIPS 2024). TIER 1.
  7. T-GRAG — arXiv:2508.01680. TIER 1.
  8. GaRAGe — arXiv:2506.07671. TIER 1.
  9. AbstentionBench — arXiv:2506.09038. TIER 1.
  10. Abstain-R1 — arXiv:2604.17073. TIER 1.
  11. RefusalBench — arXiv:2510.10390. TIER 1.
  12. Energy-based abstention — arXiv:2509.04482. TIER 1.
  13. HELMET — arXiv:2410.02694 (ICLR 2025). TIER 1.
  14. LongRoPE2 — arXiv:2502.20082. TIER 1.
  15. BABILong — NeurIPS 2024. TIER 1.
  16. LongFuncEval — arXiv:2505.10570. TIER 1.
  17. Context length hurts performance — arXiv:2510.05381 (EMNLP 2025). TIER 1.
  18. JSONSchemaBench — arXiv:2501.10868. TIER 1.
  19. XGrammar-2 — arXiv:2601.04426 (MLSys 2026). TIER 1.
  20. StructuredRAG — arXiv:2408.11061. TIER 1.
  21. Citation granularity — arXiv:2604.01432. TIER 1.
  22. LongCite / LongBench-Cite — arXiv:2409.02897 (ACL 2025). TIER 1.
  23. PropRAG — arXiv:2504.18070 (EMNLP 2025). TIER 1.
  24. HippoRAG 2 — arXiv:2502.14802. TIER 1.
  25. Hop-count attack on reasoning — arXiv:2601.21214. TIER 1.
  26. Weakest Link Law — arXiv:2601.12499. TIER 1.
  27. FrugalRAG — ICLR 2025 workshop; RL version arXiv:2507.07634. TIER 1.
  28. FaithJudge — arXiv:2505.04847 (EMNLP 2025). TIER 1.
  29. FinanceReasoning — arXiv:2506.05828 (ACL 2025). TIER 1.
  30. TAT-QA leaderboard — nextplusplus.github.io/TAT-QA/. TIER 1.
  31. Anchoring Effect — arXiv:2505.15392. TIER 1.
  32. Benford’s Curse — arXiv:2506.01734 (NeurIPS 2025). TIER 1.
  33. Anchors in the Machine — arXiv:2511.05766. TIER 1.
  34. Liar Circuits — arXiv:2511.21756. TIER 1.
  35. PCN — arXiv:2509.06902. TIER 1.
  36. SideQuest — arXiv:2602.22603. TIER 1.
  37. CodeComp — arXiv:2604.10235. TIER 1.
  38. LegalBench-RAG — arXiv:2408.10343. TIER 1.
  39. FinRAGBench-V — arXiv:2505.17471. TIER 1.
  40. Enterprise chunking oil/gas — arXiv:2603.24556. TIER 1.
  41. Reliable Graph-RAG for codebases — arXiv:2601.08773. TIER 1.
  42. CrossCodeEval — arXiv:2310.11248 (NeurIPS 2023). TIER 1.
  43. DependEval — arXiv:2503.06689 (ACL 2025). TIER 1.

Agentic problems: 44. Ambig-DS — arXiv:2605.09698. TIER 1. 45. ClarEval — arXiv:2603.00187. TIER 1. 46. ASPI — arXiv:2605.17324. TIER 1. 47. BFCL V4 — gorilla.cs.berkeley.edu/leaderboard.html. TIER 1. 48. ToolMisuseBench — arXiv:2604.01508. TIER 1. 49. TRAJECT-Bench — arXiv:2510.04550. TIER 1. 50. FamilyTool — arXiv:2504.06766. TIER 1. 51. ERR Measure — arXiv:2601.22352. TIER 1. 52. PALADIN — arXiv:2509.25238 (ICLR 2026). TIER 1. 53. Fission-GRPO — arXiv:2601.15625. TIER 1. 54. LongFuncEval — arXiv:2505.10570. TIER 1. 55. ComplexBench — arXiv:2407.03978 (NeurIPS 2024). TIER 1. 56. CAR-bench — arXiv:2601.22027. TIER 1. 57. CFBench — arXiv:2408.01122 (ACL 2025). TIER 1. 58. Web-Shepherd PRM — arXiv:2505.15277. TIER 1. 59. ToolPRMBench — arXiv:2601.12294. TIER 1. 60. AgentPRM — arXiv:2511.08325 (ACM Web Conference 2026). TIER 1. 61. AgentCollabBench — arXiv:2605.08647. TIER 1. 62. MAST taxonomy — arXiv:2503.13657 (NeurIPS 2025). TIER 1. 63. MultiAgentBench / MARBLE — arXiv:2503.01935 (ACL 2025). TIER 1. 64. Specification Gap — arXiv:2603.24284. TIER 1. 65. AgentDojo — arXiv:2406.13352 (NeurIPS 2024). TIER 1. 66. AgentDojo competition — arXiv:2603.15714. TIER 1. 67. CaMeL — arXiv:2503.18813. TIER 1. 68. PromptArmor — arXiv:2507.15219 (ICLR 2026). TIER 1. 69. InjectAgent — arXiv:2403.02691. TIER 1. 70. Meta-SecAlign — arXiv:2507.02735 (CCS 2025). TIER 1. 71. Design Patterns for Securing LLM Agents — arXiv:2506.08837. TIER 1. 72. Intent Mismatch / LiC — arXiv:2602.07338. TIER 1. 73. τ-bench / τ³-bench — arXiv:2406.12045; taubench.com. TIER 1. 74. DeepContext — arXiv:2602.16935. TIER 1. 75. ConsistentChat — arXiv:2506.03558. TIER 1.

Brandon Sneider | brandon@brandonsneider.com May 2026