← Benchmarks 🕐 16 min read
Benchmarks

Artificial Analysis — Inference Benchmarks and Independent Evaluation Landscape 2026

Brandon Sneider · May 2026

> **Source credibility: MEDIUM-HIGH. TIER 1.**

See also (wiki): wiki/inference-economics.md · wiki/ai-platform-selection.md · wiki/token-economics.md · wiki/ai-model-evaluation-benchmarks.md

Source credibility: MEDIUM-HIGH. TIER 1. Artificial Analysis (artificialanalysis.ai) = HIGH credibility for inference performance metrics (independent, vendor-neutral, reproducible, continuous testing cadence). Their Intelligence Index quality rankings = MEDIUM credibility (methodology is documented but weighting choices are opaque and text-only/English-only as of May 2026). LMSYS Chatbot Arena = MEDIUM for general preference ranking, LOW for frontier differentiation due to documented gaming vulnerabilities. Stanford HELM = HIGH for multi-dimensional capability evaluation; entering maintenance mode June 1, 2026. Epoch AI price trend analysis = HIGH (peer-reviewed, primary data). [NEEDS VERIFICATION] flags assertions that could not be confirmed against a primary source during this research pass.

Executive Summary

  • Provider choice for the same model matters more than most procurement teams realize. Artificial Analysis data shows Claude 4.5 Haiku running 16% faster on Amazon Bedrock (106.5 t/s) than on Anthropic directly (91.8 t/s), while Google Vertex delivers 37% lower time to first token (0.59s vs. 0.93s on Amazon). Azure charges a 71% price premium ($1.40 vs. $0.82/M blended tokens) for mid-tier throughput. The model on the contract is not the performance specification.
  • The Intelligence Index top tier is a statistical three-way tie. As of April 2026, GPT-5.5, Claude Opus 4.7, and Gemini 3.1 Pro Preview all cluster at 57–60 points — within the index’s ±1% confidence band. The composite score masks large differences in component strengths: Opus 4.7 leads on agentic work (GDPval-AA ELO: 1,753 vs. Gemini’s 1,314), while Gemini leads on scientific reasoning (GPQA Diamond: 94.1%).
  • Cost-per-index-point varies 5× at the same composite score. Gemini 3.1 Pro Preview and Claude Opus 4.7 both score 57 on the Intelligence Index; Gemini reaches it at $15.65 per index point vs. $77.31 for Opus 4.7. Organizations paying a premium for one top model over another on composite scores are likely not getting a performance advantage — they are getting a specialization advantage that may or may not match their workload.
  • Inference pricing declined 75% in roughly 18 months (2024–early 2026) for non-reasoning models — but frontier reasoning models have held price, creating a two-tier market. Budget tightening on legacy API lines while scaling reasoning tokens is the current CFO trap.
  • BIG-Bench Hard is saturated; the benchmark landscape is being rebuilt. Frontier models cluster at 92–94% on BBH, making it useless for discrimination. The replacement ecosystem — HLE, GPQA Diamond, SWE-bench Verified, BBH Extra Hard, LiveCodeBench — is more resistant to contamination but still evolving. HELM enters maintenance mode June 1, 2026.
  • LMSYS Arena is directionally useful but gameable. Research published January 2025 demonstrated vote manipulation can artificially move model rankings. Arena’s specialized leaderboards (Coding, Math, Hard Prompts) are more reliable than the Overall ranking.

1. Artificial Analysis: What It Measures and How

1.1 Organizational background

Artificial Analysis is an independent benchmarking organization that measures AI model and API provider performance across three dimensions: quality (Intelligence Index), inference speed, and pricing. Unlike lab-reported benchmarks, Artificial Analysis tests live API endpoints on a continuous schedule, generating reproducible, provider-level data that reflects real-world conditions. As of May 2026, their leaderboard covers 519 models.

Their vendor-neutral positioning is structurally reinforced by their business model (subscription data access, not vendor partnerships) and their methodology transparency — the full testing schedule, prompt templates, scoring formulas, and tokenization approach are publicly documented at artificialanalysis.ai/methodology.

1.2 Performance benchmarking methodology

Artificial Analysis measures three performance metrics:

Time to First Token (TTFT): Time in seconds between sending a request and receiving the first response token. For reasoning models, this captures the first reasoning token, not the final answer. Testing occurs from Google Cloud’s us-central1-a zone — a methodological note that slightly advantages providers with US-West infrastructure and slightly disadvantages those with European or Asian primary clusters.

Output Speed (tokens/second): Average tokens received per second after the first token. For reasoning models that do not expose internal reasoning tokens, Artificial Analysis uses the final 80% of answer chunks to ensure measurement consistency across model types.

Pricing: Standardized using OpenAI’s GPT-4 tokenizer (o200k_base via tiktoken) across all models to enable cross-model comparison. The methodology documentation explicitly flags this as an approximation: models use different tokenizers, so the same text generates different token counts, making direct pricing comparisons imprecise at the margins.

Testing cadence:

  • Standard workloads (1k and 10k input tokens, vision): 8 tests daily (~3-hour intervals)
  • Parallel workload tests (10 concurrent requests): once daily
  • Long-context (100k input tokens): once weekly

This produces approximately 56–64 API calls per model per day on standard workloads. The continuous cadence matters: it catches provider degradation events that one-time snapshots miss.

1.3 Intelligence Index methodology (v4.0, January 2026)

The Intelligence Index is a composite score aggregating ten evaluations across four categories, each weighted equally at 25%:

Category (25% each) Benchmarks Focus
Agents GDPval-AA (220 real-world tasks, Elo-scored), τ²-Bench Telecom (114 dual-control simulations), Terminal-Bench Hard (44 terminal-task scenarios) Agentic execution
Coding SciCode (288 scientific subproblems) Scientific computing
General AA-LCR (100 long-context reasoning questions), AA-Omniscience (6,000 knowledge questions with hallucination penalty), IFBench (294 instruction-following tasks) Breadth
Scientific Reasoning HLE — Humanity’s Last Exam (2,158 frontier academic questions), GPQA Diamond (198 graduate-level science questions), CritPt (70 research-level physics challenges) Depth

Scoring uses pass@1 methodology (correct answer on first attempt). Multi-repeat evaluations aggregate attempts: pass@1 = (Σ correct answers) / (total attempts). GDPval-AA uses Elo ratings normalized as clamp((Elo–500)/2000). AA-Omniscience averages accuracy and non-hallucination rate equally.

Version 4.0 changes (January 2026): MMLU-Pro, LiveCodeBench, and AIME 2025 were removed from the index — the three benchmarks most heavily cited in AI company marketing — because frontier models had saturated them. GDPval-AA, AA-Omniscience, and CritPt were added. The category structure shifted to emphasize agent capability alongside scientific reasoning.

Known methodology limitations:

  • Text-only, English-only — no vision, audio, or multilingual evaluation
  • Weighting (25% agents, 25% coding, 25% general, 25% scientific) represents a deliberate design bet, not a universal capability definition
  • GDPval-AA’s Elo-based scoring (16.7% of the total index) introduces human-preference variance similar to Arena-style evaluations
  • Quantization methods used by providers affect both speed and quality but are not fully disclosed

2. Intelligence Index: Key Findings

2.1 The top tier is a three-way statistical tie

As of April–May 2026, the Intelligence Index top cluster:

Model Score Cost per Index Point (est.)
GPT-5.5 (xhigh) 60 [NEEDS VERIFICATION]
GPT-5.5 (high) 59 [NEEDS VERIFICATION]
Claude Opus 4.7 (Adaptive Reasoning, Max) 57 $77.31
Gemini 3.1 Pro Preview 57 $15.65
GPT-5.4 (xhigh) 57 [NEEDS VERIFICATION]
GPT-5.3 Codex 54 [NEEDS VERIFICATION]
Claude Opus 4.6 53 $27.38
GLM-5.1 (Zhipu AI) 51 [NEEDS VERIFICATION]
GLM-5 / MiniMax-M2.7 50 [NEEDS VERIFICATION]

Source: Artificial Analysis Intelligence Index leaderboard, April 2026; cost-per-point estimates from SmartChunks analysis of Artificial Analysis data (artificialanalysis.ai).

The ±1% confidence band on the composite score means all models in the 57–60 range are statistically tied. The practical implication: a procurement decision based on composite rank differences of 1–3 points has no statistical foundation.

2.2 Component scores reveal actual differentiation

Where the composite obscures, the component scores discriminate. On GDPval-AA (agentic real-world work, 16.7% of overall index):

  • Claude Opus 4.7: 1,753 ELO
  • Gemini 3.1 Pro Preview: 1,314 ELO

This translates to approximately a 92% expected win rate for Opus 4.7 in head-to-head professional task evaluations — a gap that the composite completely hides. For organizations with high-volume agentic workloads, this component difference is the operative number.

Gemini 3.1 Pro leads on scientific reasoning — GPQA Diamond at 94.1% and HLE at 44.7% — making it the dominant choice for research, engineering, and scientific application contexts. Organizations choosing a “top model” based on the composite without examining component scores may be optimizing for a dimension irrelevant to their workload.

2.3 Speed leaders (as of May 2026)

Model Output Speed Notes
Mercury 2 755.8 t/s Diffusion-based architecture from Inception Labs; quality benchmarks (GPQA 73.6, LiveCodeBench 67.3) align with Claude 4.5 Haiku / GPT-5.2 Mini tier
Granite 4.0 H Small 371.9 t/s IBM small model; speed-first design
Granite 3.3 8B 336.5 t/s Edge-optimized
Qwen3.5 2B 326.0 t/s Sub-3B class

For fastest time to first token: Qwen3.5 2B (Non-reasoning) at 0.38s; Qwen3.5 0.8B at 0.43s. These sub-second TTFT figures are relevant to interactive and streaming applications where latency perception matters.

For intelligence-competitive models: Groq delivers 500+ t/s for open-weight models. Gemini Flash-class leads TTFT among frontier models on Artificial Analysis live testing as of the May 2026 snapshot; exact figures shift with provider infrastructure changes.

3. The Provider-Level Finding: Same Model, Different Performance

This is Artificial Analysis’s most operationally significant finding for enterprise procurement, and it is underused. The same underlying model produces substantially different performance characteristics depending on the delivery provider.

3.1 Claude 4.5 Haiku across providers (April 2026)

Provider Output Speed (t/s) TTFT (s) Blended Price ($/1M tokens)
Amazon Bedrock 106.5 0.93 $0.82
Google Vertex 92.2 0.59 $0.82
Azure 91.9 1.09 $1.40
Anthropic Direct 91.8 0.80 $0.82

Source: Artificial Analysis provider comparison, Claude 4.5 Haiku, artificialanalysis.ai/models/claude-4-5-haiku/providers

Key observations:

  • Speed: Amazon delivers 16% higher throughput than the three other providers, which cluster within 1.4 t/s of each other
  • Latency: Google Vertex is 37% faster to first token than Amazon, despite Amazon’s throughput lead — the two metrics are not correlated
  • Price: Azure charges 71% more per million tokens than the three lowest-priced options ($1.40 vs. $0.82) for mid-tier performance on both dimensions
  • The optimization decision is workload-dependent: batch/asynchronous workloads benefit from Amazon’s throughput lead; interactive/streaming workloads benefit from Google Vertex’s TTFT lead; budget-sensitive workloads should avoid Azure unless enterprise agreements offset the list-price premium

3.2 Claude Opus 4.7 provider variance

Provider Output Speed (t/s)
Amazon Bedrock 80.2
Anthropic Direct 58.2
Google 56.9

Source: Artificial Analysis, Claude Opus 4.7 provider comparison, artificialanalysis.ai/models/claude-opus-4-7/providers

Amazon Bedrock delivers 38% higher throughput for Opus 4.7 than Anthropic Direct. For high-volume Opus-class workloads, the provider routing decision is as significant as the model selection decision.

4.1 The 18-month price collapse (2024–mid-2026)

Inference pricing for non-reasoning models declined approximately 75% between early 2024 and early 2026, with the steepest drops concentrated after January 2024 (Epoch AI analysis, March 2025). Epoch AI documents decline rates of 9× to 900× annually across different capability tiers, with PhD-level science task performance (GPQA Diamond class) achieving 40× annual price reduction — the fastest-declining tier.

Reference points:

  • GPT-4 equivalent performance: ~$20/million tokens (late 2022) → ~$0.40/million tokens (2025)
  • Frontier general models: $7–30/M input tokens (November 2023) → sub-$5/M (late 2025)
  • Budget tier floor: $0.01/M blended (Qwen3.5 0.8B, May 2026)

4.2 The reasoning model exception

Frontier reasoning models — o3-class, Claude Opus adaptive reasoning, Gemini reasoning variants — have not followed the commodity price trajectory. These models generate thousands of internal reasoning tokens before producing output, which structurally inflates token counts and total cost. Organizations that migrated to reasoning models for quality gains while expecting continued price compression should audit their token budgets: the inference economics are fundamentally different from the instruct-model market.

The current market structure is two-tier:

  • Commodity instruct tier: Intense price competition; $0.01–$2/M blended pricing; largely commoditized on quality for standard tasks
  • Reasoning/frontier tier: Price stability or modest decline; $5–$30/M input; where quality differences between providers still discriminate

4.3 The price-performance frontier

On the Artificial Analysis leaderboard, the price-performance frontier identifies which models deliver the highest Intelligence Index score per dollar. Open-weight models dominate the low- and mid-cost frontier; closed-weight frontier models define the highest-quality end. (Structural observation from Epoch AI arXiv:2511.23455v2; specific Pareto frontier positions shift with each leaderboard update.)

For enterprise procurement, the actionable frame is: models within the same Intelligence Index tier (e.g., 50–54 range) can differ by 3–5× in cost per index point. Selecting within a quality tier based on cost-efficiency rather than raw score is the defensible procurement posture.

5. Benchmark Landscape: Comparison Context

5.1 LMSYS Chatbot Arena (now LMArena)

Methodology: Crowdsourced blind A/B preference voting. Users submit prompts, receive two anonymous responses, select a winner. Votes aggregate into Bradley-Terry model ratings (Elo-style presentation).

Strengths: Captures real-user preferences on real prompts; continuously updated; not dependent on a fixed test set vulnerable to contamination.

Known weaknesses:

  • A January 2025 paper (OpenReview) demonstrated vote manipulation is feasible — coordinated voting can artificially shift model rankings
  • Response length correlates with winning independent of quality; verbose models receive a systematic advantage
  • Large labs selectively submit best-performing model variants, creating disclosure asymmetry
  • “Overall” leaderboard reflects average human preference across all prompt types, which may not match enterprise use cases

What to trust: Specialized sub-leaderboards (Coding, Math, Hard Prompts) are more resistant to gaming because prompt distribution is narrower and graders have domain knowledge. The Overall ranking should be treated as directional, not definitive.

Current top cluster (April 2026): Anthropic leads Arena rankings as of March 2026 with Claude Opus 4.6/4.7, trailed closely by xAI (Grok), Google (Gemini 3.x), and OpenAI (GPT-5 family). The performance gap between top models has compressed to 2.7% per Stanford HAI AI Index 2026.

5.2 Stanford HELM (Holistic Evaluation of Language Models)

Methodology: Multi-dimensional evaluation across 42 real-world scenarios measuring accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency simultaneously. Open-source Python framework (stanford-crfm/helm), fully reproducible, with public release of all prompts and completions. HELM evaluates against standardized datasets including MMLU-Pro, GPQA, and IFEval.

Current status: GPT-4.1 leads HELM Long Context and Capabilities leaderboards as of late 2025 [NEEDS VERIFICATION — exact date of last update unclear]. HELM has expanded into MedHELM (medical), VHELM (vision-language), HEIM (text-to-image), and a Long Context leaderboard (September 2025).

Critical update: HELM enters maintenance mode June 1, 2026. New scenario additions and regular model updates will cease. Organizations building internal evaluation pipelines that reference HELM as a living benchmark need to account for this transition.

Enterprise value: HELM’s multi-metric approach (not just accuracy) is the closest public analog to a vendor-neutral enterprise evaluation. Its fairness and toxicity dimensions are specifically relevant for consumer-facing AI applications in regulated industries.

5.3 Hugging Face Open LLM Leaderboard (v2/v3)

Methodology: Automated evaluation of open-weight models against six benchmarks selected for contamination resistance: IFEval, BBH, MATH Lvl 5, GPQA, MUSR, and MMLU-Pro. V2 launched late 2024, replacing six saturated original benchmarks (including original MMLU and HellaSwag) after top models approached 90%+ on all of them through memorization rather than capability.

Current top performers: Following the v2 methodology shift, Chinese research labs — Zhipu AI (Z.AI), Alibaba, Moonshot AI, and DeepSeek — dominate top positions across multiple independent leaderboards. Rankings reshuffled completely from the v1 era.

Anti-contamination controls: Hugging Face implemented deduplication checks and held-out data partitions in v2/v3; the April 2026 v3 update specifically focused on anti-contamination controls that produced further ranking reshuffles.

Enterprise relevance: Primarily useful for selecting open-weight/self-hosted models. The leaderboard evaluates publicly downloadable weights — not API endpoints — so it does not capture provider-level performance variance documented by Artificial Analysis.

5.4 BIG-Bench Hard: saturation declared

BIG-Bench Hard reached approximately 94% state-of-the-art performance across frontier models by April 2026 and is no longer discriminating. All frontier models score 92–94%, within noise. The benchmark has been effectively retired from frontier model evaluation.

The successor, BIG-Bench Extra Hard (BBEH), was published in February 2026 (arXiv:2502.19187), designed with verifiable answers, lower random-chance baselines, and multi-hop reasoning requirements that current frontier models cannot saturate. BBEH scores are beginning to appear in benchmark suites but have not yet achieved the citation density of its predecessor.

6. Key Data Points Table

Metric Finding Source Date Tier
Intelligence Index top score (May 2026) GPT-5.5 (xhigh): 60; Claude Opus 4.7: 57; Gemini 3.1 Pro: 57 Artificial Analysis, April 2026 Apr 2026 TIER 1
Cost per index point at score 57 Gemini: $15.65; Claude Opus 4.6: $27.38; Claude Opus 4.7: $77.31 SmartChunks analysis of AA data Apr 2026 TIER 1
Provider speed variance, same model Amazon Bedrock: 106.5 t/s vs. Anthropic Direct: 91.8 t/s (Claude 4.5 Haiku) Artificial Analysis provider comparison May 2026 TIER 1
Provider TTFT variance, same model Google Vertex: 0.59s vs. Azure: 1.09s (Claude 4.5 Haiku) — 85% difference Artificial Analysis provider comparison May 2026 TIER 1
Provider price variance, same model Azure: $1.40/M vs. Amazon/Google/Anthropic: $0.82/M (Claude 4.5 Haiku) — 71% premium Artificial Analysis provider comparison May 2026 TIER 1
Fastest model (May 2026) Mercury 2: 755.8 t/s Artificial Analysis leaderboard May 2026 TIER 1
Fastest TTFT (May 2026) Qwen3.5 2B: 0.38s Artificial Analysis leaderboard May 2026 TIER 1
Cheapest model (May 2026) Qwen3.5 0.8B: $0.01/M blended Artificial Analysis leaderboard May 2026 TIER 1
Token price decline, non-reasoning (2024–2026) ~75% in ~18 months Ramp Velocity / Epoch AI 2026 TIER 1
GPQA capability price decline rate 40× annually (fastest-declining tier) Epoch AI, March 2025 Mar 2025 TIER 2
BIG-Bench Hard saturation level 94% state-of-the-art; 92–94% across all frontier models BenchLM.ai / benchmarkingagents.com 2026 TIER 1
Arena top model performance gap (March 2026) 2.7% separates top-ranked from fourth-ranked model Stanford HAI AI Index 2026 Apr 2026 TIER 1
HELM status Maintenance mode from June 1, 2026 Stanford CRFM announcement May 2026 TIER 1
Artificial Analysis testing cadence ~56–64 API calls per model per day Artificial Analysis methodology documentation May 2026 TIER 1
Models tracked (AA, May 2026) 519 Artificial Analysis leaderboard May 2026 TIER 1

7. What This Means for Your Organization

7.1 Vendor selection: read the component scores, not the composite

When evaluating frontier models, the Intelligence Index composite is a starting screen, not a selection criterion. Two organizations running the same “57-point model” may have chosen wrong for their workload:

  • Agentic workflows (autonomous task execution, multi-step reasoning, tool use): Claude Opus 4.7’s GDPval-AA lead (1,753 vs. 1,314 ELO) translates to a 92% head-to-head advantage on professional tasks. This is the operative metric.
  • Scientific, engineering, or research applications: Gemini 3.1 Pro’s 94.1% GPQA Diamond score makes it the choice for graduate-level reasoning tasks, at roughly one-fifth the cost per composite point.
  • High-volume general workloads: Models in the 50–54 range (GPT-5.3, Claude Opus 4.6, Qwen-class models) deliver near-frontier composite scores at a fraction of frontier pricing. Fit the model tier to the workload tier.

The practical step: before any model contract, run your three most common task types through the Artificial Analysis component leaderboards, not the overall ranking.

7.2 Provider selection: run the Artificial Analysis provider comparison before signing

For any model under consideration, pull the Artificial Analysis provider comparison page (artificialanalysis.ai/models/[model-name]/providers) before finalizing your deployment architecture. The data is live and frequently updated.

The decision matrix:

  • Batch/asynchronous workloads (document processing, overnight runs): optimize for output tokens/second — Amazon Bedrock leads on most models tested
  • Interactive/real-time applications (chatbots, copilots, voice): optimize for TTFT — Google Vertex leads on most models tested
  • Cost-sensitive at scale: Three providers typically offer near-identical pricing; identify the outlier (often Azure) and quantify the premium against any enterprise agreement offsets before committing

Do not assume the model developer’s direct API is either the fastest or the cheapest. For Claude 4.5 Haiku, it is neither.

7.3 Cost structure: build reasoning-token budgets separately

Non-reasoning model costs are under genuine deflationary pressure — 75% decline in 18 months is not a temporary phenomenon, it reflects structural improvements in hardware efficiency and model compression. Plan for continued decline in this tier.

Reasoning model costs are not on the same curve. If your organization has migrated to o3-class or Claude Opus Adaptive Reasoning for quality, your token budgets should not be extrapolated from non-reasoning model trends. Audit actual token consumption monthly; reasoning models generate internal tokens that do not appear in the visible output but are billed at full rate.

7.4 SLA setting: use the Artificial Analysis TTFT and throughput percentiles, not the provider’s published SLA

Published SLAs typically guarantee uptime (99.9% monthly = 8.7 hours downtime/year) but do not specify latency under load. Artificial Analysis’s parallel workload tests (10 concurrent requests, daily) provide the closest public proxy to real-world performance under concurrent usage. Pull P95 latency figures from those tests for any application with latency requirements before negotiating SLAs — the provider’s standard contract likely does not include a latency commitment.

A common practitioner target for customer-facing AI applications is 99.9% monthly uptime with P95 latency below 2,000ms for standard-context requests — though specific SLA terms vary by contract and workload class.

7.5 Benchmark hygiene: what to stop using and what to start tracking

Stop using for vendor discrimination: MMLU, GSM8K, BIG-Bench Hard. All frontier models have saturated these. A vendor quoting 90%+ on any of these three is citing a number that all of their competitors also hit.

Start tracking: HLE (Humanity’s Last Exam), GPQA Diamond, SWE-bench Verified, and LiveCodeBench for frontier quality. Artificial Analysis provider comparison pages for inference economics. LMSYS Coding/Math specialized leaderboards (not Overall) for preference signals.

Internal benchmark investment: The most defensible vendor evaluation practice for Fortune 500 procurement remains a four-part internal benchmark: domain task accuracy on your data, cost per correct answer, latency at the 95th percentile, and failure-mode distribution. No public benchmark — including the Intelligence Index — substitutes for this.

Sources

Brandon Sneider | brandon@brandonsneider.com May 2026