← Benchmarks 🕐 30 min read
Benchmarks

VLM, Computer Use, and Web Agent Benchmarks — 2026

Brandon Sneider · May 2026

> **Source credibility: VARIES BY SECTION.** Covers academic benchmarks (TIER 1 — arXiv + peer review), vendor-published benchmarks (TIER 2), and practitioner signals (TIER 3).

See also (wiki): wiki/agentic-ai-governance.md · wiki/assistive-to-agentic-shift.md · wiki/workflow-redesign.md · wiki/physical-ai-capability-levels.md

Source credibility: VARIES BY SECTION. Covers academic benchmarks (TIER 1 — arXiv + peer review), vendor-published benchmarks (TIER 2), and practitioner signals (TIER 3). OSWorld/WebArena/WorkArena scores sourced from published papers and official leaderboards. MMMU-Pro and CharXiv scores from Artificial Analysis, BenchLM, and llm-stats.com trackers (TIER 2 — methodology varies). Each section carries inline credibility notes. [NEEDS VERIFICATION] flags assertions that could not be cross-checked against a primary source during this research pass.

Executive Summary

  • VLM benchmarks for document and chart understanding are approaching saturation fast. OmniDocBench v1.5 now has two models above 94% accuracy (GLM-OCR, PaddleOCR-VL-1.5). CharXiv — chart understanding from scientific papers — is led by Claude Mythos Preview at 93.2% as of May 2026. For enterprise document processing procurement, these benchmarks are no longer discriminating at the frontier; the eval question shifts to throughput, cost per page, and formula/table fidelity in enterprise-specific formats.
  • Computer use benchmarks show rapid trajectory improvement but a hard efficiency gap. OSWorld SOTA has risen from 19.5% (GPT-4V, 2024) to ~75% (GPT-5.4, verified, March 2026) in roughly 18 months. The remaining gap is not random noise — it concentrates in tasks requiring non-standard UI interaction (custom enterprise software, dynamic content, popup dialogs, resolution variation). Even the best agents take 2.7–4.3× more steps than humans on the same task (OSWorld-Human, arXiv:2506.16042).
  • OSWorld-MCP (ICLR 2026) establishes that structured tool access materially improves computer use. Providing 158 verified MCP tools covering common applications lifted Claude 4 Sonnet from 40.1% to 43.3% (at 50 steps) and OpenAI o3 from 8.3% to 20.4% (at 15 steps). The implication for enterprise deployment: agents with curated, application-specific tool APIs reliably outperform GUI-only agents; API-first architecture is the correct design pattern before GUI fallback.
  • Web agent benchmarks span a wide reliability range. WebArena SOTA (OpAgent, CodeFuse AI) reached 71.6% in January 2026, closing to within 6 points of the 78% human baseline. But VisualWebArena — which adds visual elements and more realistic multimodal tasks — shows best agents at 16.4% vs. human 88.7%. That 72-point gap signals that multimodal web task completion is not yet operationally deployable for unstructured use cases.
  • WorkArena (ServiceNow) quantifies the enterprise gap precisely. GPT-4o reaches 42.7% on atomic WorkArena tasks but essentially 0% on WorkArena++ L3 compositional tasks (vs. 93.9% human), where the agent must plan across multiple browser sessions with retrieval, arithmetic, and memory. No list-filter task was solved by any tested model — a wall created by non-standard HTML enterprise UI patterns not present in training data.
  • The benchmark-to-deployment translation gap is structural, not temporary. UI-CUBE (UiPath, Nov 2025) found that 56.7% of computer-using agent actions miss their intended target across enterprise tasks, and performance degrades significantly across screen resolutions (1024×768 vs. 4K). Benchmarks run at normalized settings; production enterprise desktops do not. Procurement teams should require vendor evals at the specific resolution profile, application stack, and UI vintage of their deployment environment — not published leaderboard scores.

1. VLM / Vision-Language Benchmarks

1.1 MMBench / MMBench-v2 (OpenCompass)

Publisher: OpenCompass Community (Shanghai AI Laboratory + collaborators), 2023, continuously updated.

What it measures: MMBench is a structured capability inventory for vision-language models, organized into approximately 20 fine-grained ability dimensions covering perception (object/attribute/relation recognition), reasoning (commonsense, arithmetic), and OCR. It uses a choice-format evaluation — each question has a fixed set of options — to enable automated, deterministic scoring. The benchmark avoids free-text generation to eliminate judge variability.

Current SOTA: Frontier models (GPT-5.x, Claude Opus 4.x, Gemini 3.x series) now cluster in the 88–93% range on the core MMBench English test. The newer Creation-MMBench variant (released March 2025) tests creative VLM capabilities and is not yet saturated. [NEEDS VERIFICATION — specific frontier score distribution on Creation-MMBench not confirmed against primary source]

Enterprise relevance: MMBench’s structured capability breakdown is useful for identifying specific VLM weaknesses in a target application. A model scoring 91% overall may still score 60% on fine-grained spatial relation tasks — which matters for document layout parsing. Use MMBench’s dimensional breakdown in vendor evaluation, not its aggregate score.

Credibility: TIER 1 — published arXiv 2307.06281; open evaluation infrastructure; scores independently reproducible.

1.2 MMMU / MMMU-Pro

Publisher: MMMU-Benchmark consortium (CMU, University of Michigan, et al.), 2023 (MMMU) / 2024 (MMMU-Pro). MMMU-Pro arXiv:2409.02813.

What it measures: MMMU (Massive Multitask Multimodal Understanding) tests expert-level VLM knowledge across 30 disciplines (engineering, medicine, law, business, and others), using college-exam-style questions with images. Questions require integrating visual content with domain knowledge. MMMU-Pro toughens the original by (a) filtering out questions answerable from text alone, (b) expanding from 4-choice to 10-choice options, and © adding a vision-only input setting where the answer options are also rendered as images, eliminating text parsing shortcuts.

Saturation dynamics: Base MMMU has followed the saturation trajectory seen in MMLU: GPT-4V launched at ~56%; frontier models in 2025–26 cluster at 70–78%. MMMU-Pro successfully reset the difficulty floor — initial models scored 16.8–26.9% when it launched; as of May 2026, GPT-5.4 Pro leads leaderboard trackers at ~94% and Gemini 3.1 Pro at ~84%, with some trackers showing different methodology-dependent figures (source: BenchLM, Artificial Analysis, May 2026 — TIER 2). Saturation risk on MMMU-Pro is accelerating faster than its designers expected.

Credibility note on scores: Independent trackers report varying figures for MMMU-Pro top scores depending on evaluation methodology and which model variants they test. The ~94% figure for GPT-5.4 Pro should be treated as approximate until confirmed against the official leaderboard at mmmu-benchmark.github.io.

Enterprise relevance: MMMU-Pro is the most defensible VLM knowledge benchmark for multidisciplinary enterprise applications (compliance, insurance, healthcare). The vision-only setting specifically tests whether models can read and reason from scanned documents without relying on OCR pre-processing — relevant for document-heavy workflows.

1.3 MathVista / MathVerse

Publisher: MathVista — Pan et al., NeurIPS 2023 (arXiv:2310.02255). MathVerse — Zhang et al., 2024 (arXiv:2403.14624).

What it measures:

MathVista combines 6,141 examples from 28 existing multimodal datasets plus three new datasets (IQTest, FunctionQA, PaperQA), covering geometric reasoning, statistical interpretation, figure reading, and scientific diagram analysis. Critically, human performance is only 60.3% — substantially below the usual 90%+ human bar — because the problems are drawn from actual exams and research papers.

MathVerse focuses specifically on visual math diagrams: plane geometry, solid geometry, and function graphs. Each of its 2,612 problems is presented in six variations ranging from diagram-only (no text description) to text-only, enabling precise measurement of how much a model relies on the diagram vs. the text description. This makes it a diagnostic instrument rather than just a scoring benchmark.

Scores (MathVista, May 2026): Frontier models now substantially exceed the 60.3% human baseline. GPT-5 and Claude Opus 4.x series are reported above 85% on leaderboard trackers (TIER 2 — llm-stats.com). The benchmark retains some discrimination at the frontier but is approaching saturation for math-heavy VLM research tasks.

Enterprise relevance: Financial analysis, engineering documentation review, and scientific report processing all require the diagram-plus-reasoning capability MathVista and MathVerse test. A model that scores well on MathVista but poorly on the diagram-only MathVerse variants may be relying on text extraction rather than visual understanding — important for procurement teams evaluating models on mixed PDF/image document workflows.

1.4 OCRBench / OCRBench v2

Publisher: OCRBench — Huawei Noah’s Ark Lab, 2023. OCRBench v2 — expanded team, arXiv:2501.00321, January 2025; accepted at peer review (OpenReview forum:Vb6i3Dp24N).

What it measures: OCRBench v2 is a large-scale bilingual benchmark with 10,000 human-verified QA pairs across 31 document scenarios including street signs, receipts, academic paper formulas, product labels, infographics, and handwritten text. It covers four times more task types than v1. A private test set of 1,500 manually annotated images provides contamination-resistant ground truth.

Key findings from v2: After systematic evaluation of state-of-the-art models, most VLMs scored below 50/100 on the benchmark, with five recurring failure modes: (1) infrequently encountered text types (chemical formulas, handwriting); (2) fine-grained visual perception of dense or small text; (3) layout understanding across multi-column or mixed-orientation documents; (4) parsing complex elements such as tables and embedded equations; (5) logical reasoning from document context. Frontier models have improved substantially since v2’s publication — the NVIDIA Llama Nemotron Nano VLM reached the top of the OCRBench ranking as of late 2025 — but sub-50 performance from early 2025 tested models signals the benchmark still discriminates.

Enterprise relevance: OCRBench v2 is the most direct proxy for enterprise document processing pipeline quality: invoices, contracts, technical manuals, financial statements, and regulatory filings all require the capabilities the benchmark tests. The five failure modes map directly to enterprise document pain points. When evaluating VLMs for intelligent document processing, OCRBench v2 scores at the sub-category level (formula parsing, table extraction, layout) are more informative than the aggregate score.

Credibility: TIER 1 — peer-reviewed (arXiv:2501.00321; OpenReview confirmed).

1.5 OmniDocBench

Publisher: OpenDataLab (Shanghai AI Laboratory), published CVPR 2025 (arXiv:2412.07626).

What it measures: OmniDocBench tests end-to-end PDF document parsing — converting a PDF into structured Markdown while preserving text, tables, formulas, reading order, and layout. The benchmark spans nine document source types (academic papers, textbooks, government filings, handwritten notes, and others). Evaluation uses Normalized Edit Distance for text fidelity, Tree-Edit-Distance-based similarity (TEDS) for tables, and Character Detection Matching (CDM) for mathematical formulas. This makes it the first benchmark to score the complete parsing pipeline rather than any single sub-task.

Saturation and update cycle: OmniDocBench reached near-saturation faster than expected. By early 2026, GLM-OCR and PaddleOCR-VL-1.5 both surpassed 94% accuracy on v1.5. LlamaIndex explicitly flagged saturation and partnered with Kaggle to launch a successor document OCR leaderboard for agents. The benchmark maintainers have responded with an ongoing versioning cycle (v1.6 added Multi-Granularity Adaptive Matching to reduce scoring bias; v1.7 added the Qianfan-OCR track). Scores above 94% on v1.5 should be treated as benchmark-saturated pending private eval confirmation.

Enterprise relevance: OmniDocBench is the most operationally grounded document parsing benchmark available. For enterprises selecting intelligent document processing pipelines (ERP invoice ingestion, contract lifecycle management, regulatory filing extraction), OmniDocBench sub-scores on tables and formulas should be part of vendor evaluation rubrics. The saturation of v1.5 at ~94% does not mean the problem is solved — it means the standard test set is no longer discriminating; the unsolved hard cases (handwritten annotations, mixed-language filings, complex chemical/mathematical notation) are underrepresented in v1.5.

Credibility: TIER 1 — CVPR 2025 accepted; open dataset and evaluation code (github.com/opendatalab/OmniDocBench).

1.6 CharXiv

Publisher: Princeton NLP Group (arXiv:2406.18521), NeurIPS 2024.

What it measures: CharXiv evaluates chart understanding using 2,323 natural, challenging charts extracted from actual scientific papers across mathematics, physics, computer science, economics, and biology. It separates descriptive questions (reading specific values from charts) from reasoning questions (interpreting trends, comparing across subplots, identifying anomalies). The reasoning track is significantly harder and more enterprise-relevant than the descriptive track.

Current scores (May 2026): CharXiv was adopted into official evaluation suites by GPT-4.1, Qwen2.5-VL, InternVL2.5, and Doubao 1.5 Pro at launch in April 2025, signaling its status as a standard multimodal evaluation instrument. Current leaderboard trackers report Claude Mythos Preview at 93.2%, Claude Opus 4.7 (Adaptive) at 91%, and Muse Spark at 86.4%. Models augmented with code execution or Python tools have crossed 85% on the reasoning track (source: BenchLM, May 2026 — TIER 2).

Enterprise relevance: CharXiv is the most direct available proxy for BI and analytics dashboard interpretation — the capability required for automated financial reporting analysis, operational dashboard monitoring, and research intelligence. The reasoning track specifically tests whether a model can identify the right conclusion from a chart, not just extract numbers. For enterprises deploying VLMs in analyst workflows, CharXiv reasoning scores should be a required evaluation criterion.

1.7 VHELM (Stanford CRFM)

Publisher: Stanford Center for Research on Foundation Models (arXiv:2410.07112), October 2024.

What it measures: VHELM (Holistic Evaluation of Vision Language Models) extends the HELM framework to VLMs, aggregating 21 datasets across nine evaluation dimensions: visual perception, knowledge, reasoning, bias, fairness, multilinguality, robustness, safety, and toxicity. The holistic design mirrors the original HELM philosophy: no single metric; an aggregate profile across multiple dimensions including non-performance attributes.

Current status — maintenance mode: HELM (including VHELM) enters maintenance mode June 1, 2026. The Stanford CRFM team has announced the framework will no longer receive active updates after that date. This is significant: VHELM’s multi-dimensional profile approach was its key differentiator from leaderboard-style benchmarks, but the maintenance announcement means its coverage of the most recent frontier models will not be updated. For enterprises requiring a holistic safety and fairness profile alongside capability scores, VHELM remains the most structurally rigorous framework available through its last update; for ongoing model comparison, alternative frameworks will be required after June 2026.

Enterprise relevance: VHELM’s safety, bias, and robustness dimensions are directly relevant to AI governance requirements (EU AI Act, sector-specific regulations). Enterprises in regulated industries (healthcare, finance, public sector) should run VHELM evaluations on VLMs before deployment even if the scores on capability dimensions are already known from other sources.

Credibility: TIER 1 — Stanford CRFM, peer-reviewed, reproducible.

2. Computer Use / Desktop Agent Benchmarks

2.1 OSWorld

Publisher: University of Hong Kong / multiple academic contributors (NeurIPS 2024, arXiv:2404.07972).

What it measures: OSWorld provides a unified computer environment supporting Ubuntu, Windows, and macOS, with 369 tasks spanning browsers, office suites, code editors, and system utilities. Tasks are fully programmatic and verifiable — success is checked by executable scripts, not human judgment. The benchmark tests the complete computer use loop: perceiving screen state, planning action sequences, issuing keyboard/mouse commands, and verifying task completion.

Score trajectory (enterprise-relevant context):

Time Model OSWorld Score
Early 2024 GPT-4V (initial) ~19.5%
Mid 2024 Claude 3.5 Sonnet ~39%
Sept 2025 Claude Sonnet 4.5 61.4%
Feb 2026 Claude Sonnet 4.6 ~72.5–73%
Mar 2026 GPT-5.4 (verified) ~75%
Current Coasty (specialized system) ~82%

Sources: Published papers, OSWorld leaderboard trackers (Coasty blog, BenchmarkingAgents.com), May 2026 — TIER 2/3 for most recent scores.

Human baseline on OSWorld is approximately 72–84% depending on task category, meaning leading general-purpose agents are at or near the human baseline in aggregate. However, OSWorld-Human (arXiv:2506.16042) measured step efficiency and found that even the best agents take 2.7–4.3× more steps than humans — reaching the same result through trial-and-error rather than intentional, directed action. This step inefficiency translates directly to cost and latency in production.

Remaining gap composition: The 15–30% task failure rate on OSWorld clusters in non-standard UI patterns, applications with unusual interaction paradigms (drag-and-drop, hover-activated menus), tasks requiring cross-application data transfer, and tasks where state recovery after an error requires judgment about the environment’s current state.

2.2 OSWorld-MCP

Publisher: Alibaba Group / X-PLUG team (ICLR 2026, arXiv:2510.24563).

What it measures: OSWorld-MCP extends the base OSWorld framework by providing agents with 158 high-quality verified MCP tools covering seven common applications (LibreOffice Writer, Calc, Impress, VS Code, Chrome, VLC, and OS utilities). The benchmark tests 250 tasks where available MCP tools can substantially improve efficiency. Agents may choose between GUI operations and MCP tool invocations at each step. Two new metrics supplement accuracy: Tool Invocation Rate (TIR) and Average Completion Steps (ACS).

Key findings: MCP tool access consistently improves both accuracy and efficiency. OpenAI o3 improved from 8.3% to 20.4% (at 15 steps); Claude 4 Sonnet improved from 40.1% to 43.3% (at 50 steps). The improvement is larger for models and step budgets where GUI grounding is the primary constraint — structured tool invocation bypasses the visual localization step entirely.

Enterprise implication: OSWorld-MCP is the benchmark that most directly validates the enterprise agentic architecture pattern: structured API/tool access first, GUI fallback only when no API exists. Procurement teams evaluating computer use deployments should benchmark candidate agents on both OSWorld and OSWorld-MCP, because the gap between the two scores quantifies how much a given agent benefits from application-native API access — a deployment design choice the enterprise controls.

Credibility: TIER 1 — ICLR 2026 accepted; open leaderboard (osworld-mcp.github.io).

2.3 WindowsAgentArena

Publisher: Microsoft Research (arXiv:2409.08264, September 2024).

What it measures: WindowsAgentArena (WAA) is a scalable benchmark running agents in real Azure-hosted Windows environments with 154 diverse tasks across representative domains (file management, Office applications, web browser, terminal, settings). The benchmark can be fully parallelized in Azure, enabling a complete evaluation run in approximately 20 minutes. WAA-V2 (May 2025) introduced stricter task curation and automated validation protocols.

Scores: The baseline Navi agent scored 19.5% in the original paper (2024) vs. 74.5% human performance. PC Agent-E scored 36% on WAA-V2 (May 2025). More recent models have not published systematic WAA results; performance on general OSWorld serves as a reasonable proxy for the Windows-specific benchmark.

Enterprise relevance: WAA is the only major benchmark running exclusively on Windows, making it the most directly relevant to enterprises where the desktop fleet is Windows-dominated (the majority of Fortune 500 organizations). The Azure-hosted infrastructure design means enterprises can in principle run their own WAA-style evaluations at scale with minimal setup — a design feature that generic academic benchmarks rarely offer.

Credibility: TIER 1/2 — arXiv:2409.08264; Microsoft Research published; open GitHub repo (microsoft/WindowsAgentArena).

2.4 ScreenSpot / ScreenSpot-Pro

Publisher: ScreenSpot — NUS / multiple, 2024. ScreenSpot-Pro — published ACM Multimedia 2025 (arXiv:2504.07981).

What it measures: Both benchmarks test GUI grounding — the ability to locate a specific UI element in a screenshot given a natural language instruction (“click the download button in the top toolbar”). This is the visual localization sub-task that underlies all computer use: before an agent can interact with an application, it must correctly identify where on the screen to click or type.

ScreenSpot-Pro targets professional, high-resolution applications: 1,581 instructions across 23 applications in 5 industries (engineering CAD, financial terminals, medical imaging software, legal document review platforms, and others) on 3 operating systems. Instructions require interpreting densely populated professional UIs with non-standard element labeling.

Scores (ScreenSpot-Pro): GPT-5.2 leads at 86.3%; Muse Spark at 84.1%; Gemini 3 Pro at 72.7%. Specialist GUI models: GUI-CURSOR (Qwen2.5-VL-7B based) at 56.5%; UI-TARS at 47.5%; ZonUI-3B at 28.7% (source: llm-stats.com, May 2026 — TIER 2). [NEEDS VERIFICATION — specific scores for GPT-5.2 and Muse Spark not confirmed against primary ScreenSpot-Pro leaderboard]

Enterprise relevance: ScreenSpot-Pro is the most enterprise-relevant GUI grounding benchmark available. It specifically tests the resolution and application density profiles that enterprise deployments encounter. A model that grounds well on consumer web applications but fails on professional CAD or financial terminal UI is not deployable for the relevant enterprise use case, regardless of its OSWorld aggregate score.

2.5 AssistGUI

Publisher: arXiv:2312.13108, December 2023 / updated 2024.

What it measures: AssistGUI is a 100-task benchmark covering nine professional Windows applications including After Effects, Photoshop, and MS Word. Tasks require mouse and keyboard manipulation to complete multi-step professional workflows. The benchmark is notable for testing applications with complex, non-standardized UI patterns that differ substantially from consumer web UX.

Limitation: AssistGUI was published at an era when base capabilities were substantially lower. Its task set has not received the same ongoing curation as OSWorld or WindowsAgentArena. It remains a useful diagnostic for professional creative software use cases but should not be used as a primary procurement benchmark.

2.6 AndroidWorld / AndroidBench

Publisher: AndroidWorld — Google DeepMind (ICLR 2025, arXiv from 2024). AndroidBench — separate research effort.

What it measures: AndroidWorld is a dynamic benchmark with 116 programmatic tasks across 20 real-world Android apps (Gmail, Google Maps, Contacts, Calendar, and others). Tasks are non-deterministic — the environment state varies — requiring agents to handle application-state variability. The benchmark accepts multimodal input (screenshots, accessibility trees, and natural language instructions).

Scores: Initial SOTA (M3A agent at launch) was 30.6% vs. 80% human. Rapid progression followed: Claude 3.7 Sonnet + reasoning reached 64.7%; MobileUse (reflection-based) reached 62.9%. Recent leaderboard trackers show Qwen3.5-35B-A3B at 71.1%, with some specialized systems above 90% (source: llm-stats.com, minitap.ai — TIER 2/3). [NEEDS VERIFICATION — 91.4% claim from DroidRun AI could not be confirmed against peer-reviewed benchmark run]

Enterprise relevance: Mobile computer use is relevant for enterprises deploying AI agents to interact with mobile-only or mobile-primary enterprise applications, field service tools, and BYOD workflows. The dynamic task environment (non-deterministic app state) makes AndroidWorld a more realistic proxy for production mobile automation than static-screenshot benchmarks.

2.7 UI-CUBE (Enterprise Reliability Benchmark)

Publisher: UiPath Research (arXiv:2511.17131, November 2025).

What it measures: UI-CUBE evaluates computer use agents specifically for enterprise operational reliability, not just task completion accuracy. The benchmark includes 226 tasks across two difficulty tiers: simple UI interactions (136 tasks) and complex enterprise workflows including copy-paste operations (50 tasks) and multi-application scenarios (40 tasks). All tasks are evaluated across three enforced display resolutions: 1024×768, Full HD (1920×1080), and 4K (3840×2160).

Key findings: 56.7% of CUA actions miss their intended target across the full benchmark task set. Performance degrades significantly across resolution tiers — an agent that achieves a given success rate at Full HD may score substantially lower at 4K, because UI element coordinates and sizes change and the agent’s learned grounding patterns do not transfer.

Enterprise implication: This is the benchmark that most directly quantifies the gap between published OSWorld scores and real enterprise deployment performance. Published benchmarks run at normalized settings (typically Full HD); enterprise deployments span resolution environments. UI-CUBE’s findings suggest that published computer use scores should be discounted by a resolution-variability factor, and that vendor evaluations must be conducted at the target deployment resolution profile.

Credibility: TIER 1/2 — arXiv:2511.17131; published by UiPath Research; methodology documented.

3. Browser & Web Agent Benchmarks

3.1 WebArena

Publisher: CMU / multiple contributors (arXiv:2307.13854, NeurIPS 2023 Workshop).

What it measures: WebArena is the foundational web agent benchmark, providing self-hosted instances of five functional web applications: an e-commerce platform (shopping), a customer review site (Reddit-like), a source code management platform (GitLab), an online map service, and a Wikipedia instance. It contains 812 tasks requiring navigation, form interaction, information lookup, and cross-site synthesis. Tasks have programmatic verifiers, enabling reliable automated scoring.

SOTA trajectory and saturation risk:

Year System WebArena Score
2023 GPT-4 + ReAct ~14%
2024 Best systems ~35–45%
Jan 2026 OpAgent (CodeFuse AI) 71.6%

Human baseline is approximately 78%. OpAgent’s 71.6% (arXiv:2602.13559) uses a Planner-Grounder-Reflector-Summarizer multi-agent pipeline with online reinforcement learning. Its per-domain performance is uneven: 86% on Reddit-like tasks, 59.2% on shopping (form-fill and checkout flows).

Saturation timeline: At the current improvement rate, WebArena will likely reach human-level performance (78%) within one to two evaluation cycles. The benchmark’s self-hosted static design means its task distribution is fixed; an agent that optimizes for WebArena may not generalize to the dynamic state of real commercial web applications. Online-Mind2Web (300 tasks, 136 real live websites) was introduced to address this, with most agents scoring well below their WebArena performance.

Credibility: TIER 1 — arXiv:2307.13854; open-source environment and evaluation code.

3.2 VisualWebArena

Publisher: CMU (arXiv:2401.13649, 2024).

What it measures: VisualWebArena extends WebArena with 910 tasks that require multimodal reasoning — agents must interpret product images, visual search results, user-uploaded screenshots, and image-dependent UI elements to complete tasks. The three environments (Classifieds, Shopping, Reddit) surface tasks where text-only approaches systematically fail.

Scores: Best VLM agents score 16.4% vs. human performance of 88.7% — a 72-point gap that makes VisualWebArena the most clearly non-saturated major web agent benchmark. This gap reflects the combined challenge of visual grounding in web contexts (locating correct product or image element), multimodal reasoning (comparing visual options), and multi-step action sequence planning.

Enterprise relevance: The large VisualWebArena gap is operationally significant. Many enterprise web workflows involve visual elements — product catalog management, image-dependent approval workflows, dashboard screenshot-based reporting. VisualWebArena performance is a better proxy for these workflows than text-only WebArena scores. The 16.4% figure should be a stopping criterion for deploying multimodal web agents on unsupervised visual tasks in production.

Credibility: TIER 1 — arXiv:2401.13649; open-source environment.

3.3 WorkArena (ServiceNow)

Publisher: ServiceNow AI Research (arXiv:2403.07718, 2024). WorkArena++ (OpenReview:PCjK8dqrWW) extends to compositional tasks.

What it measures: WorkArena provides 33 atomic knowledge work tasks inside a live ServiceNow instance — a production enterprise SaaS platform. Tasks span ServiceNow’s core modules: service ticket management, asset inventory lookup, catalog item ordering, and list-based information retrieval. WorkArena++ extends to 682 compositional multi-step tasks requiring planning, retrieval, arithmetic reasoning, and memory across browser sessions, with a human performance baseline of 93.9%.

Scores:

Model WorkArena (atomic) WorkArena++ L3
GPT-4o (CoT) 42.7% ~0%
GPT-3.5-Turbo 6.1% —
Llama3-70B-Instruct 17.9% —
Human ~94% 93.9%

List-filter tasks: 0% for every tested model, due to non-standard HTML list widget interaction patterns in ServiceNow that fall outside standard web agent training distributions.

Enterprise relevance: WorkArena is the most operationally grounded enterprise web benchmark available. ServiceNow is deployed at approximately 85% of Fortune 500 organizations. The 0% performance on list-filter tasks — a common operational query pattern — is not a benchmark artifact; it reflects a genuine interaction pattern gap that every enterprise ServiceNow agent deployment will encounter. The WorkArena++ compositional failure is directly analogous to the enterprise reality where knowledge work requires chaining across multiple modules, sessions, and data sources.

Credibility: TIER 1/2 — arXiv:2403.07718; published by ServiceNow Research; open evaluation infrastructure (github.com/ServiceNow/WorkArena).

3.4 WebVoyager / Mind2Web

Publisher: WebVoyager — He et al., 2024 (arXiv:2401.13919). Mind2Web — Ohio State University, 2023.

What it measures: WebVoyager is an end-to-end web agent benchmark using real (non-self-hosted) websites, with tasks synthesized from Mind2Web seeds and augmented via LLM generation. Mind2Web is a dataset of 2,000+ crowd-sourced tasks across 137 real websites and 31 domains.

Known limitations (critical for sourcing decisions): Independent analysis found that WebVoyager has significant construct validity issues: task diversity is limited; a simple Google-Search-first agent can solve up to 51% of tasks without general navigation capability; LLM-as-judge evaluation has low agreement with human judgment. Agents achieving ~90% on WebVoyager typically collapse to 30–40% on Online-Mind2Web (300 tasks on live websites, dynamically changing), which only OpenAI Operator reached at 61%. These benchmarks should be used as historical baselines only; Online-Mind2Web is the more credible successor for general web agent capability assessment.

3.5 BrowseComp

Publisher: OpenAI (arXiv:2504.12516, April 2025). BrowseComp-V³ (visual variant, arXiv:2602.12876) extends to multimodal browsing.

What it measures: BrowseComp contains 1,266 questions requiring persistent, multi-hop web navigation to find hard-to-locate information. Questions are designed to require genuine browsing and synthesis across multiple pages — not answerable by a single search query or from pre-training knowledge. Questions have concise, verifiable answers checked against ground truth.

Scores: Human baseline on BrowseComp is 29.2% (counter-intuitive: humans without time pressure find information that web agents often cannot, but average human web research time budgets are short). Kimi K2.5 (Moonshot AI) scores 74.9% in standard mode and 78.4% with Swarm multi-agent configuration, substantially above GPT-5.2’s 57.8% (source: benchlm.ai, May 2026 — TIER 2).

Enterprise relevance: BrowseComp is the most direct proxy for competitive intelligence, regulatory research, and market analysis workflows where agents must synthesize information from multiple web sources. The large gap between models (74.9% vs. 57.8%) is operationally significant — it directly translates to research task completion rate in deep research agent deployments.

3.6 GAIA

Publisher: Meta AI / HuggingFace / collaborators (arXiv:2311.12983, 2023).

What it measures: GAIA contains 450 real-world multi-step questions requiring reasoning, web browsing, tool use, and multimodal handling to answer. Questions are organized into three difficulty levels. Human performance is 92% vs. GPT-4 with plugins at 15% at launch. The benchmark is maintained on HuggingFace with an active public leaderboard.

Current SOTA (May 2026): Scores vary substantially between bare-model and scaffolded-agent evaluations — a 30-point scaffold premium is documented. Bare model performance: Claude Mythos Preview leads BenchLM tracker at ~52%; GPT-5.4 Pro at ~50%. Scaffolded agent systems: Claude Sonnet 4.5 (HAL Princeton leaderboard) at 74.6%; Manus 1.5 reports 86.5% / 70.1% / 57.7% on L1/L2/L3 respectively; specialized system setups at Steel.dev reach 92%. Human baseline remains 92%.

Enterprise relevance: The scaffold premium on GAIA is the benchmark-visible signal of what enterprise agentic infrastructure contributes beyond raw model capability. A bare model at 52% reaching 74%+ with appropriate tool scaffolding quantifies the expected enterprise agentic architecture ROI. GAIA Level 3 tasks (57.7% for the best reported system vs. 92% human) represent the frontier of multi-step tool-augmented research tasks that Fortune 500 knowledge work requires.

3.7 WebTailBench

Publisher: Microsoft Research (released with Fara-7B tech report, November 2025; V2 updated May 2026).

What it measures: WebTailBench covers 609 real-world web tasks in the “long tail” of realistic consumer and enterprise workflows that existing benchmarks underrepresent: booking movie/event tickets, restaurant reservations, price comparison across retailers, job applications, real estate research, multi-step account management. V2 (May 2026) refreshed expired calendar-bound task dates from V1.

Scores — Fara-7B (November 2025): Fara-7B (Microsoft, 7B parameter on-device model) achieves 38.4 on WebTailBench, outperforming GPT-4o-based SoM agent (30.0) and OpenAI computer-use agent (25.7). For comparison, Fara-7B also scores 73.5% on WebVoyager, 34.1% on Online-Mind2Web, and 26.2% on DeepShop.

Scores — Fara1.5 family (May 2026): Microsoft released Fara1.5 alongside MagenticLite in May 2026. Three models on Qwen 3.5 base; MIT license (no per-seat restrictions):

Model Online-Mind2Web WebVoyager WebTailBench (process) WebTailBench (outcome)
Fara1.5-27B 72.0% 88.6% — —
Fara1.5-9B 63.4% — 64.5% 32.3%
Fara1.5-4B — — — —
Fara-7B (prior gen) 34.1% 73.5% 38.4 26.2%
OpenAI Operator 58.3% — — —
Gemini 2.5 CU 57.3% — — —

Fara1.5-27B’s 72.0% Online-Mind2Web is the highest published score for a non-proprietary model on that benchmark as of May 2026, exceeding both OpenAI Operator (58.3%) and Gemini 2.5 Computer Use (57.3%). [TIER 2 — Microsoft blog/release notes; independent replication not available at publication]

OSWorld and ScreenSpot scores: Not published for the Fara1.5 family as of May 2026. Do not assume desktop/GUI capability from web agent scores.

MagenticLite architecture (paired release): MagenticLite is the next-generation Magentic-UI agentic system built on top of Fara1.5. Architecture: MagenticBrain (Qwen 3 8B fine-tuned orchestrator) + Fara1.5 web agent + Quicksand sandbox execution environment. MIT license. MagenticBrain handles task planning and intent resolution; Fara1.5 handles web navigation execution. Full codebase released; no published MagenticLite-level benchmark scores separate from Fara1.5 component scores.

Enterprise relevance: WebTailBench’s long-tail task coverage is most relevant for enterprises whose agents handle workflows that fall outside the structured task categories covered by WebArena and WorkArena — mixed-mode internal/external workflows, vendor portal interactions, ad-hoc information retrieval from non-standardized sites. The Fara1.5 family’s MIT license removes per-deployment licensing friction; the 27B model is self-hostable on a single A100 or equivalent. The absence of OSWorld/ScreenSpot scores for Fara1.5 means desktop automation claims cannot be validated from published benchmarks.

4. Enterprise Agentic Workload Mapping

4.1 Benchmark-to-use-case mapping

The practical question for enterprise architects is not “what is the top MMMU-Pro score?” but “which benchmark scores should I look at when evaluating an AI system for my specific workload?” The following mapping organizes benchmarks by the enterprise workflow they most directly test.

Enterprise Workflow Primary Benchmark Secondary Benchmark Key Metric
Invoice / contract document extraction OmniDocBench (table + formula sub-scores) OCRBench v2 Table TEDS; formula CDM
Mixed PDF document processing OCRBench v2 MMMU-Pro (vision-only setting) Sub-50 score = below bar
Financial / operational BI dashboard analysis CharXiv (reasoning track) MathVista Reasoning score, not aggregate
Scientific / engineering report analysis MathVista / MathVerse MMMU-Pro Diagram-only variant score
Desktop task automation (Windows fleet) OSWorld / WindowsAgentArena-V2 OSWorld-MCP Score delta (GUI vs. MCP)
Desktop automation (professional apps) ScreenSpot-Pro UI-CUBE Resolution score variance
Mobile device automation AndroidWorld SR — Dynamic task success rate
ServiceNow / ITSM workflow automation WorkArena (atomic) + WorkArena++ — List-filter task pass rate
Web research / competitive intelligence BrowseComp GAIA Level 3 Multi-hop retrieval score
General-purpose web task automation WebArena Online-Mind2Web Do not rely on WebVoyager alone
Enterprise knowledge work (multi-step) GAIA Level 2/3 τ-bench Scaffolded vs. bare delta

4.2 Why benchmark scores systematically overstate production deployment performance

Five structural mechanisms create the gap between published benchmark scores and enterprise production performance. Each is observable in the benchmark data.

1. Benchmark resolution normalization. UI-CUBE (UiPath, Nov 2025) documented 56.7% action miss rates in enterprise GUI tasks. Almost all published OSWorld and web agent scores are collected at Full HD (1920×1080). Enterprise deployments span 1024×768 (legacy terminals), Full HD, and 4K (professional workstations). Performance degrades at non-training resolutions because pixel coordinate patterns learned at one resolution do not transfer geometrically to another.

2. Static vs. dynamic UI. WebArena and OSWorld use self-hosted, static application instances with predictable UI states. Production enterprise applications have dynamic content: live dashboards, notification banners, session-timeout modals, A/B-tested UI layouts, and feature flags. WorkArena’s 0% on list-filter tasks is an example of a real dynamic UI pattern that static benchmark environments do not surface.

3. Enterprise-specific UI patterns outside benchmark distributions. ScreenSpot-Pro was specifically designed to address this: professional applications (CAD, medical imaging, financial terminals) use non-standard HTML and widget patterns. Even GUI-specialist models trained on consumer web UI achieve only 47.5–56.5% on ScreenSpot-Pro vs. 86.3% for frontier VLMs. Benchmark training corpora over-represent consumer web UI.

4. Error recovery in multi-step sequences. Most benchmarks evaluate single-task success rates. Enterprise workflows are multi-step sequences where a failure at step N requires recovering the environment to a valid state before proceeding. The OSWorld-Human finding (2.7–4.3× more steps than humans) reflects agents using brute-force retry strategies that do not transfer to long-horizon tasks where accumulated errors compound.

5. No enterprise context. Benchmark tasks come with complete, unambiguous natural language instructions. Enterprise workflows rarely do. Agents in production must handle vague task descriptions (“process the invoices from last week”), implicit constraints (“only update rows where the status is pending”), and domain-specific terminology absent from benchmark tasks.

4.3 The benchmark-to-deployment translation framework

A four-step process for translating benchmark scores into procurement and deployment decisions.

Step 1 — Match the benchmark to the workflow (Section 4.1 table). Do not use aggregate VLM scores to evaluate a document processing deployment; use OmniDocBench table and formula sub-scores.

Step 2 — Apply a production discount factor. Based on UI-CUBE findings and the OSWorld-Human efficiency gap, apply a 15–25% discount to published task success rates for enterprise deployments running at non-normalized resolution, on non-standard enterprise UI, or on multi-step workflows requiring error recovery. A system reporting 72% on OSWorld may realistically achieve 50–60% on equivalent enterprise tasks in your specific environment.

Step 3 — Measure the MCP/tool architecture premium. OSWorld-MCP shows that providing structured tool APIs improves task success rates materially (e.g., 8.3% → 20.4% for o3; 40.1% → 43.3% for Claude 4 Sonnet). For every deployment, identify which applications offer native APIs and design the agent to use them first; GUI-only deployment is the suboptimal baseline.

Step 4 — Require vendor evaluation at deployment conditions. Published benchmark scores are necessary but not sufficient. Require that any vendor claiming benchmark performance reproduce that performance on: (a) your application stack, (b) at your display resolution profile, © on tasks drawn from your actual workflow, not benchmark task distributions.

4.4 Current capability ceiling: what agents can and cannot reliably do

The benchmark data collectively defines a capability map as of May 2026.

Reliable (benchmark score ≥70%, broadly validated):

  • Document text extraction from standard-format PDFs (OmniDocBench ~94% for top systems)
  • Chart reading from scientific papers — descriptive track (CharXiv descriptive ~90%+)
  • Web research synthesis across multiple sources (BrowseComp 74.9% top)
  • Atomic web navigation on well-structured consumer applications (WebArena 71.6%)
  • Desktop GUI task completion on standard applications at training resolution (OSWorld ~72–75%)

Emerging (40–70%, useful with human oversight):

  • Multi-step desktop workflows requiring error recovery (OSWorld with MCP ~43%)
  • Mobile device automation on real Android apps (AndroidWorld ~64–71%)
  • Professional-grade GUI grounding on enterprise applications (ScreenSpot-Pro ~47–56% for specialized models; frontier VLMs 72–86%)
  • Web agent tasks on live, dynamic websites (Online-Mind2Web ~34–61%)

Not yet reliable (<40% or validated only on specialized systems):

  • Compositional multi-step enterprise workflow automation (WorkArena++ L3 ~0%)
  • Multimodal web tasks with visual decision dependencies (VisualWebArena 16.4%)
  • Computer use on enterprise-specific custom applications (no published benchmark)
  • Long-horizon autonomous workflows with error accumulation (no dedicated benchmark at scale)

The operationally significant boundary is between “atomic task reliable” and “multi-step compositional reliable.” Every enterprise agentic deployment should be scoped to atomic or short-horizon tasks until WorkArena++ L3 and VisualWebArena scores improve substantially.

5. Key Data Points

Benchmark Publisher Current SOTA Human Baseline Gap Enterprise Use Case
MMMU-Pro CMU et al. ~94% (GPT-5.4 Pro, tracker) ~80% At ceiling Multidisciplinary doc/image understanding
OCRBench v2 Huawei Noah’s Ark >50 / 100 (frontier, early 2025) ~100% Large at launch Invoice, contract, technical doc OCR
OmniDocBench v1.5 OpenDataLab ~94% (GLM-OCR) ~100% Approaching saturation PDF parsing pipeline procurement
CharXiv (reasoning) Princeton NLP 93.2% (Claude Mythos Preview) ~85% Exceeded human BI dashboard, scientific report analysis
ScreenSpot-Pro ACM MM 2025 86.3% (GPT-5.2, tracker) ~100% ~14 pp Professional GUI grounding
OSWorld HKU / NeurIPS 2024 ~75% (GPT-5.4 verified) ~78% ~3 pp aggregate Desktop task automation
OSWorld-MCP Alibaba / ICLR 2026 +3–12 pp over GUI-only — Tool design choice API-first computer use deployment
WindowsAgentArena-V2 Microsoft ~36% (PC Agent-E) 74.5% ~38 pp Windows desktop automation
UI-CUBE UiPath 56.7% action miss rate ~100% ~43 pp Enterprise CUA reliability
AndroidWorld SR Google DeepMind / ICLR 2025 ~71% (Qwen3.5, tracker) 80% ~9 pp Mobile automation
WebArena CMU / NeurIPS 2023 71.6% (OpAgent, Jan 2026) 78% ~6 pp Web automation, ITSM, research
VisualWebArena CMU 2024 16.4% (best VLM agent) 88.7% 72 pp Multimodal web tasks — NOT production-ready
WorkArena++ (L3) ServiceNow ~0% (all tested models) 93.9% ~94 pp Enterprise SaaS multi-step workflows
BrowseComp OpenAI 2025 78.4% (Kimi K2.5 + Swarm) 29.2% Exceeds human Deep web research, competitive intelligence
GAIA (Level 3) Meta AI / HuggingFace ~58% (Manus 1.5 reported) 92% ~34 pp Multi-step tool-augmented knowledge work
WebTailBench V2 Microsoft Research 72.0% OMW / 88.6% WV (Fara1.5-27B, May 2026) — — Long-tail enterprise web workflows

Scores marked “tracker” are from third-party leaderboard aggregators (BenchLM, llm-stats.com, Artificial Analysis) and carry TIER 2 credibility — methodology varies by tracker. Official leaderboard scores should be confirmed for procurement decisions.

What This Means for Your Organization

The benchmark landscape summarized here has one practical implication: published scores are necessary but not sufficient for procurement decisions. A vendor reporting 72% on OSWorld may realistically achieve 50–60% on your specific application stack at your display resolution. WorkArena’s 0% on ServiceNow list-filter tasks is not an edge case — it is a preview of what happens when agents trained on consumer web UI encounter the non-standard HTML patterns in every major enterprise SaaS platform.

The four-step translation framework in Section 4.3 translates benchmark scores into defensible procurement criteria. Match the benchmark to the workflow. Apply the production discount. Measure the MCP/tool architecture premium. Require vendor evaluation at deployment conditions. None of these steps require ML expertise — they require procurement discipline applied to a new asset class.

The capability ceiling in Section 4.4 defines the current deployment boundary. Atomic tasks at standard resolution are reliable today. Compositional multi-step enterprise workflows — the tasks that actually move the business — are not. Deploy agents in the reliable tier with human oversight on the emerging tier; do not deploy in the not-yet-reliable tier regardless of vendor claims. If this raises questions about how to structure those vendor conversations or map benchmark selection to a specific workflow, I’d welcome the discussion — brandon@brandonsneider.com.

Sources

VLM Benchmarks — Primary:

Computer Use Benchmarks — Primary:

Web Agent Benchmarks — Primary:

Enterprise Deployment / Production Gap:

Leaderboard Trackers (TIER 2):

Brandon Sneider | brandon@brandonsneider.com May 2026