The Finance AI Evidence Base —
Every Score Sourced

The most demanding finance agent evaluation: multi-hop SEC filing retrieval, web search, and quantitative reasoning in a single task. Every frontier model available in May 2025 was tested; none crossed 50%. The ceiling has not moved in subsequent smaller evaluations.

Rubric-graded scoring exposes a critical gap: models that appear to pass binary tests fail when graded on derivation quality. The two bars below measure the same systems on the same tasks — only the grading method changes.

Global financial search is nearly solved; the China market is not. Grok 4 with web access approaches human-level performance on global topics but the gap triples on Chinese-market questions requiring access to regional data sources.

The benchmark splits recall (memorized certification facts) from reasoning (multi-step scenario problems). Models appear capable on certification; the scenario bar shows what happens under realistic task conditions.

RL post-training with GRPO boosts Excel task completion by 5–9pp versus base models. Finance tasks are the hardest domain because formula dependencies chain across cells — the same multi-step failure mode that collapses FinMathBench scores from 73% to 14%.

Tasks require a model to independently plan and execute multi-step analysis over real financial record sets — no scaffolding, no prescribed steps. The frontier/non-frontier gap is stark: this is a task class where model capability tier directly determines usability.

As formula chains lengthen, accuracy collapses at each step. This is not a knowledge gap — GPT-4o knows the formulas. It's a chaining failure: each additional step multiplies the error probability. Real finance calculations (derivative pricing, rebalancing with constraints, layered risk) require 3–5 linked steps.

Small open-weight reasoning models now match or beat frontier API models on structured math. The 3.8B Phi-4-mini leads — a result that changes the build-vs-buy calculus for any finance workflow involving formula chains.

AIME tests competition-grade mathematical reasoning — closer to FinMathBench's multi-formula chains than to single-step recall. The same small model that leads MATH-500 nearly matches o1-mini here at a fraction of API cost.

Parameter count is a hard floor for structured output reliability. On BFCL v3 (Jun 2026), the leaderboard leader is GLM 4.5 at 77.8% — the 97% figure for ToolACE-8B was from the original BFCL v1/v2 eval. Granite 4.1 8B at 68% and FunctionGemma-270M at 0% show the lower end of the spectrum.

Tasks are grounded in actual hedge fund analyst reasoning traces: the benchmark knows what a professional would do and grades deviation from it. No agent crosses 20% in any single category — showing that the failure is not domain-specific but structural.

The only finance benchmark where models consistently cross 50%: executable quantitative code that passes numerical pytest verifiers. Options pricing, factor construction, and credit modeling are tractable for frontier models when the task is code, not prose.

The top 5 models are all small, open-weight, purpose-built OCR systems — not frontier API models. GPT-4o and Claude 3 do not appear in the top tier. For financial document ingestion (10-Ks, prospectuses, filings), dedicated OCR models outperform general-purpose LLMs by a wide margin.

Open-weight embedding models now outperform both Gemini Embedding and OpenAI text-embedding-3-large on multilingual tasks. For financial RAG over SEC filings, earnings transcripts, and regulatory documents, the build-vs-API decision has shifted.

At 239M parameters, jina-v5-nano beats OpenAI's flagship on MTEB English while running entirely on-device. For financial RAG pipelines ingesting large document volumes, this is the cost-efficiency argument — API latency and per-call costs eliminated at zero quality cost.