SLM Finance Benchmark Landscape

The default procurement posture in financial services AI is to reach for the most capable frontier model — GPT-4-class, Claude Opus, Gemini Pro — for every task. The benchmark evidence, across more than twenty independent evaluations published in 2025 and 2026, does not support that posture. For specific finance task classes, domain-fine-tuned and small models match or beat frontier models. For other task classes, scale still matters. Knowing the difference is the procurement decision.

The Benchmark Landscape for Finance AI

Twenty independent or semi-independent benchmarks published between July 2025 and June 2026 now evaluate AI systems on realistic financial tasks. The coverage is broad: SEC filing analysis, financial search, spreadsheet automation, multi-formula calculation, tool-use compliance, exploratory data analysis, quant strategy coding, full workbook creation, professional investment-research quality, hedge-fund analyst reasoning, multilingual/multimodal finance, chart reading, and financial fact-level OCR.

Finance-specific benchmarks with credible methodology:

Benchmarks to use with caveats: FIRE (Du Xiaoman + Tsinghua + Renmin University, Feb 2026 — 17,000+ finance certification questions, useful for knowledge breadth but not agentic capability). AFIB/SuperInvesting (Mar 2026 — useful for failure taxonomy; vendor-originated, not independent model ranking). FinMMEval CLEF 2026 — benchmark design is sound, results are maturing. Fin-PRM — method signal for process supervision, not model ranking.

What each benchmark actually tests matters. Finance Agent Benchmark uses a live agentic harness with real tool access and post-2024 documents — minimizing contamination. FinMathBench isolates formula-chain depth specifically. QFBench runs generated code in a Docker sandbox and scores it against numerical pytest verifiers — executable verification, not answer-text grading. Benchmarks that score final answers only (without visible reasoning, source citations, or formula traces) are structurally insufficient for finance workflows where auditability is a legal requirement. BigFinanceBench’s rubric-level scoring — decomposing analyst workflow into entity identification, source selection, line-item retrieval, accounting adjustment, formula construction, calculation, and synthesis — is the right model for finance evaluation design.

Where Small Models Win

Sentiment classification and routing

On narrow sentiment labeling, ModernFinBERT-base (a small domain-fine-tuned encoder from neoyipeng) achieves 73.3% label accuracy on a finance sentiment fixture scored locally via Transformers + MPS, outperforming YorkFr’s Qwen3-family finance fine-tune (66.7%) on the same fixture. The mechanism: ModernFinBERT was trained on a narrow financial-text distribution. The Qwen3 fine-tune is not better on this specific task despite a more capable base model. This is a measured result from the finance-sentiment-routing-v0 suite (June 2026 ledger); not a paper claim.

The critical boundary: both models fail full workflow routing (0/15 accepted for routing + rationale output). Narrow classifiers solve narrow problems. Routing, abstention, rationale generation, and compliance output require a different architecture.

Math reasoning at 3.8B parameters

Phi-4-mini-reasoning (Microsoft Research, 3.8B, MIT license) achieves 94.6% on MATH-500 — outperforming DeepSeek-R1-Distill-Qwen-7B (91.4%) and DeepSeek-R1-Distill-Llama-8B (86.9%), and matching o1-mini (90.0%) [arXiv:2504.21233, TIER 1]. On AIME 2024, it scores 57.5% versus o1-mini at 63.6%.

The mechanism is not scale — it is training method. The 4-step recipe (mid-training on distilled long-CoT → SFT → Rollout DPO → RL with verifiable reward) on synthetic DeepSeek-R1 data produces math reasoning that smaller models with generic training cannot replicate. For financial modeling workflows that require structured calculation — not multi-step cross-document reasoning — a 3.8B model with a specialized training objective is viable, runs locally with 128K context, and costs a fraction of frontier API pricing.

Document parsing and OCR

OCR is solved at the sub-4B scale for structured document parsing. GLM-OCR (Zhipu AI, 0.9B, MIT) scores 94.6 on OmniDocBench v1.5 — ranked #1 across 29 evaluated models, including Gemini-3-Pro and GPT-5.2 [arXiv:2603.10910, TIER 1]. MonkeyOCR-3B (HUST + Kingsoft Office, Apache 2.0) outperforms InternVL3-8B by +12.1% and Qwen2.5-VL-72B on English document parsing, at 7× higher throughput (0.84 vs. 0.12 pages/second) [arXiv:2506.05218, TIER 1].

Note the benchmark distinction: OmniDocBench uses LLM-as-judge for some sub-tasks. olmOCR-Bench (Ai2) uses deterministic binary unit tests across 7,000+ tests — harder to game. The two are not comparable. For financial fact-level OCR (critical fields, reporting dates, monetary values, table headers), FinCRITICAL-ED (arXiv:2511.14998, Nov 2025) documents that text extraction is not sufficient — financially critical fields must survive with correct values and context. Sub-4B specialist document models beat frontier VLMs on throughput and accuracy for structured financial documents.

Embeddings and retrieval

Qwen3-Embedding-8B (Alibaba, Apache 2.0) ranks #1 on MTEB Multilingual at 70.58 — above Gemini-Embedding and OpenAI text-embedding-3-large [TIER 1 — MTEB leaderboard, Qwen official blog]. jina-embeddings-v5-text-nano (239M, Jina AI, Apache 2.0) scores 65.5 on MMTEB (131 tasks, 9 task types) and 71.0 on MTEB English [TIER 1 — Jina AI release notes].

The implication: embedding is a task class where parameter count stopped being the dominant variable in 2025. Architecture and training data distribution, not scale, determine ranking. For financial RAG pipelines with large document volumes, routing embedding through commercial APIs when open-weight 239M–8B models are competitive is an economically indefensible position.

Function calling and tool use

ToolACE-8B (Team-ACE, academic, ICLR 2025, Apache 2.0) — an 8B fine-tune of LLaMA-3.1-8B-Instruct on 26,507-API synthetic data — achieves state-of-the-art accuracy on BFCL v3, rivaling GPT-4 models at publish time [arXiv:2409.00920, TIER 1]. For financial tool routing where tool schemas are stable and well-defined, small fine-tuned models can match frontier models on binary pass/fail.

The FinToolBench caveat applies: binary tool-use accuracy does not equal regulatory compliance. Timeliness Mismatch Rate (TMR — wrong data period), Intent Mismatch Rate (IMR — escalating informational queries to transactional execution), and Domain Mismatch Rate (DMR — wrong regulatory domain) are separate from pass/fail accuracy and require separate evaluation. A model can pass FinToolBench’s binary checks while failing on all three compliance dimensions.

What Frontier Models Still Win

Scale still matters on several task classes in finance.

Complex multi-step financial research. Finance Agent Benchmark (Vals AI + Stanford, 537 questions, real EDGAR + web access): the best-performing model is OpenAI o3 at 46.8%. No model crosses 50%. Human expert performance on the same benchmark: well above 90% [arXiv:2508.00828, HIGH]. This is a 43+ percentage point gap on tasks that represent real financial analyst work. Frontier models are the ceiling here; small models are not viable for autonomous complex research tasks.

Hedge-fund analyst reasoning. Hedge-Bench (Trata + BYU + Osmosis, Jun 2026, 102 tasks grounded in professional reasoning traces): frontier agents score below 16% pass@1 [arXiv:2606.03918]. Small models are not in contention.

Multi-formula calculation chains. FinMathBench (Ant Group, AAAI-26): GPT-4o drops from 72.9% on single-formula problems to 14.0% on four-formula chains. This is a frontier model. Small models with generic training will perform worse on multi-step financial math. Phi-4-mini-reasoning is the exception, not the rule — it is a specialist, not a general-purpose small model.

Confidence calibration on underdetermined problems. RealFin (INSAIT + Newcastle + MBZUAI, Apr 2026, 2,020 questions): all 15 models tested — including finance-specialized models — systematically over-commit when essential premises are missing. Finance-specialized small models do not produce better abstention behavior than frontier models. The overconfidence failure is domain-wide, not model-class-specific.

Exploratory data analysis. DataClawBench (Sun Yat-sen University, May 2026, 492 tasks, 2.06M real records): Claude Opus 4.6 leads with 63.4% overall; 7 of 8 models are below 50%. Exploratory analysis — where the agent must identify what data is relevant, navigate noise, and synthesize across domains — breaks agent reliability in ways that structured-task performance does not predict. Frontier models are the appropriate choice here; small models are not competitive.

What This Means for a CIO Evaluating Vendors

Which task classes warrant small model evaluation

Before any vendor procurement, classify the finance AI tasks by type:

• Narrow classification (sentiment, routing, labeling): evaluate small fine-tuned encoders (ModernFinBERT-class). Do not use frontier APIs for this volume.
• Document extraction and OCR (SEC filings, contracts, invoices): evaluate sub-4B specialist parsers. Benchmark against olmOCR-Bench (deterministic), not OmniDocBench alone (LLM-as-judge).
• Embedding and retrieval: evaluate open-weight embedding models (Qwen3-Embedding, jina-v5-nano) against your actual document corpus before committing to commercial embedding APIs.
• Structured math with finite formula chains: evaluate specialist reasoning models (Phi-4-mini-reasoning class) on your specific formula types before defaulting to frontier.
• Complex research, multi-step analysis, exploratory tasks, hedge-fund reasoning: frontier models are the correct tier. No small model is competitive on these tasks in the current benchmark evidence.

Specific evaluation criteria to require from vendors

1. Which benchmark was used, and who administered it. Vendor-selected benchmarks favor the vendor. Require at least one independently administered evaluation — LiveCodeBench (execution-based), Finance Agent Benchmark (Vals AI + Stanford), or FinMathBench (Ant Group + AAAI peer review).
2. Formula-chain depth test. If the use case involves multi-step financial calculations, require the vendor to run your actual formula chains — not single-formula accuracy. FinMathBench’s GPT-4o collapse from 72.9% to 14.0% at four formulas illustrates why single-formula scores are insufficient.
3. Compliance dimension scores, not just accuracy. For regulated institutions, require TMR, IMR, and DMR rates (Timeliness Mismatch, Intent Mismatch, Domain Mismatch) on financial tool-use queries. Binary pass/fail is the wrong metric for compliance-sensitive workflows.
4. Rubric-level scoring, not final-answer accuracy. BigFinanceBench demonstrates that models can reach partial rubric credit while missing critical steps — source selection, period definition, accounting adjustment — that a human reviewer would need to audit. Require vendors to show derivation traces, not just answers.
5. Consistency across repeated runs, not single-run accuracy. The CLEAR framework (arXiv:2511.14136, Nov 2025, 300 enterprise tasks) documents that agent performance drops from 60% on a single run to 25% on eight-run consistency tests. Single-run accuracy is a demo metric. Require five-run consistency reporting.

Red flags in vendor benchmark claims

• MMLU, GSM8K, or BIG-Bench Hard as primary evidence. All frontier models score 90%+ on MMLU and ~99% on GSM8K. These benchmarks cannot discriminate among them. A vendor citing these numbers is not providing decision-relevant information.
• “Internal benchmark” without published methodology. Not reproducible, not independent, not useful.
• Finance accuracy figures without formula-chain depth breakdown. Single-formula performance does not predict multi-formula performance. The 72.9% → 14.0% collapse at four formulas is a documented, consistent failure mode.
• Benchmark scores without cost-per-correct-answer. Finance Agent Benchmark documents $3.79 per task (o3) versus $25.66 estimated for a human analyst. This is the relevant economic frame. Cost-blind accuracy comparisons are incomplete.
• Claims that finance-specialized models have better uncertainty calibration. RealFin documents explicitly that finance-specialized models do not reliably identify missing premises. Specialization on financial content does not produce calibrated uncertainty.

The build vs. buy signal from the benchmark data

For task classes where small models win — document parsing, embedding, sentiment classification, narrow tool routing — the open-weight models are Apache 2.0 or MIT licensed, run on local infrastructure, and require no API dependency. The economic case for in-house deployment is strong when volume is sufficient to justify the engineering cost. GLM-OCR (MIT, 0.9B), MonkeyOCR-3B (Apache 2.0), Qwen3-Embedding-8B (Apache 2.0), ToolACE-8B (Apache 2.0) are all commercially deployable without royalties.

For task classes where frontier models still lead — complex research, multi-step exploratory analysis, hedge-fund reasoning — the benchmark evidence makes clear that no open-weight small model is competitive. API dependency is the correct architecture for these tasks, with human review at the output layer. RealFin’s overconfidence finding applies here: do not design workflows that assume the model will flag its own uncertainty on underdetermined problems. It will not.

Citations

1. Finance Agent Benchmark — Krishnan, Wu, Nashold (Vals AI + Stanford). arXiv:2508.00828. May 2025. n=537 questions. HIGH.
2. FinSearchComp — ByteDance Seed + Columbia Business School. arXiv:2509.13160. September 2025. n=635 questions, 21 models, 70 expert annotators. HIGH.
3. FinMathBench — He, Wang, Xiong, Chen, Hu (Ant Group). AAAI-26. n=946 questions, 40 models. HIGH.
4. RealFin — Dai, Lin, Xie, Wang (INSAIT + Newcastle + MBZUAI). arXiv:2602.07096. April 2026. n=2,020 bilingual questions, 15 models. HIGH.
5. FinToolBench — Lu, Wang, Wang, Tang (Shanghai AI Lab + Tencent). arXiv:2603.08262. March 2026. n=760 tools, 295 queries. HIGH.
6. DataClawBench — Zhang, Ye, Chen et al. (Sun Yat-sen University). arXiv:2605.02503. May 2026. n=492 tasks, 2.06M financial records. MEDIUM-HIGH.
7. BigFinanceBench — Wang, Meinhardt, Katz et al. arXiv:2606.03829. June 2026. n=928 tasks, 15,656 rubric criteria. MEDIUM-HIGH.
8. Hedge-Bench — Cho, Huang, Lu, Lyu. arXiv:2606.03918. June 2026. n=102 tasks. MEDIUM-HIGH.
9. Spreadsheet-RL — Chi, Xie, Wu et al. (UIUC + Meta). arXiv:2605.22642. May 2026. n=1,660 tasks. MEDIUM-HIGH.
10. QFBench — public benchmark, V11 snapshot May 4, 2026. qfbench.com. MEDIUM-HIGH.
11. FinCRITICAL-ED — arXiv:2511.14998. November 2025. Financial fact-level OCR. MEDIUM-HIGH.
12. Phi-4-mini-reasoning — Microsoft Research. arXiv:2504.21233. 3.8B model, MATH-500: 94.6%. TIER 1.
13. GLM-OCR — Zhipu AI. arXiv:2603.10910. March 2026. 0.9B, OmniDocBench: 94.6 (#1). TIER 1.
14. MonkeyOCR-3B — HUST + Kingsoft Office. arXiv:2506.05218. 3B, +12.1% over InternVL3-8B, 7× throughput. TIER 1.
15. Qwen3-Embedding — Alibaba. arXiv:2506.05176. 8B MTEB Multilingual: 70.58 (#1 above Gemini-Embedding and OpenAI text-embedding-3-large). TIER 1.
16. jina-embeddings-v5-text-nano — Jina AI. 239M, MMTEB: 65.5, MTEB-en: 71.0. TIER 1 (Jina AI release notes).
17. ToolACE-8B — Team-ACE. arXiv:2409.00920. ICLR 2025. BFCL v3: state-of-the-art, rivals GPT-4. TIER 1.
18. Finance Measured Runtime Evidence Ledger — internal (June 2026). ModernFinBERT label accuracy: 73.3%; sentiment routing evidence. Measured.
19. CLEAR framework — Santhanam et al. arXiv:2511.14136. November 2025. n=300 enterprise tasks, 6 agents. 60% single-run → 25% eight-run consistency. HIGH.
20. OmniDocBench — CVPR 2025, opendatalab. LLM-as-judge composite, 981 PDF pages, 9 document types.
21. olmOCR-Bench — Ai2. arXiv:2502.18443. Deterministic binary unit tests, 7,000+ tests, 1,400 PDF pages.

State of AI — Executive Briefings | stateofai.pages.dev | 2026-06-13