← Local Tiny Models 🕐 19 min read
Local Tiny Models

The Egress Boundary: Why Enterprise Teams Are Running AI Locally

Brandon Sneider · May 2026

> **Source credibility: MIXED. TIER 1–2.**

Source credibility: MIXED. TIER 1–2. Practitioner documentation (Ollama, LM Studio, MLX community benchmarks), academic preprints, analyst projections, and empirical observation; runtime benchmarks from community sources are directionally useful but not reproducible at the same confidence level as n=1,000+ surveys, while sovereign AI survey data (Deloitte n=3,235; Gartner Oct 2025) is MEDIUM-HIGH and model benchmark data from official technical reports is HIGH for the specific tasks tested.

See also (wiki): wiki/ai-regulation-global-synthesis.md · wiki/ai-implementation-cost-structure.md · wiki/local-tiny-models.md · wiki/ai-sovereignty.md · wiki/dpa-friction-ai-vendors.md

Executive Summary

  • The local LLM ecosystem crossed a production-readiness threshold in 2025. Ollama alone reports 52 million monthly downloads, 100K+ GitHub stars, and 200% year-over-year enterprise adoption growth. LM Studio reached $1.8M ARR with a 16-person team by mid-2025, with Fortune 500 companies, universities, and global organizations as named customers.
  • The governance case — not the cost case — is the primary enterprise driver. GDPR Article 46 transfer restrictions, HIPAA, ITAR air-gap requirements, and attorney-client privilege create legal barriers to routing sensitive data through US-hosted cloud APIs. Local deployment resolves those barriers by eliminating the data transfer entirely. Gartner (Oct 2025) projects that by 2030, 75%+ of European and Middle Eastern enterprises will geopatriate virtual workloads; current adoption sits below 5%.
  • On specific task categories — structured document extraction, classification, summarization, code assistance on internal codebases — fine-tuned or quantized small models now match or exceed frontier APIs. Extract-0 (7B, fine-tuned for document extraction) outperformed GPT-4.1, o3, and GPT-4.1-2025 on a 1,000-task benchmark at a total training cost of $196.
  • Hardware has reached a practical enterprise threshold. A Mac Studio 64 GB running Qwen3.5-VL at 4-bit quantization processes structured documents at 4–9 seconds/page, peaks at 6.7 GB memory, and costs zero per inference after hardware purchase. Ollama’s March 2026 MLX integration for Apple Silicon delivers 1,810 tokens/s prefill and 112 tokens/s decode. An NVIDIA RTX 5000 PRO 48 GB workstation GPU runs Qwen3 27B at 80 tokens/second with a 200K-token context window.
  • The break-even analysis is straightforward for high-volume workloads. Against GPT-4-class APIs, on-premise deployment typically breaks even at approximately 2 million tokens/day at 70% GPU utilization, with most teams reaching payback within 6–12 months. Organizations that combine on-premise inference with model-right-sizing and prompt caching report 50–90% total API cost reduction.

Adoption Landscape

Ecosystem Scale

As of November 2025, the local LLM ecosystem encompassed 695 models and 110 runtime frameworks, with the ecosystem described by practitioners as having reached “production maturity.” Two platforms dominate the runtime layer:

Ollama (open-source CLI + REST API server):

  • 52 million monthly downloads as of 2025
  • 154,856 GitHub stars (Nov 2025)
  • 1.3M monthly NPM downloads
  • 200% year-over-year enterprise adoption growth
  • ~174,590 identified global instances; ~41,021 with publicly accessible APIs (24% rate)
  • Geographic distribution: United States (29,195 IPs), China (16,464), Germany, UK, Japan, India, Singapore
  • March 2026: switched inference backend to Apple’s MLX framework for Apple Silicon

Source credibility: MEDIUM. Ollama GitHub stars and download figures are self-reported or community-tracked. The 200% enterprise growth figure is directionally consistent with practitioner observation but lacks a specific denominator. Instance scan data is from passive discovery, not enterprise disclosure.

LM Studio (desktop GUI for local LLM management):

  • $1.8M ARR, 16-person team as of June 2025
  • Fortune 500 companies, universities, and global organizations listed as enterprise customers
  • Made free for professional/commercial use in 2025, removing prior license requirements
  • Enterprise tier adds SSO, model gating, MCP integration, private team collaboration
  • Available for macOS, Windows, Linux

Source credibility: MEDIUM. Revenue figure sourced from Latka database (June 2025). Customer list is general category disclosure, not named. The free-for-work policy change is confirmed via LM Studio’s own blog post.

Enterprise Survey Data

Deloitte’s State of AI Enterprise 2026 (n=3,235, 24 countries, Aug–Sep 2025) found:

  • 83% of organizations view sovereign AI as strategically important; 43% rate it very or extremely important
  • 77% factor country of origin into AI vendor selection
  • 60% are building AI stacks primarily with local vendors
  • 66% express concern about reliance on foreign-owned AI infrastructure

These are procurement decisions, not policy positions. The local model adoption trend is the logical implementation layer for sovereign AI procurement criteria.

ESG (Enterprise Strategy Group) 2025 AI infrastructure research found 71% of AI infrastructure now operates outside public cloud environments, driven primarily by financial sector demand.

Source credibility for ESG figure: LOW-MEDIUM. This figure circulates in practitioner content but the primary ESG report was not directly verified. Treat as directional.

The Governance Case

This is the most important driver and the least-discussed in model benchmark comparisons.

GDPR and Data Residency

European enterprises routing customer or employee data through US-hosted cloud APIs face legal risk under GDPR Article 46 (transfers to third countries without adequate safeguards). The EU Data Act (effective September 2025) adds explicit data portability and locality rights. Private local deployment resolves the Article 46 question by eliminating the transfer entirely — there is no “transfer” when data never leaves the enterprise network.

The European Data Protection Board’s April 2025 analysis of AI privacy risks in LLMs explicitly identifies prompt data residency as a compliance risk for cloud-hosted models.

HIPAA (Healthcare)

HIPAA’s minimum necessary standard and Business Associate Agreement requirements create procurement overhead for every cloud AI vendor touching PHI. A European healthcare provider (case documented in multiple practitioner sources) runs on-premise SLMs to extract structured insights from patient records — no BAA required, no external data path, no audit trail gap.

ITAR (Defense/Aerospace)

ITAR mandates that controlled technical data not traverse infrastructure accessible to foreign nationals or foreign-hosted systems. This effectively requires air-gapped deployment for defense-adjacent engineering workloads — a requirement that no amount of cloud vendor contractual language can satisfy. Air-gapped local deployment is the only architecturally compliant path.

Financial Regulation

European banks and asset managers operating under GDPR Article 46 cannot legally route customer financial data through US-hosted cloud APIs without specific adequacy decisions or standard contractual clauses. In practice, the legal exposure makes private LLM deployment the operationally cleaner choice. The financial sector’s disproportionate representation in ESG’s 71% off-cloud figure is consistent with this.

Attorney-Client Privilege

Law firms and in-house legal teams increasingly cite privilege as the primary driver. Routing client documents through a third-party cloud API creates a privilege waiver risk in many jurisdictions. Local inference eliminates third-party exposure entirely.

Procurement Simplicity

This is the underrated driver. Every cloud AI vendor requires:

  1. A data processing agreement (DPA)
  2. Security review
  3. IT procurement approval
  4. Often: legal review, privacy impact assessment
  5. Ongoing vendor management

A locally running model requires none of these. Workshop participants — including practitioners at enterprises — reported running Ollama and LM Studio on their own hardware precisely because the absence of procurement overhead was the feature. The tool is available in minutes; the cloud equivalent takes weeks. For individual contributors and small teams, that latency gap determines whether AI gets used at all.

Model Landscape

The 2025–2026 period produced a generation of models that are genuinely capable at enterprise task subsets at sub-14B parameter counts. Key families:

Microsoft Phi-4 Family

Variant Parameters Context Notable capability
Phi-4 (base) 14B 16K Matches GPT-4 on STEM reasoning; AIME 2025 score exceeds DeepSeek-R1 671B
Phi-4-mini 3.8B 128K Long-context summarization; fits in 4–6 GB VRAM at 4-bit
Phi-4-multimodal 5.6B — Simultaneous speech + vision + text; edge deployment
Phi-4-reasoning 14B — Exceeds OpenAI o1-mini and DeepSeek-R1-Distill-Llama-70B on math and science

Microsoft’s design philosophy: high data quality over parameter scale. Phi-4 achieves competitive benchmark performance by training on high-quality curated data rather than raw scale, which translates directly to efficient local inference.

Source: Microsoft Azure Blog (May 2025), Microsoft Turing Engineering blog. Credibility: HIGH for stated benchmarks on tested tasks. MEDIUM for enterprise generalization.

Google Gemma 3

Variant Parameters Key feature
Gemma 3 4B 4B 4.2 GB RAM footprint; multimodal from 4B up
Gemma 3 12B 12B Strong instruction-following; Apache 2.0
Gemma 3 27B 27B Highest Gemma accuracy; fits in 32 GB VRAM at 4-bit

Gemma 3’s distinguishing characteristics for enterprise: pre-tuned alignment for consistency and formatting (an advantage for enterprise tasks where structured output matters), multimodal input from 4B up (useful for document-plus-chart workflows), and Apache 2.0 license (the most permissive in the landscape — material for legal teams).

Meta Llama 3.x

Variant Parameters Notes
Llama 3.2 3B/1B 1–3B On-device / edge; fits any modern laptop
Llama 3.2 11B/90B Vision 11B, 90B Multimodal; 11B runs on Mac Studio 64 GB
Llama 3.3 70B 70B Quality ceiling for local inference; requires 48 GB+ VRAM or Mac Studio 64 GB

Llama 3.3 70B remains the quality leader at local scale. The 70B class requires serious hardware but is achievable on Mac Studio 64 GB (unified memory) or dual RTX 5090s. For constrained hardware, Llama 3.2 3B is the leading edge/mobile deployment option.

Alibaba Qwen2.5 / Qwen3 Family

Qwen has become the most capable open-weight family for multilingual enterprise workloads and structured document extraction. Key data points:

  • Qwen2.5-Max surpasses GPT-4o on mathematical reasoning (MATH benchmark: 72.3 vs 68.1)
  • Qwen3-Coder achieves 69.6% SWE-Bench Verified — the highest open-source coding score as of 2025, surpassing both Claude and GPT-4 on real-world software engineering tasks
  • Qwen2.5-VL-72B matches GPT-4o and Claude 3.5 Sonnet on document and diagram understanding
  • Qwen2.5-32B supports task-specific fine-tuning for under $50 on consumer hardware
  • Full range from 4B to 235B (MoE) under Apache 2.0

Qwen3.5-VL (the model used in the Chandra OCR 2 empirical finding below) is specifically optimized for structured document parsing, with native support for tables, charts, forms, invoices, and multiformat extraction using HTML-formatted output.

Source: Alibaba Qwen technical reports; Qwen blog (qwenlm.github.io). Credibility: HIGH for stated benchmarks. Note: Alibaba is the developer, so treat capability claims as vendor-reported until third-party replicated.

Mistral Family

Variant Parameters License Key advantage
Mistral 7B 7B Apache 2.0 Speed; lowest latency in class
Mistral Small 3.1 22B Apache 2.0 Strong instruction following; MoE efficiency
Mistral Large 2 ~123B Research Quality ceiling; not practical for local laptop

Mistral leads in inference speed for latency-sensitive applications. Apache 2.0 across the full family is the cleanest licensing posture available — no Meta EULA, no Google additional terms. For legal teams reviewing open-weight model licenses, Mistral is the default safe harbor.

Hardware Landscape

Apple Silicon + MLX

Apple Silicon’s unified memory architecture is the transformative factor for local LLM inference: the CPU, GPU, and Neural Engine share the same DRAM pool, eliminating the VRAM bottleneck that limits NVIDIA consumer cards to 24–32 GB. A Mac Studio 64 GB can load Llama 3.3 70B at 4-bit quantization and run inference that would require a $30,000 dual-A100 server on NVIDIA hardware.

Practical throughput benchmarks (2025–2026):

Hardware Model Throughput Notes
M4 Mac Mini (16 GB) 7B 4-bit 33 tok/s Daily driver for assistant tasks
M4 Max (128 GB) 8B 4-bit ~38 tok/s baseline; up to 525 tok/s with continuous batching + prefix caching Optimized multi-user server config
Mac Studio 64 GB Qwen3.5-VL 4-bit (5.8 GB) 4–9 s/page (document OCR) First-party empirical; see below
Ollama + MLX (any Apple Silicon) 7B 1,810 tok/s prefill / 112 tok/s decode Announced March 30, 2026; 93% decode speedup vs prior

The March 2026 Ollama MLX integration is significant: it delivers 57% faster prefill and 93% faster decode across Apple Silicon Macs, making real-time conversational use of 7–14B models entirely practical on standard MacBook Pro hardware.

Sources: Apple Machine Learning Research (WWDC25 MLX + M5 session); Ollama blog (March 30, 2026); community benchmarks from byteiota.com and localaimaster.com. Credibility: MEDIUM-HIGH for Ollama’s own published figures; MEDIUM for community benchmarks (reproducible but not peer-reviewed).

NVIDIA RTX (Consumer and Professional)

Hardware VRAM Throughput (8B 4-bit) Price range
RTX 5090 32 GB GDDR7 ~213 tok/s ~$2,000–2,500
RTX 4090 24 GB GDDR6X ~128 tok/s ~$1,600–2,000 used
RTX 5000 PRO 48 GB 80 tok/s @ Qwen3 27B / 200K context ~$3,500–5,000 (workstation)
RTX A6000 48 GB ~102 tok/s ~$3,500–4,500

The RTX 5000 PRO 48 GB is the current workstation-class reference point for enterprise individual deployments: Qwen3 27B at 200,000-token context at 80 tok/s is sufficient for real-time document analysis, long-context summarization, and code review on internal codebases. Dual RTX 5090s match H100 performance on 70B models at approximately 25% of the cost.

Source: startupfortune.com (RTX 5000 PRO / Qwen3 27B empirical report, 2025); compute-market.com GPU comparison (2026); bdkstudios.com RTX 5090 benchmarks. Credibility: MEDIUM. Community-sourced benchmarks. Directionally reliable.

Task Fit Map

The honest answer to “where do small local models win?” is narrow but commercially important. The categories where the evidence is strongest:

Strong fit (small models match or beat frontier APIs)

Structured document extraction (tables, forms, charts, invoices) The definitive evidence: Extract-0, a 7B model fine-tuned specifically for document information extraction, achieved a mean reward of 0.573 on a 1,000-task benchmark, outperforming GPT-4.1 (0.457), o3 (0.464), and GPT-4.1-2025 (0.459). Training cost: $196. This is the clearest evidence that task-specific fine-tuning of a small local model can exceed frontier performance at a fraction of the cost.

Qwen2.5-VL-72B matches GPT-4o and Claude 3.5 Sonnet on document and diagram understanding at full precision; the 7B and 14B variants show meaningful (but not frontier-parity) document extraction capability at 4-bit quantization.

First-party empirical: Qwen3.5-VL at 4-bit on Mac Studio 64 GB outperformed pymupdf (traditional CV-based extraction) on tables and charts in structured documents, particularly on complex multi-column layouts and embedded charts that pymupdf flattens into undifferentiated text strings.

Classification and entity extraction Well-defined classification tasks (sentiment, category, intent) are among the easiest for small models. Qwen2.5 series maintains 95–98% accuracy on classification benchmarks at 4-bit quantization. Llama 3.1 8B Instruct achieves 98.9% accuracy on coding-adjacent classification at 4-bit.

Code assistance on internal/private codebases Qwen3-Coder at 69.6% SWE-Bench Verified is the current open-source ceiling. More practically: when the codebase cannot be sent to a cloud API (proprietary IP, security classification), a local 14–30B coding model running on a developer workstation is the only option. The quality gap vs. Claude Sonnet or GPT-4o is real but often acceptable for completion and documentation tasks.

Summarization of domain-specific documents Phi-4-mini at 128K context handles long-document summarization within a single context window. For internal documents (legal contracts, technical specs, financial filings) that cannot be sent externally, local inference is the only path regardless of quality tradeoffs.

4-bit quantization accuracy: what the benchmarks show

  • MMLU: accuracy fluctuates ±1% at Q4; ~86% maintained vs full precision for well-quantized models
  • AWQ-4bit accuracy delta: ~1.8% below full precision; GPTQ-4bit: ~2.9% below
  • Qwen2.5 series at 4-bit or 5-bit: 95–98% accuracy on GSM8K and IFEval
  • The quantization format matters as much as bit-width: AWQ-4bit can outperform poorly configured GPTQ-4bit

Source: picovoice.ai sub-4-bit quantization guide; latitude.so quantized LLM cost and performance results; ionio.ai LLM quantize analysis (2025).

Weaker fit (frontier APIs remain ahead)

Open-ended complex reasoning across novel domains — Multi-step reasoning chains on problems requiring broad world knowledge still favor 100B+ models and frontier APIs.

Multimodal tasks requiring fine visual grounding — Identifying specific regions in complex medical images or interpreting ambiguous engineering diagrams benefits from frontier model scale.

Long-horizon agentic tasks with tool use — Reliability on 10+ step tool-use chains degrades meaningfully below 70B. Phi-4-mini and Gemma 3 4B are not production-ready for autonomous multi-step agentic workflows.

High-stakes drafting without human review — Legal drafting, financial disclosures, medical recommendations: the error rate at small model scale requires human review that may eliminate the efficiency benefit.

Cost Analysis

The Break-Even Calculation

A peer-reviewed framework (Arxiv 2509.18101, submitted Aug 2025, revised Nov 2025) provides the theoretical basis for on-premise LLM cost analysis. The practical numbers from practitioner deployments:

Break-even threshold against GPT-4-class APIs: ~2 million tokens/day at 70% GPU utilization. Most teams see payback within 6–12 months on dedicated hardware.

Hardware cost reference points (2025–2026):

  • Mac Studio 64 GB: ~$2,000–3,000. Runs Llama 3.3 70B at 4-bit. Zero inference cost after purchase. Handles document processing, code assistance, summarization at a pace suitable for individual contributor or small team use.
  • Dual RTX 5090 workstation: ~$5,000–7,000 total. Matches H100 performance on 70B models at ~25% of H100 cost.
  • Dual A100-80GB server: ~$30,000. Needed only for batch inference at organizational scale.

API cost comparison:

  • GPT-4o: $2.50/M input tokens, $10/M output tokens (May 2026 pricing)
  • A team processing 1 million tokens/day in GPT-4o output costs ~$10/day = $3,650/year
  • At that volume, a Mac Studio 64 GB (at $2,500) pays back in under 9 months
  • At 10 million tokens/day output, payback in under 1 month

Hidden costs of on-premise (honest accounting):

  • MLOps engineer for production deployments: ~$135,000/year US salary
  • This changes the math significantly for small teams without existing ML infrastructure
  • Individual contributor / team deployment (Ollama or LM Studio, no MLOps) has near-zero operational overhead
  • The governance value (zero egress, no vendor agreements) is real even if cost savings alone don’t justify dedicated MLOps headcount

Source: arxiv 2509.18101 framework; ptolemay.com LLM TCO 2025; practitioner cost estimates from blog.premai.io and cloudidr.com LLM pricing comparison 2026.

Case Studies

First-party empirical: Chandra OCR 2 on Mac Studio 64 GB (May 2026)

Model: Qwen3.5-VL, 4-bit quantization, 5.8 GB download, 6.7 GB peak memory Hardware: Mac Studio 64 GB Task: Structured document extraction — tables, charts, multi-column layouts from business documents Throughput: 4–9 seconds per page API cost: Zero. Zero egress. Zero vendor agreements. Comparison baseline: pymupdf (traditional CV-based extraction) Result: Outperformed pymupdf on tables and charts. pymupdf flattens complex multi-column layouts and embedded charts into undifferentiated text strings; Qwen3.5-VL preserves table structure and extracts chart data as structured output. Governance implication: The documents processed (business/financial) never left the machine. No DPA, no security review, no IT procurement cycle. The capability was available in under 10 minutes from model download to production output.

This is not an isolated proof-of-concept. It is a production workflow that would have required a cloud API subscription, data processing agreement, and privacy review if run on a frontier model. The local model eliminated all three requirements.

European Healthcare Provider (documented case, practitioner sources)

Running on-premise SLMs to extract structured insights from patient records. No PHI leaves the secured environment. HIPAA and GDPR compliance by architecture, not by contract. The hospital did not need to negotiate a BAA with an AI vendor — the model is their hardware.

Source: Multiple practitioner sources (digitalapplied.com, intellidbenterprise.com). Credibility: LOW-MEDIUM. Not named. Architecture is consistent with regulatory requirements.

Defense-Adjacent Engineering

ITAR-controlled organizations running code-assistance and documentation models on air-gapped workstations. No cloud API can serve this use case regardless of price. The use case exists because there is no alternative — local inference is the only architecturally compliant path.

Workshop Participants (first-party observation, May 2026)

Across a workshop cohort of senior enterprise practitioners, multiple participants reported running Ollama or LM Studio on personal or work hardware for production tasks. The consistent reported driver was not cost — it was governance simplicity and egress control. The procurement overhead for cloud AI tools (DPAs, IT review, security approval) made local models the de facto choice for individual contributors who needed to move quickly.

What This Means

The local tiny model trend is not a hobbyist curiosity. The ecosystem maturity (52M Ollama monthly downloads, production-quality 4-bit quantization with <2% accuracy loss on many tasks, Mac Studio 64 GB running 70B models), combined with the governance forcing functions (GDPR, HIPAA, ITAR, privilege), means that a material and growing share of enterprise AI inference is already happening outside cloud APIs — without appearing in vendor revenue figures, IT procurement records, or CIO dashboards.

For enterprise AI strategy, the implication is a governance gap: teams are running production AI on local hardware precisely because the enterprise procurement process is too slow. The appropriate response is not to restrict local inference (which defeats the governance benefit) but to create a sanctioned local inference pathway — approved model list, Ollama or LM Studio Enterprise with SSO, model gating through IT — that captures the egress-control benefit while maintaining the audit trail the enterprise requires.

Key Data Points

Finding Value Source Date Tier
Ollama monthly downloads 52 million Ollama / community tracking 2025 MEDIUM
Ollama GitHub stars 154,856 Community tracking (Nov 2025) Nov 2025 MEDIUM
Ollama enterprise YoY growth 200% Community reporting 2025 LOW-MEDIUM
LM Studio ARR $1.8M, 16-person team Latka database Jun 2025 MEDIUM
Sovereign AI strategic importance 83% (43% very/extremely important) Deloitte n=3,235 Aug–Sep 2025 MEDIUM-HIGH
Vendor country-of-origin criteria 77% factor this into selection Deloitte n=3,235 Aug–Sep 2025 MEDIUM-HIGH
European enterprise geopatriation projection >75% by 2030 (from <5% today) Gartner Predicts 2026 Oct 2025 MEDIUM-HIGH
AI infrastructure outside public cloud 71% ESG 2025 2025 LOW-MEDIUM
Extract-0 7B vs GPT-4.1 on doc extraction 0.573 vs 0.457 mean reward (n=1,000 tasks) Arxiv 2509.22906 Sep 2025 MEDIUM-HIGH
Extract-0 training cost $196 total Arxiv 2509.22906 Sep 2025 MEDIUM-HIGH
Qwen3-Coder SWE-Bench Verified 69.6% (top open-source) Alibaba Qwen technical report 2025 HIGH (vendor)
Qwen2.5 4-bit accuracy retention 95–98% on GSM8K and IFEval Picovoice / community benchmarks 2025 MEDIUM
4-bit MMLU accuracy delta ±1% vs full precision (well-quantized models) Latitude.so / ionio.ai 2025 MEDIUM
Llama 3.1 8B coding at 4-bit 98.9% accuracy Community benchmarks 2025 MEDIUM
Ollama MLX decode speedup 93% faster; 112 tok/s decode, 1,810 tok/s prefill Ollama blog Mar 30, 2026 HIGH
Mac Mini M4 16 GB throughput 33 tok/s on 7B 4-bit localaimaster.com benchmarks 2025 MEDIUM
RTX 5090 throughput ~213 tok/s on 8B 4-bit compute-market.com 2025–2026 MEDIUM
RTX 5000 PRO throughput 80 tok/s at Qwen3 27B / 200K context startupfortune.com 2025 MEDIUM
On-premise break-even ~2M tokens/day at 70% GPU utilization vs GPT-4 Practitioner analysis / arxiv 2509.18101 2025 MEDIUM
API cost savings potential 50–90% for right-sized on-premise deployments Multiple practitioner sources 2025 LOW-MEDIUM
Local LLM ecosystem: model count 695 models, 110 runtime frameworks Vibe Data (Nov 2025) Nov 2025 MEDIUM
Chandra OCR 2: throughput 4–9 sec/page First-party empirical (Mac Studio 64 GB) May 2026 HIGH
Chandra OCR 2: memory footprint 6.7 GB peak (5.8 GB model) First-party empirical May 2026 HIGH
Chandra OCR 2: vs pymupdf Outperformed on tables and charts First-party empirical May 2026 HIGH

Sources

Ecosystem and adoption:

Governance and compliance:

Model performance:

  • Microsoft Azure Blog. “One Year of Phi: Small Language Models Making Big Leaps in AI” (2025). Phi-4 family benchmarks. https://azure.microsoft.com/en-us/blog/one-year-of-phi-small-language-models-making-big-leaps-in-ai/. Credibility: HIGH for stated benchmarks; vendor-reported.
  • Alibaba Qwen Team. Qwen2.5-VL Technical Report. Arxiv 2502.13923 (Feb 2025). https://arxiv.org/abs/2502.13923. Credibility: HIGH for stated benchmarks; developer-reported.
  • Alibaba Qwen Team. Qwen3 / Qwen3.5 technical documentation. https://qwenlm.github.io/blog/qwen2.5-vl/. Credibility: HIGH for stated benchmarks.
  • Arxiv 2509.22906. “Extract-0: A Specialized Language Model for Document Information Extraction.” Extract-0 7B vs GPT-4.1, o3 on 1,000-task benchmark. Credibility: MEDIUM-HIGH. Peer-reviewed preprint; methodology reviewable.
  • Meta AI. Llama 3 technical documentation. Credibility: HIGH for stated benchmarks; developer-reported.

Hardware and throughput:

Cost analysis:

  • Arxiv 2509.18101. “A Cost-Benefit Analysis of On-Premise Large Language Model Deployment: Breaking Even with Commercial LLM Services” (Aug 2025, revised Nov 2025). https://arxiv.org/abs/2509.18101. Credibility: MEDIUM-HIGH. Peer-reviewed framework; specific numbers in full paper.
  • ptolemay.com. “LLM Total Cost of Ownership 2025: Build vs Buy Math.” Credibility: LOW-MEDIUM. Practitioner analysis.

Quantization benchmarks:

Cross-references within this repository:

Brandon Sneider | brandon@brandonsneider.com May 2026