← Benchmarks 🕐 20 min read

Benchmarks

Local Model Benchmarks: A CIO's Hardware and Quality Reference Guide

Brandon Sneider · May 2026

> Hardware throughput benchmarks (llama.cpp GitHub discussions, MLX Apple Silicon benchmark paper arxiv:2510.18921, singhajit.com measured results, llmcheck.net standardized tests) = **HIGH** — all re

See also (wiki): wiki/local-tiny-models.md · wiki/inference-economics.md · wiki/token-economics.md

Source credibility notes: Hardware throughput benchmarks (llama.cpp GitHub discussions, MLX Apple Silicon benchmark paper arxiv:2510.18921, singhajit.com measured results, llmcheck.net standardized tests) = HIGH — all reproducible with public tooling. Model quality benchmarks from vendor technical reports (Qwen, Mistral, Microsoft Phi, Google Gemma) = LOW-MEDIUM — vendor-run, but often corroborated by independent community replication. Independent community comparisons (jdhodges.com, modelpiper.com, morphllm.com, localllm.in) = MEDIUM-HIGH. Enterprise cost modeling (Lenovo LP2368 2026, arxiv:2509.18101) = HIGH for directional claims; vendor-sponsored but uses auditable MLPerf benchmarks and published hardware pricing.

Executive Summary

Local model inference crossed an enterprise-viability threshold in 2025 and accelerated sharply through early 2026. Three converging forces produced that shift:

Model quality per parameter has tripled since mid-2023. A quantized 14B model in May 2026 matches what required 70B parameters in 2024 on most structured enterprise tasks.
Hardware throughput has doubled or tripled on Apple Silicon and NVIDIA consumer GPUs in two years, driven by memory bandwidth improvements and native inference framework optimization (MLX on Apple, TensorRT-LLM on NVIDIA).
The break-even against cloud APIs has compressed from 17 months (2024) to 4 months (2026) for sustained enterprise workloads — see on-premise-llm-tco-break-even.md.

The practical result: a Fortune 500 legal team can run a 32B model on a $3,500 workstation for document review at zero marginal token cost, with no data leaving the building. That was not true 18 months ago.

Four deployment profiles now have workable hardware-model pairings:

Developer workstation / AI PC (RTX 4090, 24GB VRAM): 32B parameter models at 30–45 tok/s — usable for code completion and RAG
Mac Studio / Mac Pro (M4 Max/Ultra, 64–192GB unified memory): 70B models at 12–40 tok/s via MLX — best latency-per-dollar for knowledge worker deployments
8GB VRAM laptop (RTX 4060 / integrated): 9B models at 40–55 tok/s — viable for summarization and classification
On-premise inference server (8x H100/B200): 70B–405B models at 30,000–99,000 tok/s aggregate — full fleet automation

1. Framework Landscape

1.1 llama.cpp

The foundational inference engine. Written in C/C++ with zero dependencies, it runs on every platform (CPU, NVIDIA GPU, AMD GPU, Apple Metal, Vulkan) using the GGUF file format. It is the engine underpinning Ollama and Llamafile.

Source credibility: HIGH. llama.cpp benchmarks are community-run on known hardware and published in the GitHub discussions thread #4167. The testing protocol is consistent: llama-bench with standardized prompt tokens and generation length.

Performance methodology: Token generation speed (tok/s) is memory-bandwidth-limited, not compute-limited. The model’s weights must be read from RAM or VRAM for each token generated. This is why Apple Silicon — with unified memory bandwidth of 120–546 GB/s depending on chip variant — outperforms many discrete GPU setups for models that fit in memory, and why quantization (reducing weight size) has such a large impact on speed.

Key performance data:

Hardware	Model	Quantization	Tok/s	Source
M4 Mac Mini (16GB)	Llama 3.1 8B	Q4_K_M	~33	llmcheck.net
M4 Pro (48GB)	Qwen 3-32B	Q4_K_M	~28	community
M4 Max (128GB)	Llama 3.1 70B	Q4_K_M	~12	llmcheck.net
RTX 3060 12GB	Llama 3.1 8B	Q4_K_M	42	singhajit.com
RTX 4070 12GB	Llama 3.1 8B	Q4_K_M	52	singhajit.com
RTX 4090 24GB	Llama 3.1 8B	Q4_K_M	104	singhajit.com
RTX 4090 24GB	Qwen 2.5-14B	Q4_K_M	69	singhajit.com
RTX 4090 24GB	Qwen 2.5-32B	Q4_K_M	30–45	community
RTX 5090	Llama 3.1 8B	Q4_K_M	213	localllm.in

Source: singhajit.com benchmark, llmcheck.net, localllm.in RTX guide

1.2 MLX (Apple Silicon)

Apple’s machine learning framework, optimized for the Metal GPU and unified memory architecture of M-series chips. At WWDC 2025, Apple made MLX the official preferred inference framework. In March 2026, Ollama announced it would transition its Apple Silicon backend to MLX.

Key advantage over llama.cpp on Apple Silicon: MLX uses the Metal compute shader to keep weights in unified memory and avoids CPU↔GPU copy overhead that llama.cpp incurs via the Metal backend. For models under ~27B parameters, MLX is 20–87% faster than llama.cpp on identical Apple Silicon hardware. Above 27B, both frameworks saturate the available memory bandwidth and the gap narrows. [NEEDS VERIFICATION for the 87% upper bound — community-reported, not independently audited]

Apple Silicon memory bandwidth by chip:

Chip	Memory Bandwidth	Max Unified RAM	Largest Practical Model (Q4)
M1	68 GB/s	16 GB	8B
M2	100 GB/s	24 GB	8B
M2 Pro	200 GB/s	32 GB	14B
M2 Max	400 GB/s	96 GB	70B
M3	100 GB/s	24 GB	8B
M3 Pro	150 GB/s	36 GB	14B
M3 Max	400 GB/s	128 GB	70B
M4	120 GB/s	32 GB	14B
M4 Pro	273 GB/s	48 GB	32B
M4 Max	546 GB/s	128 GB	70B+

Source: modelpiper.com Apple Silicon benchmarks, Apple published chip specs

MLX throughput benchmarks (Q4_K_M, May 2026):

Model	Chip	Tok/s (MLX)	Tok/s (Ollama/llama.cpp)	MLX advantage
Gemma 4 E2B	M5 Max	~158	—	—
Phi-4 Mini 3.8B	M5 Max	~135	—	—
Qwen 3.6-35B-A3B	M5 Max	~55	—	—
Gemma 4 26B-A4B	M5 Max	~50	—	—
Qwen3-Coder-30B-A3B	M4 Pro (64GB)	~130	~43	3×
Qwen 3.5-35B-A3B	M4 Max (128GB)	~130	~43.5	3×
Llama 3.1 8B	M2 Air	~26 tok/s (MLX) vs ~18 tok/s (Ollama)	18	44%

Source: llmcheck.net, yage.ai MLX analysis

1.3 Ollama

The most widely deployed local inference runner: 52 million monthly downloads as of late 2025, 200% YoY enterprise growth. Ollama wraps llama.cpp (transitioning to MLX on Apple Silicon as of March 2026) with a model registry, REST API, and simple CLI. It is not the fastest backend but is the most operationally convenient.

Performance vs raw llama.cpp: Ollama adds approximately 3–10% overhead vs raw llama.cpp on NVIDIA GPUs for single-user inference. On Apple Silicon, the gap is larger because Ollama has not yet migrated all models to the MLX backend — as of May 2026 the MLX preview covers Qwen3.5 models, with broader rollout planned.

Multi-user production note: For multi-user or high-throughput production workloads, Ollama is not the right tool. At peak throughput, vLLM reaches ~793 tok/s versus Ollama’s ~41 tok/s on identical hardware. Ollama is designed for single-developer or low-concurrency use.

Source: codersera.com vLLM vs Ollama production benchmark

1.4 LM Studio

A desktop GUI application (macOS, Windows, Linux) for browsing, downloading, and running GGUF models, with a built-in OpenAI-compatible API server. LM Studio uses MLX natively on Apple Silicon, delivering 26–60% higher throughput than Ollama for the same model and hardware. Primary advantage over Ollama: visual model discovery and side-by-side comparison. LM Studio version 0.4.6 was used for the JD Hodges tool-calling benchmark (see Section 4.2).

1.5 Llamafile

Mozilla’s single-file executable format — bundles model weights and inference runtime into one portable binary. Llamafile 0.10.0 (March 2026) re-added CUDA support, making it viable beyond CPU-only deployments. Community benchmarks report 3–4× higher throughput than Ollama and 30–40% lower power consumption under low-core operation. [Source credibility: MEDIUM — these figures are community-reported against Ollama’s prior llama.cpp backend, and the gap may narrow as Ollama migrates to MLX.] Primary use case is enterprise deployment without IT infrastructure — a single file that runs on any compatible machine.

2. VRAM Tiers and Model Fit (NVIDIA)

The single most important hardware constraint for local inference is VRAM capacity. At Q4_K_M quantization, parameter counts map to VRAM as follows (approximate, varies by architecture):

Model Size	VRAM Required (Q4_K_M)	Fits In
3–4B	2–3 GB	Any modern GPU; integrated graphics
7–9B	4–6 GB	RTX 4060 (8GB), GTX 1070
14B	8–10 GB	RTX 4070 (12GB), RTX 3060 (12GB)
30–32B	18–20 GB	RTX 4090 (24GB), RTX 3090 (24GB)
70B	35–40 GB	2× RTX 4090, or A100 40GB (tight)
70B	40–48 GB	H100 80GB (comfortable), RTX 5090 (32GB) [NEEDS VERIFICATION]

Source: localllm.in VRAM guide, localllm.in 8GB guide

MoE efficiency note: Mixture-of-Experts architectures (Qwen3-30B-A3B, Qwen3.6-35B-A3B, Gemma 4 26B-A4B) activate only a fraction of parameters per token — typically 3–4B of a 30–35B total. This means a 30B MoE model has VRAM requirements closer to a 17–18B dense model at Q4 quantization, while generating tokens 3–5× faster than a dense 30B would. The Qwen3-30B-A3B fits in approximately 17GB VRAM, making it viable on a single RTX 4090 (24GB) with room for context.

8GB VRAM sweet spot (2026): Qwen3.5-9B at Q4_K_M is the community consensus best-for-8GB model, achieving 55+ tok/s on an RTX 4060. For coding tasks specifically, Qwen2.5-Coder-7B is preferred. Both fit within the 8GB envelope.

Context length tax: Doubling the context window reduces throughput by 10–20%. An RTX 3060 12GB running an 8B model at Q4_K_M achieves 42 tok/s at standard context but drops to ~32 tok/s at 16K context.

3. Small Model Quality Benchmarks

3.1 The Sub-7B Tier

The sub-7B tier has improved faster than any other model class. Three models punch significantly above their parameter weight in 2026:

Phi-4 Mini (3.8B, Microsoft) The clearest outlier at this parameter count. On MATH (competition mathematics), Phi-4 Mini scores 62% — versus Llama 3.2 3B at 48%. On MMLU, it scores 73% versus Llama 3.2 3B’s 65%. The training approach — synthetic data curation and high-quality textbook-style corpora — transfers unusually well to small parameter counts. Natively supports 128K token context, enabling document-length RAG inputs on minimal hardware.

Source: localaimaster.com Phi-4 mini, meta-intelligence.tech SLM comparison

Qwen3 8B (Alibaba / Qwen) Leads the sub-8B class on HumanEval (code generation) at 76.0 — a 3.4-point margin over Llama 3.3 at the same scale. Strong multilingual capability across CJK languages and European languages. The 8B model is the practical ceiling for most 8GB VRAM laptops.

SmolLM3 3B (HuggingFace) Strong performance at 3B that outperforms Llama 3.2 3B and Qwen 2.5 3B on 12 benchmarks. Relevant for browser-based or mobile deployments where even 4GB is constrained.

Source: bentoml.com SLM guide 2026

3.2 The 14B–32B Tier (Workstation Range)

Phi-4 (14B, Microsoft) The flagship: surpasses many 70B-class models on math reasoning and code generation despite 14B parameters. MATH score: 80.4% vs Llama 3.3 8B’s 68.0% and Qwen 2.5 14B’s 75.6%. Fits on an RTX 4080 (16GB) at Q4. Inference speed on RTX 3060: ~23 tok/s; on RTX 4090: ~69 tok/s.

Source: callsphere.ai SLM benchmarks

Qwen 2.5-32B / Qwen3-32B (Alibaba) The practical ceiling for a single RTX 4090 (24GB VRAM). On the RTX 4090, 32B models run at 30–45 tok/s — usable for interactive coding assistance. Qwen 2.5 Coder 32B scores 92.7% on HumanEval, making it competitive with closed-source coding models available 18 months ago.

Mistral Small 3.1 (24B) 150 tok/s reported on RTX 4090 (vendor-reported; treat as upper bound). Beats GPT-4o Mini on MMLU, HumanEval, and LongBench v2 per Mistral’s own testing. Multilingual strength in European and East Asian languages. 128K context window. Requires a 24GB GPU (RTX 4090 or RTX 3090) at Q4 for comfortable inference; an M4 Pro (48GB unified) runs it comfortably.

Source: mistral.ai Mistral Small 3.1 release, openrouter.ai Mistral specs

Gemma 3 27B (Google) 140+ language support; multimodal (text + image). Fits on a single 48–80GB GPU. Scores 10 on the Artificial Analysis Intelligence Index, below median for open-weight non-reasoning models of similar size, suggesting it trades general benchmark performance for specific strengths (multilingual breadth, vision). Best choice when the use case is non-English documents with mixed-media content.

Source: artificialanalysis.ai Gemma 3 27B, Gemma 3 technical report

3.3 The MoE Wildcard

Qwen3-30B-A3B / Qwen3.6-35B-A3B (Alibaba) The most consequential architectural change for enterprise local deployment in 2026. These Mixture-of-Experts models have 30–35B total parameters but activate only 3B per token. The result:

VRAM requirements of a ~17B dense model
Token generation speed of a ~3–7B dense model
Benchmark quality approaching 30B dense models on most tasks
On RTX 4090: ~120 tok/s at Q4 (community-reported for 35B-A3B variant)
On M4 Pro (64GB) via MLX: ~130 tok/s

For enterprise deployments where hardware is constrained to a single RTX 4090 or an M4 Pro Mac, the Qwen3 MoE family is the strongest all-around choice as of May 2026.

Source: Qwen3 blog post, github Qwen3 MoE benchmark gist, codersera.com Qwen 3.6 guide

4. Task-Specific Benchmarks

4.1 Coding

HumanEval (single-function completion) is being displaced by SWE-bench (real GitHub issue resolution) as the primary coding benchmark, since SWE-bench tests multi-file reasoning, test-passing, and PR-quality output — skills that matter for actual software development workflows.

Local coding model rankings (May 2026):

Model	HumanEval	Notes
Qwen2.5-Coder 32B	92.7%	Best locally-runnable; needs RTX 4090
Qwen 3 8B	76.0%	Best under 8B; fits 8GB VRAM
Phi-4 Mini 3.8B	~62% (coding)	Best at 3–4B range
Qwen2.5-Coder 14B	~89.9%	Strong on 12GB VRAM

Source: toolhalla.ai Qwen2.5-Coder guide, morphllm.com coding benchmarks 2026

For code completion specifically (autocomplete, infilling, cross-file context), Qwen2.5-Coder achieves SOTA on CrossCodeEval, CrossCodeLongEval, RepoEval, and SAFIM — the benchmarks closest to actual IDE usage patterns.

Enterprise implication: For engineering teams where source code cannot leave the building (defense contractors, financial services with proprietary trading logic, life sciences with pre-patent chemistry), Qwen2.5-Coder 32B on a single RTX 4090 workstation is the practical recommendation as of May 2026.

4.2 Tool Calling / Function Calling

Tool calling is the capability that unlocks agentic workflows: a model parsing a JSON schema, selecting the right tool, populating arguments correctly, and handling edge cases (missing arguments, choosing not to call). This is where quantization sensitivity is highest and where small-model performance diverges most sharply from large-model performance.

Independent benchmark (JD Hodges, May 2026): 13 local LLMs tested at Q4_K_M, 40 test cases across 5 categories (tool selection, argument accuracy, multi-tool calling, edge cases, format compliance). Hardware: AMD AI Max+ 395, 128GB unified RAM. Inference: LM Studio v0.4.6.

Rank	Model	Size (disk)	Pass Rate (40 cases)
1	Qwen3.5 4B	3.4 GB	97.5%
2	GLM-4.7-Flash	18 GB	95.0%
2	Nemotron Nano 4B	4.2 GB	95.0%
4	Mistral Nemo 12B	7.5 GB	92.5%

Source: jdhodges.com tool calling benchmark 2026

Key finding: Qwen3.5 4B at 97.5% accuracy is the most striking result — a 3.4GB model outperforming 18GB models on tool calling. This reflects training methodology more than raw parameter count. The community consensus for multi-step agent workflows requiring 2+ tool calls in sequence is larger models (Qwen 2.5 14B or Mistral Small 24B), as smaller models lose coherence after 2–3 steps. [NEEDS VERIFICATION — no published multi-step benchmarks found; community-reported.]

Quantization sensitivity for tool calling: A @stevibe community benchmark on Qwen3 found Q6 quantization achieved 15/15 correct tool calls while Q2 dropped to 13/15 — a meaningful degradation that would surface as 13% error rate in production. Tool calling is among the most quantization-sensitive tasks because JSON format compliance fails in an all-or-nothing way: a malformed closing bracket breaks the entire tool call. For agentic workflows, Q4_K_M is the practical minimum; Q6_K is preferred where VRAM allows.

Source: Community-reported (not independently verified); consistent with general quantization theory on structured output tasks.

4.3 RAG (Retrieval-Augmented Generation)

The most important finding for enterprise RAG deployments: local LLMs are more likely than frontier cloud models to hallucinate when used as RAG generators, particularly on domain-specific retrieval where context is dense and factual precision is required. Training a local RAG model on a specific domain (fine-tuning) substantially closes this gap.

Practical guidance for local RAG:

Minimum viable model for RAG: 14B parameters at Q4. Models below this threshold show meaningful hallucination rates on dense retrieval contexts.
Context window matters more than parameter count for RAG quality. Phi-4 Mini’s 128K context window makes it viable for long-document RAG at minimal VRAM cost.
Preferred local RAG stack (May 2026): Qwen2.5-14B or Mistral Small 24B as generator + any competent embedding model (nomic-embed-text, mxbai-embed-large) + llama.cpp or Ollama as inference server.

Source: ncbi.nlm.nih.gov RAG for radiology, pricepertoken.com RAG leaderboard

4.4 Multilingual

Local multilingual capability is dominated by two model families:

Qwen 3 series: Strongest CJK (Chinese, Japanese, Korean) capability at any parameter count in the open-weight ecosystem. Also strong on Arabic, Russian, and major European languages. Qwen3 7B leads sub-8B for multilingual tasks.

Mistral Small 3.1 (24B): Strongest European-language performance outside CJK. Outperforms GPT-4o Mini on multilingual benchmarks per Mistral’s testing (vendor-reported).

Gemma 3 27B: 140+ language coverage, including lower-resource languages not well-represented in Qwen or Mistral training data. The right choice for Southeast Asian, African, or South Asian language requirements.

For enterprises with significant non-English document volume — particularly APAC operations handling Chinese, Japanese, or Korean — local deployment of a Qwen3 model eliminates both the API cost and the data sovereignty risk of routing those documents to US-based cloud providers.

Source: benchlm.ai multilingual leaderboards, sitepoint.com best local LLM models 2026

5. The Quantization Benchmarking Problem

Quantization reduces the numerical precision of model weights to shrink file size and accelerate inference. The tradeoffs are not linear across tasks or quantization levels.

5.1 Quality Degradation by Level

Level	Size Reduction vs FP16	Perplexity Degradation	Benchmark Impact	Structured Output
Q8_0	~50%	<1%	Statistically indistinguishable from FP16	Negligible impact
Q6_K	~60%	~2%	Imperceptible on most tasks	Minimal impact
Q4_K_M	~75%	1–3%	5–15% on coding/reasoning benchmarks	Noticeable on complex tool calls
Q3_K_M	~80%	3–7%	Visible in complex reasoning	Significant; not recommended for agents
Q2_K	~85%	>10%	Substantial; model noticeably weaker	13% error rate observed (tool calling)

Source: willitrunai.com quantization guide, bmdpat.com GGUF quantization 2026

Task sensitivity hierarchy (most to least sensitive to quantization):

Structured output / tool calling / JSON schema compliance
Multi-step mathematical reasoning
Coding (especially at 15+ line function level)
Factual question answering
Summarization, classification, document extraction

Practical recommendation: Q4_K_M is the floor for production enterprise use. Q6_K is the preferred default when VRAM allows. The 25–30% speed advantage of Q4 over Q6 rarely justifies the quality tradeoff for business-critical tasks.

5.2 Format Comparison: GGUF vs GPTQ vs AWQ

These are three different approaches to storing and loading quantized models. The choice matters primarily for NVIDIA GPU deployments.

Format	Best Use Case	Throughput (Qwen2.5-32B, vLLM)	Quality Retention	Platform
AWQ (Marlin kernels)	NVIDIA GPU production	741 tok/s	~95%	NVIDIA only
GPTQ (Marlin kernels)	NVIDIA GPU production	712 tok/s	~90%	NVIDIA only
FP16 (unquantized)	Maximum quality	461 tok/s	100%	NVIDIA/AMD
GGUF Q4_K_M	CPU, Apple, mixed	93 tok/s (vLLM context)	~92%	Universal
bitsandbytes	Fine-tuning	168 tok/s	~88%	NVIDIA only

Critical caveat: AWQ’s 741 tok/s advantage over GGUF’s 93 tok/s applies only in the vLLM multi-user serving context with Marlin kernels on NVIDIA. On a single-user Ollama deployment or Apple Silicon, AWQ’s advantage collapses — GGUF remains the universal format and is the correct choice for developer workstations and mixed-hardware enterprise fleets.

Source: localaimaster.com quantization comparison, bestaiweb.ai quantization formats

6. AMD ROCm for Local Inference

AMD’s ROCm software stack has matured substantially in 2025–2026, but a gap remains versus NVIDIA’s CUDA ecosystem.

Consumer GPU performance: AMD RX 7900 XTX achieves approximately 80% of RTX 4090 throughput on Q4 inference tasks, at roughly 50% of the price. For organizations that cannot purchase NVIDIA hardware (supply constraints, procurement policy), AMD consumer GPUs are a viable path for local inference up to 30B parameter models.

Enterprise GPU (MI300X, MI355X): AMD’s data-center accelerators reached 90–95% of H100 throughput on standard LLM inference workloads (PyTorch + vLLM), and MI355X results at MLPerf Inference 6.0 (April 2026) were within single-digit percentage points of NVIDIA B200 on server inference. AMD MI300X on-demand pricing runs 15–30% below H100 SXM5 comparable.

The software gap: Framework support has improved — PyTorch, TensorFlow, JAX, vLLM, llama.cpp, and SGLang all have official ROCm backends. The remaining gap is NVIDIA-specific libraries: TensorRT-LLM and FlashAttention 3 have no full ROCm equivalents, meaning the highest-efficiency NVIDIA inference kernels are not available on AMD hardware. For organizations already running vLLM, the migration to ROCm is operationally straightforward. For TensorRT-LLM deployments, the migration is non-trivial.

Source: spheron.network ROCm vs CUDA 2026, craftrigs.com AMD ROCm 2026

7. CPU-Only Inference

CPU-only inference is viable for specific enterprise use cases — primarily scheduled batch processing, overnight summarization, and air-gapped environments where no GPU is available.

Practical ceiling (modern workstation):

8-core laptop CPU: 10–20 tok/s on 7–8B models at Q4 (usable but noticeable latency)
High-end Threadripper (tuned llama.cpp): 92+ tok/s on small models in optimized configurations
General pattern: 4–5B models at 20+ tok/s is the CPU sweet spot for interactive use

Where CPU inference works:

Overnight batch classification or extraction jobs where latency is irrelevant
Air-gapped environments where no GPU procurement is possible
Embedding generation (not generation tasks) — CPU is competitive with GPU for embedding workloads on models under 1B parameters
Scheduling-tolerant RAG pipelines where quality matters more than throughput

Where it does not work:

Interactive chat or code completion (latency exceeds user tolerance for models above 7B)
Agentic workflows with multiple sequential model calls (total elapsed time becomes prohibitive)

The critical constraint is memory bandwidth: consumer CPUs access DDR5 RAM at 50–100 GB/s, versus 120–546 GB/s for Apple Silicon unified memory. A modern Mac Mini (M4, 120 GB/s) runs 8B models faster than a CPU-only workstation with a more powerful processor.

Source: thedroptimes.com CPU inference benchmark, arxiv.org CPU vs GPU local LLM

8. Enterprise Deployment Economics

8.1 The Break-Even Threshold

The economics of local vs. cloud LLM inference have compressed dramatically. Full analysis: on-premise-llm-tco-break-even.md.

2026 break-even benchmarks (Lenovo LP2368, enterprise-grade hardware):

Comparison	Break-Even Timeline	Assumption
vs. cloud on-demand	3.7 months	8x H100 configuration, sustained use
vs. 1-year reserved	6 months	Same hardware
vs. 3-year reserved	9.3 months	Same hardware
vs. MaaS APIs (GPT-4 tier)	<3 months	70B model, high utilization

Cost per million tokens (sustained utilization):

Scenario	$/1M tokens
Self-hosted 70B (8x H100)	$0.11
Azure H100 on-demand	$0.89
Frontier API (GPT-5 mini tier)	~$2.00
Self-hosted 405B (8x B300)	$4.74
AWS equivalent for 405B	$29.09

The 18× cost advantage of self-hosted 70B vs. frontier MaaS APIs is the most operationally relevant figure for Fortune 500 legal, HR, and finance teams where agentic workflows are consuming millions of tokens daily.

Source: Lenovo LP2368 On-Premise vs Cloud TCO 2026

8.2 Utilization Reality Check

The break-even calculus depends entirely on GPU utilization. Most production LLM inference teams operate at 40–65% GPU utilization due to traffic variability and batching constraints. The Lenovo analysis assumes ≥20% utilization (~4 hours daily) as the threshold for economic justification.

The agentic multiplier: Conversational AI generates roughly 50K tokens per user per day. An agentic coding assistant generates approximately 1M tokens per developer per day. A 10-developer engineering team using agentic coding tools likely crosses the break-even threshold within weeks, not months — this is why the economics are accelerating for software-intensive enterprises.

Source: wiki/local-tiny-models.md

8.3 Privacy as a Non-Cost Driver

Cost is not always the primary driver. A Deloitte survey found 55% of enterprises avoid at least some AI use cases due to data security concerns; an IBM survey found 57% cite data privacy as the biggest inhibitor to AI adoption. For these organizations, local inference eliminates the driver regardless of cost comparison:

HIPAA workloads: No BAA required for local inference
ITAR environments: Air-gap requirements satisfied by definition
GDPR Article 46: No third-country transfer when inference is local
Attorney-client privilege: No third-party data processor in the chain
Agentic workflows on proprietary code: No source code leaves the building

Source: signitysolutions.com on-premise vs cloud LLM, unifiedaihub.com on-prem LLMs

9. Key Data Points Summary

Metric	Value	Source	Credibility
Fastest local model on Apple Silicon (May 2026)	Gemma 4 E2B ~158 tok/s on M5 Max (MLX)	llmcheck.net	MEDIUM-HIGH
RTX 4090 throughput, 8B model	104 tok/s (Q4_K_M)	singhajit.com measured	HIGH
RTX 4090 throughput, 32B model	30–45 tok/s (Q4_K_M)	community	MEDIUM
MoE speed advantage over dense	3–5× faster for equivalent benchmark quality	Qwen community benchmarks	MEDIUM
MLX advantage over llama.cpp on Apple Silicon (<27B)	20–87%	community-reported	MEDIUM
Qwen3.5 4B tool-calling accuracy	97.5% (40 test cases, Q4_K_M)	jdhodges.com	HIGH
Q6 vs Q2 tool-calling degradation (Qwen3)	15/15 vs 13/15	@stevibe community benchmark	MEDIUM — not independently audited
Q4_K_M quality retention vs FP16	~92% on most tasks	willitrunai.com	MEDIUM-HIGH
AWQ+Marlin throughput advantage vs GGUF	741 vs 93 tok/s (vLLM context)	bestaiweb.ai	MEDIUM-HIGH
On-prem break-even vs cloud on-demand (enterprise)	3.7 months	Lenovo LP2368 2026 (vendor)	MEDIUM-HIGH
Cost/1M tokens: self-hosted 70B vs MaaS	$0.11 vs ~$2.00 (18× advantage)	Lenovo LP2368 2026 (vendor)	MEDIUM-HIGH
Enterprises avoiding AI use cases (data security)	55%	Deloitte n=3,235	HIGH
Qwen2.5-Coder 32B HumanEval	92.7%	vendor + community	MEDIUM-HIGH
Phi-4 (14B) MATH score	80.4%	Microsoft/community	MEDIUM-HIGH

10. What This Means for Your Organization

Which models are viable for which enterprise use cases

Coding assistance on proprietary codebase (data cannot leave building): → Single workstation: Qwen2.5-Coder 32B on RTX 4090 (24GB VRAM, ~35 tok/s). Best-in-class local coding model. → Team deployment: Qwen3-30B-A3B on RTX 4090 (MoE efficiency; ~120 tok/s; fits 17–18GB VRAM).

Document processing / summarization / classification (regulated data): → 8B parameter class (Qwen3 8B, Llama 3.3 8B) handles structured extraction well at Q4 on 8GB VRAM. → For long documents (legal, clinical), Phi-4 Mini’s 128K context window on an 8GB GPU is the most practical sub-10B option.

Multilingual document workflows (APAC or multi-regional operations): → Qwen3 series for CJK + Arabic + European languages. → Gemma 3 27B for breadth (140+ languages), including lower-resource ones. → Mistral Small 3.1 for European-language strength with smaller hardware footprint than Gemma 3 27B.

Agentic workflows with tool calling: → Do not run below Q4_K_M. Prefer Q6_K where VRAM allows. → For simple single-tool workflows: Qwen3.5 4B at Q4 achieves 97.5% accuracy. → For multi-step agentic pipelines: Qwen3-30B-A3B or Mistral Small 3.1 24B.

High-volume production inference (multi-user, thousands of requests/day): → Ollama is not sufficient. Use vLLM with AWQ quantization on NVIDIA, or SGLang. → At this scale, on-premise NVIDIA hardware (H100 or L40S) is economically superior to cloud within 4 months at ≥20% daily utilization.

How to evaluate local models for your environment

Define the task before selecting the model. Summarization, coding, tool calling, and RAG have different failure modes and different sensitivity to quantization. A model that benchmarks well on MMLU may perform poorly on your specific document extraction task.
Test at the quantization level you will deploy. Q4 and Q8 can differ by 5–15% on reasoning tasks. Vendor-reported benchmark figures are typically at FP16 or Q8. Your production deployment will likely run Q4.
Benchmark latency at your expected context length. A model that runs 42 tok/s at 2K context may run 32 tok/s at 16K context. If your RAG pipeline returns 10K-token context windows, benchmark at that length.
Measure on your data. HumanEval does not predict performance on your internal codebase. MMLU does not predict performance on your contracts. Build a 50-item evaluation set from your actual documents before committing to a model.
Run a cost model before a procurement decision. The break-even math changes dramatically based on utilization. An organization routing 500K tokens/day through cloud APIs is in a different calculus than one routing 50M tokens/day.