← Local Tiny Models 🕐 25 min read
Local Tiny Models

Enterprise Workstations and Lightweight LLM Inference Servers

Brandon Sneider · May 2026

> **Source credibility: MEDIUM** — GPU specs and OEM workstation configs are drawn from official OEM and NVIDIA/AMD announcements (TIER 1–2).

See also (wiki): wiki/local-tiny-models.md · wiki/ai-implementation-cost-structure.md · wiki/inference-economics.md

Source credibility: MEDIUM — GPU specs and OEM workstation configs are drawn from official OEM and NVIDIA/AMD announcements (TIER 1–2). Inference server performance benchmarks are from community testing and published benchmarks (TIER 2–3). Treat benchmark numbers as directional, not certified. Concurrency claims reflect community consensus and published comparisons as of May 2026.

TIER 1–2 — OEM/NVIDIA/AMD vendor announcements and community benchmarks as of May 2026; GPU specs subject to revision.

Executive Summary

The enterprise workstation GPU market underwent a significant transition in Q1 2026. NVIDIA’s RTX Pro 6000 Blackwell (96 GB GDDR7, TIER 1 vendor announcement) became available in OEM workstations from Dell, HP, and Lenovo — tripling the single-GPU VRAM ceiling from 48 GB (RTX 6000 Ada) to 96 GB. This means a standard enterprise tower workstation can now hold Llama 3.1 405B at Q2 quantization or run 70B models in FP16 with headroom for multiple concurrent requests.

Simultaneously, the lightweight inference server ecosystem matured. Ollama, llama.cpp/llama-server, TabbyAPI, Aphrodite, and LocalAI collectively cover the full deployment envelope from “one developer in 10 minutes” to “20-user team serving quantized models at production throughput” — without Kubernetes, vLLM, or dedicated MLOps headcount.

The practical implication: the infrastructure question for local SLM deployment has shifted from “is this feasible?” to “which workstation + which inference server for which team size?”

Part 1: GPU Landscape for Enterprise Workstations

NVIDIA Professional Desktop GPUs (Current, Non-Data-Center)

GPU Architecture VRAM Bandwidth TDP MSRP Ships in
RTX Pro 6000 Blackwell (600W) Blackwell GB202 96 GB GDDR7 ECC 1,792 GB/s 600 W ~$8,500 Dell T4/T6, HP Z4/Z8, Lenovo P8
RTX Pro 6000 Blackwell Max-Q (300W) Blackwell GB202 96 GB GDDR7 ECC 1,792 GB/s 300 W OEM-only Dell T4, HP Z4, Lenovo P5/P7/P8
RTX 6000 Ada Generation Ada Lovelace 48 GB GDDR6 ECC 960 GB/s 300 W ~$6,800 All P/Z-series
RTX 5000 Ada Generation Ada Lovelace 32 GB GDDR6 ECC 576 GB/s 250 W ~$4,000 All P/Z-series
RTX 4500 Ada Generation Ada Lovelace 24 GB GDDR6 ECC 432 GB/s 210 W ~$2,250 All P/Z-series
RTX 4000 Ada Generation Ada Lovelace 20 GB GDDR6 ECC 360 GB/s 130 W — Mid-range P/Z

Key notes:

  • The RTX Pro 6000 Blackwell 600W (full-power) sets a new single-slot ceiling: 96 GB GDDR7 with 1,792 GB/s memory bandwidth — nearly 2× the Ada generation. This is the card for running Llama 3.1 405B at quantized precision or 70B at FP16.
  • The 300W Max-Q variant delivers the same 96 GB but fits in mid-tower systems with standard power supplies (Dell T4, HP Z4, Lenovo P3/P5).
  • The RTX 6000 Ada (48 GB) is now the mid-tier workstation card. Widely available and the realistic “available in stock today” choice for most enterprise procurement cycles.
  • The RTX 5000 Ada (32 GB) is the practical sweet spot: Llama 3 70B at Q4_K_M requires ~28 GB VRAM — fits with headroom on a $4K GPU.
  • Consumer cards (RTX 4090/5090) appear in some workstations (Dell Precision lower tiers, ASUS ProArt laptops) but lack ECC memory and professional driver stack. Not recommended for unattended production inference. The RTX 4090’s 24 GB ceiling cannot fit 70B Q4_K_M (~38 GB needed).

AMD Professional Desktop GPUs

GPU Architecture VRAM Bandwidth TDP MSRP Ships in
Radeon PRO W7900 RDNA 3 48 GB GDDR6 ECC 864 GB/s 295 W ~$3,999 HP Z4/Z6/Z8, Dell Precision
Radeon PRO W7800 RDNA 3 32 GB GDDR6 ECC 576 GB/s 260 W ~$2,499 HP Z4/Z6, Dell Precision

Key notes:

  • The W7900 at $3,999 for 48 GB is significantly cheaper than the RTX 6000 Ada at $6,800 for the same VRAM count. AMD’s pitch: within 7% of RTX 6000 Ada performance at less than half the cost.
  • Critical constraint: ROCm (AMD’s GPU compute stack for inference) is Linux-only in production. Windows AMD GPU acceleration for inference tools (Ollama, llama.cpp ROCm backend) is not production-ready as of May 2026. Workarounds: Vulkan backend (lower performance, Windows-compatible) or WSL2 with ROCm. See inference server section for per-tool details.
  • HP supports up to two AMD Radeon PRO W7900 in dual-GPU Z8 configurations (96 GB aggregate).

VRAM Requirements for Reference Models

Model Precision VRAM Required Minimum GPU
Llama 3.2 3B Q4_K_M (GGUF) ~2.5 GB Any modern GPU
Phi-4 14B Q4_K_M ~8 GB RTX 4000 Ada (20 GB)
Llama 3 8B FP16 ~16 GB RTX 5000 Ada (32 GB)
Llama 3 70B Q4_K_M ~38 GB RTX 6000 Ada (48 GB)
Llama 3 70B FP16 ~140 GB 2× RTX Pro 6000 Blackwell
Llama 3.1 405B Q2_K ~92 GB 1× RTX Pro 6000 Blackwell (96 GB)
Llama 3.1 405B Q4_K_M ~215 GB 3× RTX Pro 6000 Blackwell Max-Q

Part 2: Enterprise Workstations by OEM

Dell

Dell Pro Precision 9 Series (Announced March 2026, Shipping May 2026)

Dell rebranded its workstation line. The old “Precision” series became “Pro Precision 9.” Three tower configurations:

Dell Pro Precision 9 T2

  • Intel Xeon 600-series (Granite Rapids), entry core-count configs
  • GPU: up to mid-range NVIDIA RTX Pro (300W Max-Q class; exact ceiling not separately published)
  • Target: entry-level workstation for moderate inference workloads (sub-30B models)

Dell Pro Precision 9 T4

  • Mid-tower, 6 PCIe slots
  • CPU: 12–86 core Intel Xeon 600 options
  • GPU: 1× 600W GPU (RTX Pro 6000 Blackwell, 96 GB) OR 2× 300W GPUs (2× RTX Pro 6000 Blackwell Max-Q = 192 GB aggregate) OR 4× 50W GPUs
  • Memory: up to 4 TB DDR5 ECC across 16 DIMMs
  • PSU: 1,500W or 2,400W
  • Storage: up to 124 TB NVMe

Dell Pro Precision 9 T6

  • Maximum scalability tower, 15 PCIe slots
  • CPU: 18–86 core Intel Xeon 600 options
  • GPU: 2× 600W GPUs (192 GB aggregate) OR 5× 300W GPUs (5× RTX Pro 6000 Max-Q = 480 GB aggregate) OR 7× 50W
  • Memory: up to 4 TB DDR5 ECC
  • This is Dell’s answer to a small inference cluster in a single chassis

GPU range across all T4/T6: GeForce RTX 5060 up through RTX Pro 6000 Blackwell 600W. Consumer GeForce cards are available but lack ECC and professional driver support.

Pricing: Not publicly listed at launch. Enterprise quotes required. A T6 with dual RTX Pro 6000 Blackwell (600W) will be $25,000+ based on GPU cost alone.

Previous-gen still available and in-stock: Dell Precision 3660/5860/7960 Tower with RTX 6000 Ada (48 GB) and RTX 5000 Ada (32 GB). These are the actual available options at most enterprise resellers today.

Dell XPS desktop: The XPS line is not positioned as a workstation and does not ship with professional-class GPUs. XPS 8960/9010 supports consumer GeForce up to RTX 4090 — useful for a single developer workstation, not production serving.

Sources: AEC Magazine Dell Pro Precision 9 launch coverage (March 2026, TIER 2); SiliconANGLE workstation AI coverage (March 2026, TIER 2); StorageReview Dell Precision portfolio expansion (TIER 2).

HP

HP Z4 G6i Desktop Workstation (2025/2026 — Xeon 600 generation)

  • CPU: Intel Xeon W5-2500 series (Granite Rapids)
  • GPU: up to 2× NVIDIA RTX Pro 6000 Blackwell Max-Q 300W (192 GB aggregate), OR 1× RTX Pro 6000 Blackwell 600W (96 GB), OR 2× AMD Radeon PRO W7900 (96 GB aggregate)
  • Memory: up to 512 GB DDR5 6400 MT/s
  • Mid-tower; reasonable desk or rack footprint

HP Z8 Fury G6i Desktop Workstation (2025/2026)

  • CPU: Intel Xeon W9-3500 series (Granite Rapids), up to dual socket
  • GPU: up to 4× NVIDIA RTX Pro 6000 Blackwell Max-Q 300W (384 GB aggregate)
  • Also supports: 1× RTX Pro 6000 Blackwell 600W, 2× AMD Radeon PRO W7900
  • Memory: up to 2 TB DDR5 6400 MT/s
  • Full tower; HP’s maximum-performance workstation

HP Z6 G5 (Previous generation, widely available in stock)

  • GPU options: NVIDIA RTX 6000 Ada (48 GB), RTX 5000 Ada (32 GB), NVIDIA A800 40 GB
  • The Z6 G5 is the current “order today, receive this quarter” answer for most enterprise procurement cycles

HP ZBook Mobile Workstations

  • ZBook X G2i (2025): up to RTX Pro 6000 Blackwell Max-Q (96 GB) — 300W variant in a large mobile workstation chassis
  • ZBook 8 G2i/G2a: Mid-range mobile; RTX Pro 3000–5000 class GPUs (16–32 GB VRAM)

Sources: AEC Magazine HP Z workstation launch (TIER 2); HP Official product pages (TIER 1).

Lenovo

ThinkStation P3 Tower Gen 2 (2025/2026)

  • CPU: Intel Core Ultra (Arrow Lake) — consumer-class CPU, not Xeon
  • GPU: up to 1× RTX Pro 6000 Blackwell Max-Q 300W (96 GB)
  • Target: entry-level workstation; positioned for a single-user local AI developer

ThinkStation P5 Gen 1 (Current)

  • CPU: Intel Xeon W3-2400 series
  • GPU: up to 2× RTX Pro 6000 Blackwell Max-Q 300W (192 GB aggregate); previously configured with up to RTX 6000 Ada 48 GB
  • 7 GPU options from NVIDIA T400 (4 GB) up through RTX Pro 6000 Blackwell Max-Q
  • Single-socket mid-tower

ThinkStation P7 Gen 1 (Current)

  • CPU: Intel Xeon W5-3400 or W9-3500 series (single socket, high core count)
  • GPU: up to 3× RTX 6000 Ada (previous gen) / 2× RTX Pro 6000 Blackwell Max-Q (new configs)
  • Also supports: RTX Pro 5000 Blackwell (48 GB GDDR7), RTX Pro 4500 Blackwell (32 GB GDDR7)
  • Highest single-socket Lenovo tower

ThinkStation P8 Gen 1 (Current)

  • CPU: Dual Intel Xeon W9-3500 (up to 120 total cores)
  • GPU: up to 3× RTX 6000 Ada (48 GB) OR up to 3× RTX Pro 6000 Blackwell Max-Q 300W OR 1× RTX Pro 6000 Blackwell 600W
  • Maximum Lenovo workstation; dual-socket = deep memory capacity
  • Also supports: full RTX Pro Blackwell stack (5000/4500/4000 variants)

Lenovo Desktop/Tower Options (Non-Workstation)

  • ThinkCentre M-series and IdeaCentre G5 are standard business desktops — these do NOT support professional-class GPUs. Maximum GPU is typically a consumer GeForce RTX 3060/4060 (12 GB VRAM) in an optional discrete slot.
  • For local SLM use on Lenovo non-workstation hardware, the ceiling is sub-30B models at 4-bit quantization. The ThinkCentre is suitable for running Phi-4 14B or Llama 3 8B locally; it cannot run 70B models.
  • Lenovo AI PC desktops (ThinkCentre with Intel Core Ultra or AMD Ryzen AI) include NPU hardware but the NPU performance inversion for LLM decode (see model-portability note) means these are not meaningful for inference acceleration — the NPU is useful for background tasks, not chat.

Lenovo ThinkPad P-Series Mobile Workstations

  • ThinkPad P1 Gen 7 / P16 Gen 2: NVIDIA RTX 3000–5000 Ada mobile GPUs (8–16 GB VRAM)
  • Mobile form factor caps at Max-Q variants; 16 GB is the realistic current ceiling for ThinkPad P mobile

Sources: NotebookCheck ThinkStation P5/P7/P8 coverage (TIER 2); thinkstation-specs.com P8 specs (TIER 2); Lenovo official product pages (TIER 1).

ASUS / MSI

ASUS ProArt Studiobook (Mobile)

  • ProArt Studiobook Pro 16/17: NVIDIA RTX 3000–5000 Ada Laptop GPUs (8–16 GB VRAM)
  • ProArt P16 (2025): RTX 50-series laptop GPU, up to RTX 5070/5080 class (12–16 GB VRAM)
  • No current ASUS desktop workstation with 48+ GB VRAM — ProArt is a mobile/creator line, not a professional tower competing with Dell/HP/Lenovo P/Z/ThinkStation class
  • Exception: ASUS ProArt GoPro PX13 (CES 2026) uses AMD Ryzen AI Max+ with up to 128 GB unified memory — relevant for Apple Silicon-style inference without discrete GPU

MSI

  • Creator Z16/Z17 / WS76: Creator/workstation laptops with RTX 4060–4090 (8–24 GB VRAM); consumer-class cards only
  • No MSI tower workstation with professional-class GPUs (W7900, RTX 6000 class) in the US market

Procurement Decision Matrix

Budget (system total) GPU VRAM Recommended Platform Runs What
~$3,000–4,000 RTX 4500 Ada (24 GB) or RTX 4000 Ada (20 GB) 20–24 GB Dell Precision 3660, Lenovo P3 Phi-4 14B FP16, Llama 3 8B FP16, 34B Q4_K_M with CPU offload
~$5,000–8,000 RTX 5000 Ada (32 GB) 32 GB Any mid-tower P/Z-series Llama 3 70B Q4_K_M (fits with headroom)
~$4,000–6,000 Radeon PRO W7800 (32 GB) 32 GB HP Z4/Z6, Dell Precision Same as above; Linux only for full ROCm
~$10,000–14,000 RTX 6000 Ada (48 GB) 48 GB Dell T4, HP Z6 G5, Lenovo P7/P8 Llama 3 70B FP16 (fits), 2× 34B models in FP8
~$5,500–7,000 Radeon PRO W7900 (48 GB) 48 GB HP Z4/Z6/Z8, Dell Precision Same as RTX 6000 Ada; Linux ROCm caveat
~$18,000–25,000 RTX Pro 6000 Blackwell (96 GB) 96 GB New Dell T4/T6, HP Z4 G6i, Lenovo P8 Llama 3.1 405B Q2_K (fits in 92 GB)

Part 3: Lightweight LLM Inference Servers

These are the inference servers that run on a single workstation without Kubernetes, container orchestration, or dedicated MLOps staff. They expose an OpenAI-compatible API endpoint that connects to standard tooling (Claude Code, LangChain, Open WebUI, etc.).

llama.cpp / llama-server

GitHub: ggml-org/llama.cpp | License: MIT | Stars: ~111,000

What it is: The foundational C++ inference runtime. llama-server is the built-in HTTP server, not a separate project. Single binary, no Python required. The engine under the hood of Ollama and Jan.ai.

GPU backends:

Backend OS Notes
CUDA (NVIDIA) Linux, Windows Primary, fastest path
ROCm/HIP (AMD) Linux Requires ROCm 6.x install
Metal (Apple Silicon) macOS Supported
Vulkan (cross-platform) Linux, Windows Enables AMD GPU on Windows without ROCm
SYCL/oneAPI (Intel Arc) Linux Supported

OpenAI compatibility: Yes. /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/responses. Not claimed as 100% spec-compliant but covers 90%+ of what clients need.

Concurrency: Configurable parallel slots via --parallel N. Continuous batching. On a single RTX 4090 (24 GB) running Llama 3 8B Q4_K_M: ~4 parallel slots at 4K context without OOM.

Enterprise features: --api-key flag (multiple keys), TLS via --ssl-key-file + --ssl-cert-file, Prometheus /metrics endpoint, multi-model router mode.

Performance (directional):

GPU Model Quant Tok/s
RTX 4090 (24 GB) Llama 3 8B Q4_K_M ~125–135
RTX 4090 (24 GB) Llama 3 70B Q2_K ~18 (fits in 24 GB)
RTX 6000 Ada (48 GB) Llama 3 70B Q4_K_M ~18–22
RTX 5090 (32 GB) Llama 3 8B Q4_K_M ~160+

IT admin deployment estimate: 30–90 minutes. Pre-built binaries for CUDA 11/12/13, ROCm, Metal. Main friction: GPU driver compatibility and --ngl tuning.

Ollama

GitHub: ollama/ollama | License: MIT | Stars: ~172,000 | Latest: v0.24.0 (May 14, 2026)

What it is: Model manager + inference server built on llama.cpp. Handles model download, GPU memory allocation, REST and OpenAI-compatible APIs. The “Docker for LLMs” framing is apt — the easiest entry point in the category.

GPU support:

  • NVIDIA CUDA: full support
  • AMD ROCm: Linux only (ROCm 6.x). Windows AMD not supported natively
  • Apple Silicon: supported via MLX backend (v0.18+)
  • Intel Arc: limited

Concurrency model: OLLAMA_NUM_PARALLEL (parallel requests per model, default: 1), OLLAMA_MAX_LOADED_MODELS (models in GPU memory simultaneously), OLLAMA_MAX_QUEUE (queue depth before rejection). Uses static KV cache pre-allocation — not PagedAttention.

Critical production limitation: The static KV cache pre-allocation means multi-user concurrency degrades architecturally, not just from under-tuning. A published benchmark comparing Ollama vs vLLM under load measured 41 tok/s vs 793 tok/s and P99 latency 673ms vs 80ms respectively. Practical ceiling: 1–4 simultaneous users before noticeable degradation. Appropriate for an IT team’s internal tool; not adequate for 50+ user department deployment without load balancing across multiple Ollama instances.

Auth: None native. Standard practice: nginx reverse proxy with auth_request.

IT admin deployment estimate: 10–15 minutes. curl -fsSL https://ollama.com/install.sh | sh on Linux, installer on Windows/macOS. GPU detection is automatic.

Best for: Getting a team of 1–10 users running local LLMs in under an hour. The shortest path from hardware to working API endpoint.

LM Studio (Server Mode)

URL: lmstudio.ai | License: Proprietary (free personal use, commercial licensing available) | Latest: v0.4.2 (February 2026)

What it is: GUI-first LLM manager for individual developers and small teams. Server mode added for headless/API use. The lms server start CLI (headless mode, added v0.4.0) enables Linux deployment without the GUI.

API: OpenAI-compatible at port 1234. Also Anthropic-compatible endpoints. Continuous batching added v0.4.0 via llama.cpp parallel-slot support; extended to MLX engine in v0.4.2.

GPU backends: NVIDIA CUDA, AMD (Vulkan; ROCm in some builds), Apple Silicon (MLX backend, well-optimized), CPU fallback.

Enterprise limitations: No built-in auth, rate limiting, or access control. Best suited for solo developers, small team tooling (2–5 people), evaluations.

IT admin deployment estimate: 5–10 minutes on Windows/macOS. Linux headless: 20 minutes with CLI.

Unsloth Studio

GitHub: unslothai/unsloth | License: Apache 2.0 (training library) | Stars: ~30,000 | Status: Studio in beta, May 2026

What it is: Primarily a fine-tuning accelerator (2× faster training, 70% less VRAM vs standard HuggingFace training) with a no-code web UI. Also includes inference serving.

Inference API: OpenAI-compatible at localhost:8000/8888. Bearer token auth. Supports both /v1/chat/completions and /v1/messages (Anthropic-style). Built-in tool calling, web search, code execution. Handles GGUF and safetensor models.

Hardware support:

Platform Training Inference/Chat
NVIDIA RTX 30/40/50/Blackwell Full Full
Intel GPU Full Full
AMD Chat works Training: coming
Apple Silicon Chat + Data Recipes MLX training coming
CPU Chat inference only Yes

Best for: When your team wants to fine-tune a base model and serve it without switching platforms. Not the best pure inference server if fine-tuning is not needed.

IT admin deployment estimate: 30–60 minutes. Still in beta; expect rough edges on AMD and Apple Silicon.

TabbyAPI

GitHub: theroyallab/tabbyAPI | License: AGPL-3.0 | Stars: ~1,200

What it is: The official HTTP API server for ExLlamaV2 and ExLlamaV3 inference backends. OpenAI-compatible. Purpose-built for single-GPU, maximum-throughput quantized inference.

Why ExLlamaV2/V3 matters: EXL2/EXL3 uses measurement-based, mixed-bit-width quantization — profiling each layer and assigning bits based on sensitivity rather than uniform quantization. Result: better quality at a given model size than uniform GGUF quantization. Community consensus as of 2026: “on a single RTX 3090/4090/5090, no other framework matches ExLlamaV2’s speed for INT4 quantized models — not vLLM, not llama.cpp, not Ollama.”

GPU support: NVIDIA Ampere and newer (RTX 3000+) required for paged attention. AMD: not supported. Apple Silicon: not supported. CUDA-only.

AGPL-3.0 warning: AGPL requires source disclosure if you modify and expose over a network. Many Fortune 500 legal policies prohibit AGPL software in commercial deployments. Verify with legal before deploying.

IT admin deployment estimate: 45–90 minutes. Requires Python environment, pip install, ExLlamaV2 install, and EXL2-format models from HuggingFace.

Best for: A single NVIDIA workstation where one engineer needs maximum tokens/second for their own tools. Not multi-user scale.

Aphrodite Engine

GitHub: aphrodite-engine/aphrodite-engine | License: AGPL-3.0 | Stars: ~1,700 | Latest: v0.21.0 (May 2, 2026)

What it is: A vLLM fork from the PygmalionAI community. Inherits vLLM’s PagedAttention architecture (the correct concurrency model) with lower configuration overhead for quantized models.

Differences from vLLM:

  • Broader quantization formats out of the box: AQLM, AWQ, GPTQ, EXL2/ExLlamaV3, GGUF, QuIP#, SqueezeLLM, Marlin, VPTQ, MXFP4
  • aphrodite run <model> single command (analogous to ollama run) — handles download and serving
  • Less configuration overhead for quantized model deployment than upstream vLLM

Hardware: NVIDIA CUDA 12+ (primary), AMD ROCm (supported), Linux primary. Windows via WSL2.

AGPL-3.0 warning: Same concern as TabbyAPI — verify with legal.

IT admin deployment estimate: 60–120 minutes. pip install aphrodite-engine, requires CUDA 12+, Python 3.10+.

Best for: When you need vLLM-class concurrency (PagedAttention) on quantized models without full vLLM configuration complexity. Better fit than vLLM for single-workstation quantized serving.

LocalAI

GitHub: mudler/LocalAI | License: MIT | Stars: ~46,400 | Latest: v4.2.6 (May 16, 2026)

What it is: The broadest-scope local AI engine. Not just LLMs — supports image generation (Stable Diffusion, LTX-2), speech synthesis, voice transcription (Whisper), video generation, embeddings. OpenAI API drop-in with Anthropic API compatibility added in v3.10.0 (January 2026).

GPU backends:

Backend OS
NVIDIA CUDA 12/13 Linux, Windows
AMD ROCm Linux
Intel oneAPI/SYCL Linux
Apple Silicon Metal macOS
Vulkan (cross-platform) Linux, Windows
NVIDIA Jetson (L4T) Edge
CPU only All — first-class feature

CPU-only deployment is intentional and first-class. LocalAI is the only tool in this list specifically designed to run well without any GPU for smaller models — relevant for deploying on standard enterprise desktops without discrete GPUs.

Multi-model routing: Single API endpoint routes to multiple backend model instances. Automatic multi-GPU model fitting added December 2025. MIT license — no AGPL concern.

Deployment: Docker is the recommended path: docker run -p 8080:8080 localai/localai:latest --gpus all. Model gallery: localai models install llama-3-8b.

IT admin deployment estimate: 60–120 minutes via Docker.

Best for: Multi-modal local AI (LLMs + image + voice in one API). Also the best answer for heterogeneous hardware environments or CPU-only deployment targets.

GPT4All

GitHub: nomic-ai/gpt4all | License: MIT | Stars: ~77,400 | Latest: v3.10.0 (February 2025)

What it is: Desktop application for non-technical users to run local LLMs. API server mode available but secondary to the GUI.

GPU acceleration: Vulkan (cross-platform: NVIDIA + AMD on Windows and Linux). Metal (macOS). The Vulkan backend is the cleanest path to AMD GPU acceleration on Windows without ROCm — the most underserved combination in this category.

API server: Enabled via Settings → Application → Enable Local Server. Port 4891. Implements a subset of the OpenAI API. Not designed for programmatic production use. As of early 2026: streaming is partial, no embeddings endpoint.

GPT4All Enterprise: Nomic offers a commercial product with usage analytics, model performance tracking, centralized model distribution, and MDM-style deployment. Pricing not public; worth a pricing conversation for organizations wanting central LLM management for non-technical staff.

IT admin deployment estimate: 5–10 minutes (installer). API mode: 10 minutes.

Best for: Deploying a local AI chat assistant to non-technical employees who should not interact with a terminal. GPT4All Enterprise for MDM-managed fleet deployment.

Part 4: Hardware × Software Decision Matrix

By Deployment Scenario

Scenario Hardware Inference Server Notes
1 developer, fastest possible, NVIDIA RTX 5090 or RTX 4090 TabbyAPI + ExLlamaV3 EXL3 quantized models from HuggingFace
Small team (2–10), NVIDIA, low ops overhead Any NVIDIA workstation Ollama 10-minute setup, acceptable concurrency at ≤10 users
Medium team (10–50), NVIDIA, production serving RTX 6000 Ada 48 GB or better Aphrodite Engine PagedAttention concurrency, simpler than vLLM
Fine-tune + serve, NVIDIA RTX 4090 or RTX 6000 Ada Unsloth Studio One platform for both operations
Multi-modal (text + image + voice), any GPU Any supported GPU LocalAI Broadest capability surface; MIT license
AMD GPU, Linux Radeon PRO W7900 llama.cpp (ROCm) or LocalAI ROCm 6.x required; well-tested path
AMD GPU, Windows Radeon PRO W7900 or W7800 GPT4All (Vulkan) or llama.cpp Vulkan Vulkan avoids ROCm Windows limitation
Non-technical staff, managed deployment Any GPU or CPU GPT4All or GPT4All Enterprise GUI-first; no terminal required
Multi-modal, Apple Silicon Mac Studio or MacBook Pro LocalAI or LM Studio MLX backend for Apple; LocalAI for multi-modal
CPU-only (no discrete GPU) Standard enterprise desktop LocalAI or Ollama Sub-14B models; governance benefit without GPU cost

By GPU Platform

Platform Fastest inference Best concurrency Easiest ops Multi-modal
NVIDIA CUDA (Linux) TabbyAPI/ExLlamaV3 Aphrodite Engine Ollama LocalAI
NVIDIA CUDA (Windows) llama-server llama-server Ollama LocalAI (Docker)
AMD ROCm (Linux) llama-server Aphrodite Ollama LocalAI
AMD (Windows) GPT4All (Vulkan) llama.cpp Vulkan GPT4All LocalAI (Docker)
Apple Silicon (macOS) LM Studio llama-server Ollama LocalAI
CPU only LocalAI LocalAI Ollama (CPU mode) LocalAI

Part 5: AGPL-3.0 License Advisory

Three tools in this list are AGPL-3.0 licensed: TabbyAPI, Jan.ai/Cortex, and Aphrodite Engine. The AGPL requires that if you modify AGPL software and expose it over a network (even internally), you must make the modified source code available.

Many Fortune 500 legal policies prohibit AGPL software in commercial products and internal production deployments without a commercial exception or legal review. Before deploying these tools:

  1. Check with your organization’s legal or open source compliance team
  2. If AGPL is prohibited, the MIT-licensed alternatives (llama.cpp, Ollama, LocalAI, GPT4All) cover the same use cases without license risk

Part 6: Quick-Reference Repository Data (May 20, 2026)

This section captures specific version and adoption data verified at time of research.

Project License Stars Latest Version / Build Language
llama.cpp (ggml-org) MIT 111,000+ Build b9239 (May 20, 2026; daily rolling) C/C++
Ollama (ollama/ollama) MIT 172,000+ v0.24.0 (May 14, 2026) Go (67%), C (27%)
LocalAI (mudler/LocalAI) MIT 46,400+ v4.2.6 (May 16, 2026) Go, Python
GPT4All (nomic-ai/gpt4all) MIT 77,400+ v3.10.0 (Feb 25, 2025) C++ (52%), QML (30%)
Aphrodite Engine AGPL-3.0 1,700+ v0.21.0 (May 2, 2026) Python
TabbyAPI (theroyallab) AGPL-3.0 1,200+ Rolling (no versioned releases) Python/FastAPI
Jan.ai (janhq/jan) Apache 2.0 — v0.7.9 (March 23, 2026) TypeScript, C++
LM Studio Proprietary — v0.4.2 (Feb 2026) Electron/C++
Unsloth Studio Unsloth License ~30,000 (training lib) Beta (May 2026) Python

llama.cpp milestone: Reached 100K GitHub stars in March 2026, faster than PyTorch (~7 years) or TensorFlow (~8 years) reached the same milestone. As of May 20, 2026 (the date of this note), build b9239 is the latest daily release.

Ollama milestone: 172K stars / 16,200 forks reflects the highest community adoption of any local LLM tool. The project has published 217 total releases as of v0.24.0.

Part 7: Detailed Concurrency Architecture Analysis

Understanding why different servers hit concurrency ceilings at different user counts matters for sizing decisions. The root cause in all cases is KV cache memory management.

The KV Cache Problem

Every transformer inference run generates a Key-Value cache that stores intermediate attention states for all tokens in the context. For a 70B model at 4K context, this is roughly 2 GB of GPU memory per concurrent request. At 96 GB VRAM, the theoretical ceiling is ~48 concurrent requests (96 GB / 2 GB each) — but in practice, model weights themselves consume 40+ GB, leaving 50+ GB for KV cache, which supports approximately 25 concurrent 4K-context requests.

The implementation strategy for KV cache differs meaningfully across tools:

Static allocation (Ollama, LM Studio default): VRAM is pre-allocated for the maximum configured context length (num_ctx) regardless of actual sequence length. A num_ctx=32768 setting on a 70B Q4_K_M model reserves ~16 GB of VRAM for KV cache even if most requests are short. This wastes memory and limits concurrency to a conservative estimate.

Slot-based batching (llama-server): A fixed number of slots are allocated (--parallel N), each sized for the configured context. More controlled than Ollama’s approach; the administrator tunes slot count and context length as a pair. Continuous batching ensures GPU is not idle between token batches.

PagedAttention (vLLM, Aphrodite, TabbyAPI paged mode): KV cache is managed as paged virtual memory. Only the tokens actually in a request’s current context consume VRAM. This enables near-optimal memory utilization and is the reason vLLM achieves 16–20x higher throughput than Ollama under load. TabbyAPI implements paged attention for Ampere+ NVIDIA GPUs, which is why it achieves higher throughput than llama.cpp for quantized NVIDIA inference.

Practical Concurrency Sizing Guide

For IT architects provisioning a departmental inference workstation:

Formula: Usable VRAM ÷ (Model weight footprint + KV cache per slot × slots) = Maximum concurrent requests

Example with RTX 6000 Ada (48 GB) + Llama 3 70B Q4_K_M (~38 GB weights):

  • Available for KV cache: 48 - 38 = 10 GB
  • At 4K context, KV cache per slot ≈ 0.5 GB
  • Maximum concurrent: ~20 slots (theoretical); 8–12 practical (safety margin, system overhead)

Example with RTX Pro 6000 Blackwell (96 GB) + Llama 3 70B Q4_K_M (~38 GB weights):

  • Available for KV cache: 96 - 38 = 58 GB
  • At 4K context: ~30–40 concurrent (theoretical); 20–25 practical

For teams requiring more than 20 concurrent users on a single GPU, the lightweight servers in this note hit their ceiling. vLLM’s PagedAttention implementation handles this range efficiently; the decision to adopt vLLM’s operational overhead becomes justified above ~20 simultaneous users.

Part 8: AMD ROCm Platform Notes for Enterprise IT

AMD ROCm is the open-source compute stack equivalent to NVIDIA CUDA. It is mature on Linux but requires attention in enterprise deployments.

Current State (May 2026)

ROCm version: ROCm 6.x is the current major version. Full GPU compute (including inference with llama.cpp ROCm backend, Ollama ROCm, LocalAI ROCm backend) is supported on Linux with ROCm 6.x installed.

Windows status: ROCm on Windows is not production-ready for inference workloads as of this note. Ollama explicitly documents that Windows Radeon GPU defaults to OLLAMA_MAX_LOADED_MODELS=1 and OLLAMA_NUM_PARALLEL=1 due to ROCm v5.7 VRAM reporting limitations on Windows. The expectation from the Ollama project is that ROCm v6.2 for Windows will enable proper parallel request handling — but this was not shipped at time of writing.

Workaround for Windows AMD: Use Vulkan backend (available in llama.cpp, GPT4All). Vulkan GPU acceleration works on Windows with AMD GPUs without ROCm. Performance is lower than ROCm on Linux (roughly 60–80% of ROCm performance in community testing) but meaningfully faster than CPU-only.

Supported AMD GPUs for ROCm Inference

ROCm officially supports RDNA 2 (RX 6000 series) and RDNA 3 (RX 7000/Radeon PRO W7000 series). The Radeon PRO W7900 (48 GB) and W7800 (32 GB) are confirmed supported with ROCm 6.x.

Consumer RX 7900 XTX (24 GB GDDR6) is ROCm-supported but requires manual HSA_OVERRIDE_GFX_VERSION flag setting in some configurations. For enterprise workstations with IT policy against manual overrides, the professional W7900/W7800 series is the preferred choice.

Enterprise Decision: NVIDIA Linux vs AMD Linux

For organizations choosing between NVIDIA RTX 6000 Ada (48 GB, ~$6,800) and AMD Radeon PRO W7900 (48 GB, ~$3,999) on Linux:

  • Cost advantage: AMD saves ~$2,800 per workstation at the 48 GB tier
  • Software maturity: NVIDIA CUDA ecosystem is more mature; all inference tools tested against CUDA first
  • Performance gap: AMD claims within 7% of RTX 6000 Ada on professional visualization workloads; LLM inference on ROCm is typically within 10–20% of CUDA on comparable hardware
  • Risk: ROCm updates occasionally break compatibility; enterprises should pin ROCm version and test upgrades before fleet deployment

For budget-constrained deployments where Linux is the OS, the W7900 is a rational choice. For risk-averse IT departments with NVIDIA standardization already in place, the premium for CUDA ecosystem stability is defensible.

Part 9: Enterprise API Gateway Pattern

No tool in this note provides enterprise-grade authentication, rate limiting, audit logging, or RBAC natively. For departmental deployments that need these controls, the standard architecture is:

[Users / Internal Tools]
         |
    [API Gateway]
    (Nginx / Traefik / Kong / agentgateway)
    - mTLS client authentication
    - Bearer token validation
    - Rate limiting per user/team
    - Audit logging (all requests, all responses)
    - RBAC (which teams access which models)
         |
    [Inference Server]
    (llama-server / Ollama / Aphrodite)
    - API key from gateway only
    - Prometheus /metrics exposed to internal monitoring
    - Not exposed directly to users

agentgateway (open source, 2025) is purpose-built for this pattern and includes LLM-specific features like per-user token quota, request/response logging for compliance, and model routing. For enterprises with existing Nginx/Kong infrastructure, those work equally well with a simpler configuration.

llama-server supports the API key and TLS layer directly (--api-key, --ssl-key-file, --ssl-cert-file) — sufficient for small teams where a reverse proxy is overhead. For teams requiring audit logs, the gateway layer is necessary because llama-server logs requests but does not log response content.

Sources

  1. AEC Magazine, “Dell launches new Pro Precision workstations” (March 2026). TIER 2.
  2. SiliconANGLE, “Dell workstations get major AI-focused upgrades” (March 2026). TIER 2.
  3. StorageReview, “Dell expands professional workstation portfolio” (2026). TIER 2.
  4. Dell Newsroom, “Dell reimagines commercial PCs” (March 25, 2026). TIER 1 (vendor).
  5. NVIDIA, RTX Pro 6000 Blackwell Workstation Edition product page (2026). TIER 1.
  6. ThunderCompute, “NVIDIA RTX Pro 6000 Blackwell pricing May 2026” (TIER 3).
  7. VideoCardz, “NVIDIA RTX PRO 6000 Blackwell desktop GPU drops to $7,999” (TIER 2).
  8. AEC Magazine, “HP Z workstations: Z4 G6i, Z8 Fury G6i, ZBook X G2i, ZBook 8 G2i” (2025/2026). TIER 2.
  9. HP Official, Z8 Fury G6i product page (2026). TIER 1.
  10. HP Official, Z4 G6i product page (2026). TIER 1.
  11. NotebookCheck, “Lenovo ThinkStation P5/P7 feature Intel Xeon Scalable and NVIDIA RTX 6000 Ada.” TIER 2.
  12. thinkstation-specs.com, ThinkStation P8 specs (TIER 3).
  13. Lenovo, ThinkStation P8 spec sheet (psref.lenovo.com). TIER 1.
  14. AEC Magazine, “AMD Radeon Pro W7900 and Pro W7800 review.” TIER 2.
  15. VideoCardz, “AMD Radeon Pro W7900 and W7800 RDNA3 launch.” TIER 2.
  16. AMD, Radeon PRO W7900 datasheet (amd.com). TIER 1.
  17. VideoCardz, “NVIDIA RTX 5000 Ada 32GB, RTX 4500 24GB, RTX 4000 20GB workstation specs.” TIER 2.
  18. llama.cpp GitHub (ggml-org/llama.cpp), llama-server README, build b9239. MIT. TIER 1.
  19. Ollama GitHub (ollama/ollama), v0.24.0. MIT. TIER 1.
  20. Ollama FAQ (docs.ollama.com/faq). TIER 1.
  21. AICompetence.org, “Ollama production limitations.” TIER 3.
  22. CoderSera, “vLLM vs Ollama vs LM Studio 2026 benchmark.” TIER 3.
  23. Markaicode, “Ollama vs vLLM throughput benchmark 2026.” TIER 3.
  24. TabbyAPI GitHub (theroyallab/tabbyAPI), AGPL-3.0, ~1.2K stars. TIER 1.
  25. ExLlamaV2 GitHub (turboderp-org/exllamav2). TIER 1.
  26. ExLlamaV3 GitHub (turboderp-org/exllamav3). TIER 1.
  27. Unsloth Studio documentation (unsloth.ai/docs/new/studio). TIER 2.
  28. LocalAI GitHub (mudler/LocalAI), v4.2.6. MIT, 46.4K stars. TIER 1.
  29. LocalAI release notes v3.10.0 (January 2026) and v4.x series. TIER 1.
  30. GPT4All GitHub (nomic-ai/gpt4all), v3.10.0. MIT, 77.4K stars. TIER 1.
  31. Aphrodite Engine GitHub (aphrodite-engine/aphrodite-engine), v0.21.0, AGPL-3.0. TIER 1.
  32. Jan.ai documentation, v0.7.9 (jan.ai/docs). Apache 2.0. TIER 2.
  33. LM Studio documentation, v0.4.2 (lmstudio.ai/docs). TIER 2.
  34. BestGPUsForAI, “RTX 6000 Ada vs RTX 4090 LLM benchmarks 2026.” TIER 3.
  35. Phoronix, “llama.cpp AI performance with GeForce RTX 5090.” TIER 3.
  36. Mustafa.net, “Local LLM tokens/sec benchmarks RTX 4090, 3090, 4060 Ti 2026.” TIER 3.
  37. AIThinkerLab, “llama.cpp 100K GitHub Stars 2026.” TIER 3.
  38. ServiceStack, “Self-host LLMs in production with llama-server.” TIER 3.

Appendix: Model Size to VRAM Quick Reference

Model Family Parameter Count Q4_K_M VRAM FP16 VRAM Minimum GPU (Q4_K_M) Notes
Phi-4 14B ~8 GB ~28 GB RTX 4000 Ada (20 GB) Microsoft; strong coding + reasoning
Llama 3.2 3B ~2.5 GB ~6 GB Any 4+ GB GPU Best tiny model for constrained hardware
Llama 3 / Qwen 2.5 8B ~5 GB ~16 GB Any 8+ GB GPU Production quality for most enterprise tasks
Mistral Nemo 12B ~7 GB ~24 GB RTX 4000 Ada (20 GB) Multilingual; strong instruction following
Llama 3 34B (estimate) ~20 GB ~68 GB RTX 4500 Ada (24 GB) tight Significant quality step above 8B
Qwen 2.5 Coder 32B ~19 GB ~64 GB RTX 4500 Ada (24 GB) Best open code model at 32B
Llama 3 / Qwen 2.5 70–72B ~38–42 GB ~140–144 GB RTX 6000 Ada (48 GB) Production ceiling for 48 GB GPU
Llama 3.1 405B ~92 GB (Q2_K) ~810 GB RTX Pro 6000 Blackwell 96 GB (Q2 only) Q2 quality is acceptable for many retrieval tasks

Notes on quantization quality:

  • Q8_0: near-lossless; 2× model size vs FP16
  • Q4_K_M: recommended default; ~3% quality degradation vs FP16 for most tasks; 4× compression
  • Q2_K: significant degradation; only appropriate for retrieval/classification tasks, not generation

Brandon Sneider | brandon@brandonsneider.com May 2026