← Local Tiny Models 🕐 19 min read
Local Tiny Models

LLM Model Portability Across Hardware Backends: Conversion Pipelines, Gaps, and Enterprise Decision Matrix

Brandon Sneider · May 2026

> **Source credibility: TIER 2 (vendor documentation, community benchmarks, arXiv preprints).**

See also (wiki): wiki/inference-economics.md · wiki/ai-platform-selection.md · wiki/local-tiny-models.md

Source credibility: TIER 2 (vendor documentation, community benchmarks, arXiv preprints). Hardware backend documentation (AWS Neuron, ROCm, OpenVINO, MLX, QNN) is primary source — HIGH for documented capabilities, MEDIUM for performance claims. Community benchmarks (llama.cpp, Ollama throughput figures) are directionally reliable but not independently reproduced. The AMD MI300X peak utilization gap (arXiv:2510.27583) is a peer-reviewed finding. ONNX Runtime GenAI figures are vendor-published. Intel NPU vs CPU performance inversion is confirmed across multiple independent practitioner reports. All figures should be treated as order-of-magnitude anchors, not contractual specifications.

Executive Summary

  • AMD ROCm on enterprise GPUs (MI300X/MI325X/MI355X) is the closest thing to a drop-in CUDA replacement as of 2026. PyTorch code runs without changes; vLLM, TGI, and llama.cpp all have first-class ROCm backends; 93% of AMD vLLM CI tests passing by March 2026. Performance gap to H100 has narrowed to 10–25% on inference workloads.
  • AWS Neuron requires a fundamentally different mental model. Models must be compiled to static NEFF binaries — batch size, sequence length, and precision are fixed at compile time. Compilation for 70B models takes 45–90 minutes. The economics justify it at scale; for dev/test or small teams, it is operationally expensive.
  • Intel NPU on Copilot+ PCs is not viable for LLM inference today. On Snapdragon X Elite, CPU inference reaches 25.88 tok/s vs NPU’s 2.64 tok/s for Llama 3.2 3B — a 10× performance inversion. The NPU’s tiling architecture is optimized for vision workloads, not autoregressive decode.
  • Apple MLX is the best single-machine inference stack for Mac. 4,255+ pre-converted models on mlx-community, seconds to convert, 4-bit quantization, LoRA fine-tuning included. M5 Neural Accelerators deliver 4× TTFT speedup over M4 for compute-bound prefill.
  • ONNX portability is real but limited by dynamic shapes. The ORT GenAI Model Builder handles the conversion; the problem is execution providers that require static shapes (QNN, many NPUs). For CPU and CUDA, ORT GenAI is competitive; for edge NPUs, static shape constraints are a hard wall.
  • Qualcomm Snapdragon NPU underperforms CPU for LLM decode on X Elite. Qualcomm’s own published acceleration work (April 2026) now targets CPU via QMX (matrix extensions) rather than NPU — a signal that CPU is the intentional path for Windows-on-Arm LLM inference.

1. AWS Neuron (Trainium2/3, Inferentia2)

The Core Constraint: Static Compiled Artifacts

Neuron uses a two-stage compilation pipeline. The XLA-based neuronx-cc compiler ingests PyTorch/XLA graphs and produces NEFF (Neuron Executable File Format) binaries. NEFFs encode fixed batch size, sequence length, and precision at compile time. There is no dynamic shape support at the NEFF level — any deviation at runtime crashes.

Production consequence: Every deployment must bucket sequence lengths. A conversation application needs pre-compiled NEFFs for [512, 1024, 2048, 4096] token lengths. Request routing must select the appropriate NEFF bucket.

Three Conversion Pathways

Path A — optimum-neuron (easiest, least flexible):

optimum-cli export neuron \
  --model meta-llama/Meta-Llama-3.1-8B \
  --batch_size 1 --sequence_length 4096 \
  --auto_cast_type bf16 \
  --output ./llama-neuron

Checks the aws-neuron Hugging Face cache before compiling. Cache is sparse — if your configuration doesn’t match a cached entry, compilation starts from scratch (20–60 minutes for 8B, 45–90 minutes for 70B). NeuronModelForCausalLM handles inference.

Path B — Transformers Neuronx (manual, more control): save_pretrained_split() to shard weights, then to_neuron() to compile. Needed for non-standard parallelism configurations.

Path C — NxD Inference (production GA, May 2025): NeuronLlamaForCausalLM with NeuronConfig specifying tensor/pipeline/expert parallelism degrees. This is the path for Trn2 at scale. vLLM on Neuron uses this backend.

Model Support

Model Status Notes
Llama 3.1 8B / 70B Confirmed Explicit multi-instance documentation
Llama 3.3 70B Community confirmed Trn2 reports working
Qwen3 235B A22B (MoE) Explicit AWS docs tp_degree=64; Qwen3-0.6B/32B had known head_dim bug — patched
Mixtral / DBRX Listed as supported NxD Inference overview
Mistral Ecosystem docs Less explicitly benchmarked
DeepSeek-V3 [NEEDS VERIFICATION] MoE routing compatibility not confirmed

Quantization Support

Format Neuron Support
BF16 Default, full support
FP8 Native
INT8 --auto_cast_type int8 in optimum-neuron
INT4 NxD Inference SDK 2.x — sparse documentation
GPTQ (external) Load quantized checkpoint — kernel maturity below CUDA
bitsandbytes NF4 Not supported

BF16 auto-cast can cause precision loss in rotary embeddings — documented AWS known issue.

Known Failure Modes

  1. Static shape crashes — runtime sequence length ≠ compiled length = crash
  2. Silent CPU fallback — unsupported PyTorch ops silently offload to CPU, destroying throughput with no error
  3. Long compile times — 45–90 min for 70B; Persistent Cache mitigates for repeat deployments
  4. BF16 precision loss — rotary embeddings, attention masks with dynamic indices
  5. Speculative decoding — known failure when sequence length approaches model maximum
  6. Trn1/Inf2 deprecation — NxD Inference dropped older instance types; old path is Transformers Neuronx

Economics

Blend360 comparison on inf2.8xlarge (2× NeuronCores v2) vs GPU for Llama 3.1 8B:

  • inf2.8xlarge: 1.65s avg response, $1.97/hr
  • g6.xlarge (L4 GPU): 3.83s, $0.80/hr
  • g6e.2xlarge (L40S GPU): 1.67s, $2.24/hr

Neuron justifies its complexity at model sizes ≥70B where Trn2’s large HBM capacity eliminates tensor parallelism overhead. For ≤8B at low concurrency, GPU instances are simpler and cheaper.

Tooling: optimum-neuron · neuronx-distributed (NxD Inference) · vLLM on Neuron (SDK 2.26+) · TGI on Neuron

2. AMD ROCm (MI300X, MI325X, MI355X, Consumer RDNA3/4)

The Conversion Story

PyTorch on ROCm uses the HIP toolchain — CUDA kernels are auto-translated at build time via hipify. For framework-level code (vLLM, TGI, HuggingFace Transformers), this translation is transparent. Install the ROCm PyTorch wheel; existing model code runs.

The nuance: AMD’s wavefront-64 architecture (64-wide) differs from NVIDIA’s 32-wide warps. Hand-tuned CUDA kernels (Flash Attention implementations, GPTQ/AWQ kernels, custom fusions) produce suboptimal performance when hipified without re-tuning to Wave64. The highest-efficiency NVIDIA kernels — TensorRT-LLM, FlashAttention 3 — have no ROCm equivalents.

2025–2026 milestones:

  • AMD upstreamed llama.cpp Wave64 performance patches: July 2025
  • vLLM ROCm CI pipeline launched: December 2025
  • vLLM ROCm CI test pass rate: 37% (November 2025) → 93% (March 2026)
  • AITER (AI Tensor Engine for ROCm) Flash Attention backend: 2.8–4.6× faster TPOT vs legacy ROCM_ATTN kernel

Quantization on ROCm

Framework ROCm Status
bitsandbytes Supported (ROCm-aware fork)
GPTQ Fully supported, HIP kernels in vLLM
AWQ Supported; Triton kernels; native ROCm-optimized kernels recommended
FP8 Native on MI300X+
FP4 Available in vLLM V1 for MI300X/MI325X/MI355X
llama.cpp ROCm Production-ready, RDNA3/4 upstream since July 2025
bitsandbytes NF4 on consumer RDNA Unsupported — LoRA/fine-tuning on consumer AMD effectively blocked

Performance vs H100

MI300X structural advantage: 192GB HBM3 (vs H100’s 80GB); 5.3 TB/s bandwidth (vs 3.35 TB/s). Eliminates tensor parallelism for most models below 175B, reducing inter-GPU communication overhead. Memory-bandwidth-bound decode: ~40% lower latency for Llama2-70B vs H100.

MI300X structural disadvantage: Peak FLOPs utilization ~45% (arXiv:2510.27583) vs H100/B200 at ~93%. Software maturity gap in kernel optimization. H100 TensorRT-LLM has lower prefill latency for compute-bound workloads.

Net for enterprise inference:

  • TTFT for Qwen models: 2.7× faster on MI300X (memory bandwidth advantage)
  • TPOT (memory-bound decode): MI300X competitive or better vs H100
  • Prefill (compute-bound): H100 TensorRT-LLM faster

Consumer (RX 7900 XTX, 24GB, ~$850 used): ~107 tok/s on Llama 2 7B Q4_0 llama.cpp Ubuntu. 10–25% slower than equivalent NVIDIA.

Ollama ROCm: Linux only as of May 2026. Supported: RX 7000 (RDNA3), RX 6000 (RDNA2), Ryzen AI, Instinct. Workaround for unsupported GPUs: HSA_OVERRIDE_GFX_VERSION. Vulkan backend: experimental (OLLAMA_VULKAN=1).

Enterprise stack: vLLM on ROCm (first-class, pre-built Docker), TGI on ROCm (Flash Attention, GPTQ/AWQ), vLLM V1 pip wheels (early 2026).

3. Intel Gaudi3 / OpenVINO / Core Ultra NPU

Intel Gaudi3

Dual-die, 128GB HBM2e, 3.7 TB/s bandwidth. More memory than H100 (128GB vs 80GB).

Conversion claim: 3–5 lines of code via optimum-habana — swap Trainer import, add HPU device specifier. Holds for standard HuggingFace Transformers. Custom CUDA kernels require explicit HPU ports.

Confirmed models: Llama 3.1, Llama 3.3 70B (TGI on Gaudi2), Mistral, Mixtral.

Intel’s own benchmark claims (vendor data — unverified independently):

  • 95–170% of H100 performance for Llama variants
  • Up to 4× H100 for Falcon 180B (memory-constrained)

Critical caveat: vLLM integration is a fork (HabanaAI/vllm-fork), not merged upstream. Intel’s organizational restructuring in 2025–2026 creates uncertainty about the Gaudi roadmap. Ecosystem tooling depth is significantly below NVIDIA. Treat as viable for specific workloads, not as a general NVIDIA replacement.

OpenVINO (CPU, integrated GPU, Arc GPU, NPU)

OpenVINO converts models to Intermediate Representation (IR: .xml + .bin). Modern path via optimum-intel:

optimum-cli export openvino \
  --model meta-llama/Meta-Llama-3.1-8B \
  --weight-format int4 \
  --sym --ratio 1.0 --group-size 128 \
  --output ./llama-openvino

Critical NPU footgun: --weight-format int4 alone produces models that will not run on Intel NPU. NPU-targeted export requires --sym --ratio 1.0 --group-size 128 flags explicitly.

OpenVINO 2026.0 additions: INT4 data-aware compression for MoE 3D MatMuls; mixed-precision INT8/FP16 hybrid (2.5× speedup, <1% perplexity degradation); dynamic prompt support; 8K context for NPU.

Intel NPU Reality (Core Ultra Series 2 / Lunar Lake)

The performance inversion: On Snapdragon X Elite (comparable NPU-class hardware), CPU inference delivers 25.88 tok/s vs NPU’s 2.64 tok/s for Llama 3.2 3B. Intel NPU shows a similar picture. Root cause: transformer decode is memory-bandwidth-bound and autoregressive (token-by-token) — NPU tiling architectures are optimized for parallel vision/batch workloads, not sequential decode.

Where NPU wins: Power efficiency for background/always-on inference, vision models, batch embedding generation. Not raw LLM throughput.

Hardware capability:

  • Lunar Lake NPU: 6 compute engines, ~48 TOPS, NF4 data type support
  • INT4 is the practical sweet spot — fits 3B–7B models in NPU SRAM
  • OpenVINO 2025.3 added dynamic prompt support and 8K context for NPU

4. Apple Silicon: MLX vs Core ML vs ExecuTorch

MLX — Primary Path for Mac Inference and Fine-Tuning

MLX operates on Apple Silicon unified memory via Metal GPU kernels. WWDC 2025 formally designated it the preferred LLM inference stack with three dedicated sessions.

Conversion:

mlx_lm.convert \
  --hf-path meta-llama/Meta-Llama-3.1-8B-Instruct \
  --mlx-path ./llama-3.1-8b-mlx \
  -q   # 4-bit quantization

Seconds to convert. mlx-community org on Hugging Face: 4,255+ pre-converted models.

M5 Neural Accelerator performance (Apple ML Research, Jan 2026):

  • TTFT: up to 4× faster vs M4 (compute-bound prefill)
  • Decode: 19–27% faster vs M4 (153 GB/s vs 120 GB/s bandwidth)
  • Qwen 14B BF16: TTFT under 10 seconds
  • Qwen 30B MoE: TTFT under 3 seconds
  • Test: 4096-token prompt, 128-token generation, MacBook Pro M5 24GB

Quantization options: BF16, FP16, FP32, 2/4/8-bit affine, NVFP4 (via ExecuTorch delegate). 4-bit is the deployment standard.

Fine-tuning: mlx_lm.lora — LoRA and QLoRA on Apple Silicon, no CUDA required.

Known gaps vs CUDA:

  • No PagedAttention / continuous batching in base MLX — Ollama adds this via its March 2026 MLX integration
  • No FSDP equivalent for distributed training
  • Long-context (100K+): MLC-LLM handles better than base MLX
  • arXiv:2511.05502 (Nov 2025): MLX leads vs Ollama/llama.cpp/MLC-LLM for most model sizes on Apple Silicon; llama.cpp competitive on 4-bit quantized inference with better portability

Note: Apple added NVIDIA GPU backend to MLX in July 2025 (matrix multiplication, softmax, reduction, sorting, indexing on CUDA). Still in development — not a production parity claim.

Core ML vs MLX Decision

Dimension MLX Core ML
Best for Mac inference, fine-tuning, research iOS/iPadOS production deployment
Model selection 4,255+ on mlx-community Limited, growing
Fine-tuning LoRA/QLoRA native Not supported
Max model size Full unified memory (up to 192GB M4 Ultra) Constrained by iOS RAM limits
Enterprise use Developer workstations, air-gapped Macs Mobile app deployment

ExecuTorch MLX Delegate

ExecuTorch MLX delegate: 3–6× throughput improvement vs other ExecuTorch delegates on macOS for generative AI. Supports BF16/FP16/FP32 and 2/4/8-bit affine quantization and NVFP4.

Pipeline: torch.export → ExecuTorch edge IR → MLX delegate partition → .pte for on-device execution.

5. ONNX Runtime GenAI: Cross-Platform Edge Path

What It Actually Provides

ONNX Runtime GenAI (onnxruntime-genai, v0.9.2 as of April 2026) is not just a format — it adds production LLM inference primitives: optimized KV cache with memory reuse, GQA/MHA/MLA attention kernels, INT4/INT8/QDQ quantization, and the Model Builder tool.

Model Builder conversion:

python -m onnxruntime_genai.models.builder \
  -m meta-llama/Meta-Llama-3.1-8B-Instruct \
  -o ./llama-ort \
  -p int4 \
  -e cpu   # or cuda, directml, rocm

Confirmed working: Phi-2, Phi-3.x, CodeLlama, Llama-2, Mistral, Gemma, Qwen2. Also accepts GGUF input files.

Performance: ORT claims 2–3× faster than native PyTorch for supported models. Custom CK Flash Attention for ORT on ROCm reduced end-to-end latency from 4,918ms to 127ms (38.7× improvement) in a published 2025 paper. CUDA path trails TensorRT-LLM by 20–40% throughput but is far simpler to operate.

Known Gaps

Gap Detail
Dynamic shapes Single largest limitation — hardware NPU backends require static shapes; symbolic dimensions work on CPU/CUDA but not QNN-HTP
Custom ops Flash Attention v2, Triton kernels, RoPE variants → no standard ONNX representation; decompose to primitives (perf cost) or register custom ops (breaks portability)
MLA attention DeepSeek-style multi-head latent attention lacks ORT GenAI kernel; MHA and GQA are supported
Quantization encoding ONNX QDQ format differs from GPTQ/AWQ/GGUF; cross-format conversion may require re-calibration
Opset version fragmentation Opset 21 (current); older ORT deployments may not support new opset features

QNN-HTP dynamic shape issue: Confirmed open GitHub issue (onnxruntime #23832) — QNN HTP backend requires static shapes, causing compilation failures for LLMs with variable sequence lengths.

2026 improvements: ONNX opset 21 better LLM primitive support; AMD Quark integration for Ryzen AI NPU quantization targeting ORT GenAI; LFM2-12B and other new models shipping ONNX as first-class format.

6. Qualcomm Snapdragon NPU / QNN / AI Hub

Conversion Pipeline

Two levels:

AI Engine Direct (low-level, maximum control):

PyTorch/ONNX → qnn-pytorch-converter → QNN C++ graph
  → quantization (50–200 sample calibration dataset)
  → .dlc binary → QNN runtime

AI Hub (managed, simplest):

import qai_hub
model = qai_hub.get_model("llama-3-2-3b-chat-quantized")
model.export(device="Snapdragon X Elite")

AI Hub handles quantization, compilation, deployment. QNN SDK 2.28+ fixed upload timeout issues for Llama family specifically.

Performance Reality

The inversion: Snapdragon X Elite NPU = 2.64 tok/s for Llama 3.2 3B. CPU = 25.88 tok/s. The NPU is 10× slower for LLM decode. Root cause: NPU tiling architecture optimized for vision/batch workloads; autoregressive decode is memory-bandwidth-bound and sequential — a poor fit.

Qualcomm’s own response (April 2026): Published QMX (Qualcomm Matrix Extensions) work targeting CPU acceleration for Llama — up to 19.0× speedup for mixed-precision GEMM, 2.2× for Softmax. This signals that CPU is the intentional LLM inference path on Snapdragon, not NPU.

Mobile (Snapdragon 8 Elite) vs desktop (X Elite): Mobile shows better NPU LLM performance (~10 tok/s for 3B) than X Elite (~2.64 tok/s) — different driver maturity, different NPU architecture generations.

Dynamic shape hard wall: QNN-HTP backend does not support dynamic shapes. Fixed-length compilation required for all deployments.

Confirmed AI Hub models: Llama 3.2 3B Instruct, Llama 3.1 8B Instruct, Phi-3.5 Mini Instruct, Llama 3.2 3B Chat quantized.

Architecture Gaps and Hard Incompatibilities

The sections above cover conversion pipelines and performance. This section documents confirmed architecture-level incompatibilities — cases where a model or quantization format silently degrades, crashes, or delivers the opposite of its intended performance on a given backend. These are not configuration issues; they are structural mismatches between model design and hardware execution model.

1. bitsandbytes NF4 — CUDA-Only in All Production PyPI Releases

HuggingFace load_in_4bit=True / bnb_4bit_quant_type="nf4" silently fails or errors on non-NVIDIA backends. bitsandbytes is CUDA-bound in every production PyPI release as of May 2026.

Blocked backends:

  • AMD ROCm: source-build-only workaround; consumer RDNA LoRA fine-tuning effectively blocked (noted in Section 2)
  • AWS Neuron: not supported (confirmed in Section 1 quantization table)
  • Intel OpenVINO / NPU: not supported
  • Qualcomm QNN: not supported

Enterprise impact: Most HuggingFace model cards that advertise “requires 8GB VRAM” assume NF4 quantization. Those models are CUDA-only unless reformatted to GGUF, AWQ, or GPTQ for the target backend. Teams that pull model cards at face value and deploy to non-NVIDIA hardware will find models that either error on load or silently consume 2× the expected VRAM.

2. MoE Sparse Routing — Backend Support Matrix

Mixture-of-Experts models (Mixtral 8x7B, Qwen3 235B, DeepSeek-V3) rely on conditional expert dispatch: only 2–8 experts activate per token instead of the full parameter set. Backend support for this conditional routing varies significantly.

Backend MoE Support Notes
NVIDIA CUDA Full TensorRT-LLM, vLLM native
AMD ROCm Full PyTorch HIP; vLLM ROCm supports all MoE architectures
AWS Neuron Full (with config) GroupLimitedRouter + selective expert loading; Mixtral 8x7B, Qwen3 235B A22B documented
Intel Gaudi3 Partial MoE supported; custom kernel required for expert routing
Apple MLX Full mlx-community has Mixtral and Qwen MoE models converted
ONNX Runtime Regresses MoE converted to dense mask-based equivalent — all experts activated; sparsity benefit eliminated
Qualcomm QNN Not Supported Cannot do conditional dispatch; MoE runs as dense

The Qualcomm and ONNX cases are critical:

  • Mixtral 8x7B on QNN executes as a 47B dense equivalent, not the expected ~12B active-parameter model. VRAM and compute are 4× the advertised requirement.
  • ONNX Runtime converts MoE routing to mask-based dense ops. A model marketed as “8×7B, 12B active” becomes a full 47B forward pass. The architectural purpose of MoE — compute efficiency — is completely negated.

3. ONNX Linear Attention / SSM / Mamba — 10–50× Throughput Penalty

ONNX has no native operator for selective scan, the core computational primitive in Mamba and related State Space Models. When these models are exported to ONNX, the selective scan is decomposed into primitive elementwise and reduction ops, producing catastrophic throughput regression.

Affected architectures: Mamba-2, Jamba, Zamba, RWKV-6, and any SSM-based or linear attention architecture that relies on selective scan for its efficiency.

Impact: ONNX Runtime is not viable for linear attention models on any backend. The throughput penalty (10–50× vs native PyTorch) eliminates the latency and memory advantages that are the entire reason these architectures exist. The only correct deployment path for SSM-based models is a backend with native selective scan support (CUDA via mamba-ssm, MLX via custom kernels, or llama.cpp with GGUF conversion).

4. CoreML ANE — 4B Parameter Ceiling

Apple’s Neural Engine (ANE) has an internal buffer size limit that gates which models actually run on the ANE versus falling back to Metal GPU.

Constraints:

  • FP16: ~4B parameter ceiling for ANE execution
  • 4-bit quantization: ~8B parameter ceiling
  • macOS 15+ required for fused Scaled Dot-Product Attention (SDPA); older macOS uses elementwise attention, which is materially slower

Practical consequences:

Model ANE Eligible?
Llama 3.2 3B (4-bit) Yes
Phi-3.5 Mini 3.8B (4-bit) Borderline
Phi-4 14B No — routes to Metal GPU
Llama 3.1 8B (FP16) No — routes to Metal GPU
Llama 3.1 8B (4-bit) Yes

Marketing materials for Apple Silicon inference often cite ANE TOPS numbers. For any model above ~4B in FP16 or ~8B in 4-bit, those numbers are irrelevant — the model runs on Metal GPU. MLX performance figures remain accurate (MLX uses Metal GPU directly), but Core ML deployments targeting ANE are constrained to small models only.

5. Qualcomm QNN Group-wise Quantization Hardware Gap

AWQ and GPTQ — the two dominant 4-bit quantization formats in HuggingFace — both use group-size=128: weights are quantized in groups of 128, with separate scale/zero-point per group.

Qualcomm’s HMX accelerator (the high-performance matrix unit in Snapdragon NPUs) only accelerates per-channel quantization (group-size = full channel). Group-size=128 dequantization is handled by scalar engines, not HMX, eliminating the NPU’s entire performance advantage.

Result: AWQ/GPTQ models on Snapdragon NPU frequently run slower than CPU baseline in measured configurations. This compounds the general NPU autoregressive decode problem (see Section 6 below).

Workaround: Reformat to QNN-native per-channel quantization via the QNN SDK quantization pipeline. This format is incompatible with standard HuggingFace repos — models must be maintained in a separate QNN-quantized version. Teams that download standard AWQ/GPTQ checkpoints and deploy to QNN NPU will not recover NPU performance.

6. Intel and Qualcomm NPU — Autoregressive Decode Performance Inversion

This is the single most counterintuitive finding in the hardware portability landscape: running LLMs on the NPU is slower than running on CPU, across Intel, Qualcomm, and MediaTek NPU datasheets.

Confirmed data point (Intel OpenVINO benchmark):

  • Snapdragon X Elite NPU: 2.64 tok/s for Llama 3.2 3B
  • Snapdragon X Elite CPU: 25.88 tok/s for Llama 3.2 3B
  • NPU is 10× slower

Root cause: Transformer autoregressive decode is memory-bandwidth-bound and inherently sequential — one token generated before the next can begin. NPU tiling architectures are designed for parallel, batch-oriented workloads: CNNs, ViTs, ASR, TTS, image classification. They deliver excellent performance-per-watt for vision and batch embedding. They are structurally mismatched to sequential token generation.

Where NPUs do perform well: Background/always-on inference for wake-word detection, image classification, batch embedding, ASR, TTS — all workloads where the NPU’s parallel tile execution maps to the computation shape.

The enterprise implication: Copilot+ PC procurement decisions that cite “NPU TOPS” as the LLM inference metric are measuring the wrong thing. For LLM workloads, CPU tok/s is the relevant figure.

7. AWS Neuron — Static Compilation Hard Constraints

Neuron’s static NEFF compilation model (covered in Section 1) creates several hard incompatibilities beyond the general “must recompile” inconvenience:

Shape inflexibility at deployment:

  • A model compiled for batch_size=1, seq_len=2048 cannot serve batch_size=2 or seq_len=4096 requests without recompilation
  • Recompilation for 70B+ models: 45–90 minutes
  • Production deployments must pre-compile a bucket of shapes and route requests accordingly; any shape outside the bucket is unservable

Format incompatibilities:

  • bitsandbytes NF4: not supported (no workaround)
  • GPTQ/AWQ from HuggingFace: must be reformatted through the NxD quantization pipeline before Neuron can consume them
  • EAGLE v2 speculative decoding: not supported — dynamic draft trees are incompatible with static compilation

Operational consequence: Teams accustomed to “pull model card, load, serve” HuggingFace workflows must build a compilation and cache management layer before first deployment. This is not a one-time cost — every new model, every sequence length change, and every batch size change requires recompilation and cache invalidation.

8. Architecture × Backend Support Matrix

The table below summarizes confirmed support status across all six backends covered in this document. “Regresses” indicates the feature is accepted without error but produces materially worse outcomes than not using it (e.g., MoE on ONNX RT, group-128 on QNN).

Architecture / Feature NVIDIA CUDA AMD ROCm AWS Neuron Apple MLX ONNX RT GenAI Qualcomm QNN
Standard dense Transformer Full Full Full Full Full Full
MoE (sparse routing) Full Full Full Full Regresses Not Supported
SSM / Mamba / linear attention Full Full Partial Partial Regresses Not Supported
NF4 (bitsandbytes) Full Partial (source build) Not Supported Not Supported Not Supported Not Supported
AWQ group-128 Full Full Partial Full Full Regresses
GPTQ group-128 Full Full Partial Not Supported Full Regresses
Speculative decoding (EAGLE v2) Full Full Not Supported Partial Partial Not Supported
Dynamic batching Full Full Not Supported Partial Partial (CPU/CUDA) Not Supported
FP8 precision Full Full (MI300X+) Full Not Supported Not Supported Not Supported

Key for table values:

  • Full — supported with no material caveats
  • Partial — supported with documented limitations or requiring non-standard configuration
  • Regresses — accepted without error; produces worse performance than baseline (do not use)
  • Not Supported — fails at load time, compile time, or produces incorrect output

Enterprise Decision Matrix

Use Case Recommended Backend Why
AWS cloud inference at scale (≥70B) Neuron Trn2 + NxD Inference HBM capacity, cost at volume
On-prem server inference, NVIDIA alternative AMD MI300X + vLLM ROCm Drop-in, 192GB HBM, 15–30% below H100 cost
Developer workstation, Mac Apple Silicon + MLX Zero ops, 4,255+ models, LoRA included
Windows enterprise workstation NVIDIA RTX + llama-server or Ollama Best ecosystem, no conversion required
Cross-platform edge deployment ONNX Runtime GenAI CPU/CUDA/DirectML; avoid QNN for dynamic workloads
iOS app with on-device AI Core ML Only production-grade iOS LLM path
Copilot+ PC NPU (Intel/AMD) CPU (llama.cpp) NPU currently slower than CPU for LLM decode
Copilot+ PC NPU (Snapdragon) CPU (llama.cpp) 10× NPU performance inversion confirmed
Air-gapped compliance deployment MLX (Mac) or llama.cpp (NVIDIA/AMD Linux) No external calls, no cloud compilation

The Portability Ladder

From most portable to most locked-in:

  1. llama.cpp — GGUF format, runs on CUDA/ROCm/Metal/Vulkan/CPU. Most portable, not fastest
  2. ONNX Runtime GenAI — cross-platform but dynamic shape limitations for NPU targets
  3. MLX — Apple Silicon only but zero conversion friction, best Mac performance
  4. PyTorch + ROCm — AMD enterprise GPU drop-in from CUDA, framework-level portability
  5. optimum-neuron / NxD Inference — AWS cloud only, static compilation, 45–90 min compile
  6. QNN / AI Hub — Qualcomm devices only, static shapes, CPU often faster for LLMs
  7. TensorRT-LLM — NVIDIA only, maximum NVIDIA performance, no portability

Sources

Brandon Sneider | brandon@brandonsneider.com May 2026