Audience: Platform engineers and FinOps leads evaluating managed Bedrock inference vs self-hosted models on SageMaker or EC2.
TL;DR: Bedrock wins below ~200M tokens/month on commodity tasks. Self-hosting on SageMaker with Savings Plans wins above ~400–500M tokens/month at sustained utilization. The hybrid pattern — Bedrock for complex tasks, SageMaker for high-volume commodity inference — is the most defensible architecture at scale.
Bedrock Managed Inference (On-Demand)
| Model | Input ($/1M tokens) | Output ($/1M tokens) | Notes |
|---|---|---|---|
| Claude Haiku 4.5 | $1.00 | $5.00 | As of April 2026, up 20–35% from 2025 |
| Claude Sonnet 4.5 | $3.00 | $15.00 | Complex reasoning tier |
| Llama 3.3 70B (Bedrock) | $0.72 | $0.72 | Symmetric pricing; ~3x premium vs Together AI |
| Amazon Nova Lite | $0.06 | $0.24 | Cheapest managed option for lightweight tasks |
Bedrock cross-region inference adds ~10–15% latency but can improve availability; pricing is identical to single-region.
Bedrock Provisioned Throughput requires 1-month or 6-month commitments in Model Units (MUs). At scale, PTUs reduce effective per-token cost by 20–40% over on-demand but require accurate volume forecasting and carry idle-capacity risk.
SageMaker Real-Time Inference — Instance Pricing (us-east-1, on-demand)
| Instance | GPUs | GPU Memory | On-Demand $/hr | SageMaker vs EC2 Equivalent |
|---|---|---|---|---|
| ml.g5.2xlarge | 1× A10G (24GB) | 24 GB | $1.52 | ~25% premium vs EC2 g5.2xlarge ($1.21/hr) |
| ml.g5.12xlarge | 4× A10G (96GB) | 96 GB | $5.67 | ~23% premium vs EC2 g5.12xlarge ($4.59/hr) |
| ml.g5.48xlarge | 8× A10G (192GB) | 192 GB | $16.29 | ~21% premium |
| ml.p4d.24xlarge | 8× A100 (320GB) | 320 GB | $32.77 | ~18% premium; post-June 2025 45% price cut applied |
| ml.p5.48xlarge | 8× H100 (640GB) | 640 GB | $98.32 | Overkill for pure inference |
| ml.inf2.8xlarge | 1× Inferentia2 chip | 32 GB | $0.97 | No direct GPU equivalent |
| ml.inf2.48xlarge | 12× Inferentia2 chips | 384 GB | $12.98 | ~60% cheaper than p4d for compatible workloads |
SageMaker bills per-second with a 1-minute minimum. Endpoints that are idle still accrue full compute cost.
EC2 On-Demand (us-east-1, Linux)
| Instance | On-Demand $/hr | Notes |
|---|---|---|
| g5.12xlarge | $4.59 | Most common for 70B inference |
| p4d.24xlarge | ~$27.70 | Post-2025 price reduction |
| inf2.48xlarge | $12.98 | Same instance family, EC2 = SageMaker for Inf2 |
The SageMaker premium is 15–40% over raw EC2 depending on instance type and workload pattern. For g5.12xlarge: $5.67/hr (SageMaker) vs $4.59/hr (EC2) = ~23.5% premium. This premium buys managed endpoint health, rolling deploys, auto-scaling integration, and Model Monitor — valuable for teams without dedicated MLOps but an overhead tax for mature platform teams.
2. Break-Even Analysis: Self-Hosted Llama 3 70B vs Bedrock
Throughput Baselines
Throughput on Llama 3 70B varies significantly by instance and serving stack:
| Instance | Serving Stack | Throughput (tokens/sec) | Notes |
|---|---|---|---|
| ml.g5.12xlarge (4× A10G) | vLLM (BF16) | 120–180 tok/sec | Batch size 8–16; limited by 96GB HBM |
| ml.g5.12xlarge | llama.cpp (Q4) | 60–90 tok/sec | 4-bit quant; Llama 2 benchmark: 4.08 tok/sec single-user, ~90 tok/sec batched |
| ml.p4d.24xlarge (8× A100) | vLLM (BF16) | 600–900 tok/sec | Near-linear scaling over g5; preferred for high-concurrency |
| ml.inf2.48xlarge (12 chips) | Optimum Neuron | 200–250 tok/sec | Llama 3.3 70B Hugging Face benchmark: ~223 tok/sec |
Llama 2 70B on g5.12xlarge with llama.cpp (8-bit): 2.07 tok/sec single-user. Batched throughput via vLLM is dramatically higher — single-user latency benchmarks should not be used for capacity planning.
Monthly Token Capacity by Instance
Assuming continuous operation (720 hours/month):
| Instance | Throughput (tok/sec) | Monthly capacity (1M tokens) | On-Demand monthly cost |
|---|---|---|---|
| ml.g5.12xlarge | 150 tok/sec | 388,800 M tokens / month … | Wait — let me restate at realistic utilization |
At 150 tok/sec continuous: 150 × 3600 × 720 = 388,800,000 tokens/month = 388.8B tokens/month. That is theoretical maximum. At real utilization rates:
| Utilization | Effective throughput | Monthly tokens (g5.12xlarge) | Effective $/hr (amortized) |
|---|---|---|---|
| 90% | 135 tok/sec | ~350B tokens | $5.67 |
| 70% | 105 tok/sec | ~272B tokens | $5.67 |
| 50% | 75 tok/sec | ~194B tokens | $5.67 |
Cost per 1M Tokens — Self-Hosted (SageMaker On-Demand)
Formula: Cost per 1M = (instance $/hr) / (tokens/sec × 3600) × 1,000,000
ml.g5.12xlarge at $5.67/hr:
| Utilization | Throughput (tok/sec) | Cost per 1M tokens |
|---|---|---|
| 90% | 135 | $11.65 |
| 70% | 105 | $14.97 |
| 50% | 75 | $20.95 |
ml.p4d.24xlarge at $32.77/hr (750 tok/sec at 70%):
| Utilization | Throughput (tok/sec) | Cost per 1M tokens |
|---|---|---|
| 90% | 675 | $13.47 |
| 70% | 525 | $17.32 |
| 50% | 375 | $24.24 |
ml.inf2.48xlarge at $12.98/hr (220 tok/sec at 70%):
| Utilization | Throughput (tok/sec) | Cost per 1M tokens |
|---|---|---|
| 90% | 198 | $18.22 |
| 70% | 154 | $23.42 |
| 50% | 110 | $32.77 |
Inferentia2 is most cost-effective above 70% utilization for Llama 3 70B, but the hardware lock-in (Neuron SDK compile step, ~1–2 hour compile per model version, no CUDA ecosystem) adds operational friction.
With SageMaker Savings Plans (3-year, all-upfront)
SageMaker Savings Plans discount up to 64% vs on-demand for real-time inference endpoints:
| Commitment | Typical discount | g5.12xlarge effective $/hr |
|---|---|---|
| 1-year no-upfront | ~32% | $3.86 |
| 1-year all-upfront | ~40% | $3.40 |
| 3-year no-upfront | ~52% | $2.72 |
| 3-year all-upfront | ~64% | $2.04 |
g5.12xlarge cost per 1M tokens with 3-year all-upfront Savings Plan:
| Utilization | Cost per 1M tokens |
|---|---|
| 90% | $4.19 |
| 70% | $5.39 |
| 50% | $7.54 |
Break-Even Against Bedrock
Bedrock Llama 3.3 70B on-demand: $0.72/1M tokens (input+output symmetric). Bedrock Claude Haiku 4.5: $1.00 input / $5.00 output (~$3.00 blended at 1:1 ratio).
Self-hosted Llama 3 70B never beats Bedrock Llama on-demand pricing on a pure per-token basis unless you are at sustained high utilization with Savings Plans committed. But the use case comparison is different: Bedrock Llama is for tasks where you need zero operational overhead; self-hosted Llama with Savings Plans is for tasks where you fine-tune, need data residency, or batch-process at very high volume.
The realistic break-even for self-hosting over Bedrock managed (at equivalent quality):
| Scenario | Bedrock cost/1M | Self-hosted (SM + 3yr SP, 70% util) | Break-even volume (tokens/month) |
|---|---|---|---|
| Llama 3 70B vs Llama 3 70B | $0.72 | $5.39 | Self-hosting never wins on cost alone until fine-tuning or residency needed |
| Llama 3 70B vs Claude Haiku 4.5 (blended) | $3.00 | $5.39 | Break-even: ~100M tokens if you accept capability downgrade |
| Llama 3 70B vs Claude Haiku 4.5 (output-heavy) | $5.00 (output) | $5.39 | Near break-even at 70% utilization with 1yr SP |
| Fine-tuned Llama vs Claude Haiku 4.5 | $3.00 | $3.40 (1yr all-up SP, 90% util) | Breaks even at ~500M tokens/month |
Practical conclusion: self-hosting beats Bedrock managed inference when:
- You have a fine-tuned model (Bedrock can’t serve your weights without custom model import, which adds cost)
- Volume exceeds ~500M tokens/month on commodity tasks at 70%+ sustained utilization
- You commit to 1-year SageMaker Savings Plans
- The capability match (Llama vs Claude Haiku equivalent) holds for your task type
3. SageMaker Inference Cost Structure
Billing Model
- Billed per-second with a 1-minute minimum
- Endpoint stays running until you delete it — idle time costs the same as active time
- Multi-AZ deployments run two instances by default; always check
ProductionVariant.InitialInstanceCount - Data capture (Model Monitor) writes to S3; adds minor cost but is non-trivial at high request volumes (~$0.023/GB in S3 standard)
Instance Selection by Model Size
| Model Size | Parameters | Minimum Instance | Recommended Instance | Notes |
|---|---|---|---|---|
| 7B | ~14GB FP16 | ml.g5.xlarge (24GB) | ml.g5.2xlarge | Fits single A10G |
| 13B | ~26GB FP16 | ml.g5.2xlarge (24GB)* | ml.g5.12xlarge | *Requires quantization on single A10G |
| 34B | ~68GB FP16 | ml.g5.12xlarge (96GB) | ml.g5.12xlarge / ml.p4d.24xlarge | Fits 4× A10G |
| 70B | ~140GB FP16 | ml.g5.48xlarge or ml.p4d.24xlarge | ml.p4d.24xlarge (higher throughput) | g5.48xlarge is 8× A10G for 70B |
| 405B | ~810GB FP16 | ml.p4d.24xlarge × 3 (multi-node) | ml.p5.48xlarge | Requires tensor parallel + EFA |
Quantized 7B (Q4) fits ml.g5.xlarge at 4.55GB. Q4 13B fits ml.g5.2xlarge at 7.9GB.
Multi-Model Endpoints (MME) vs Dedicated Endpoints
Dedicated endpoints: One model, one endpoint. Best for: high-throughput production, predictable latency, models requiring full GPU memory.
Multi-Model Endpoints (MME): Multiple models share one endpoint. Models load/unload from GPU memory dynamically. Best for: dozens of fine-tuned variants with uneven traffic, A/B testing fleets.
SageMaker Inference Components (IC): Introduced 2023, GA 2024. Fractional GPU allocation — assign specific GPU counts and memory per model on a shared endpoint. Key capability for LLM hosting:
- Salesforce reported 8× cost reduction hosting multiple LLMs via IC vs dedicated endpoints
- A single ml.g5.2xlarge can host 4 models sharing the 24GB A10G — reduces per-model endpoint cost from $1.52/hr to $0.38/hr effective per-model
- Requires careful memory planning: model weights + KV cache + serving overhead must fit allocated fraction
- Not suitable for models that spike to full GPU memory during prefill of long contexts
SageMaker JumpStart
JumpStart provides one-click deployment for 600+ foundation models. Cost model:
- Compute cost: Same SageMaker instance rates apply
- JumpStart model surcharge: Some proprietary or commercially-licensed models add a per-token or per-hour surcharge on top of compute. Open models (Llama, Mistral, Falcon) have no surcharge.
- Foundation Model Hub: As of 2025, AWS expanded FM Hub to include models with Bedrock-like managed serving. These carry a per-token charge even when accessed through SageMaker APIs — read the model card pricing before deployment.
- Container startup time: JumpStart containers for 70B models can take 8–15 minutes to pull and initialize. For endpoints with auto-scaling, this cold-start time creates latency spikes during scale-out events. Mitigation: keep
MinInstanceCount >= 1and use SageMaker async inference with SQS buffering for burst traffic.
TGI vs vLLM on SageMaker
Both run as custom containers via SageMaker’s Large Model Inference (LMI) container:
| Serving Stack | Strengths | Weaknesses | SageMaker Support |
|---|---|---|---|
| TGI (Hugging Face) | Streaming, continuous batching, PEFT LoRA | Less aggressive memory optimization than vLLM | DLC container, tested on g5/p4d |
| vLLM | PagedAttention (highest memory efficiency), highest throughput, multi-LoRA | More complex to configure tensor parallelism | AWS-maintained fork; 19% OTPS improvement over upstream vLLM 0.15.0 per AWS benchmarks |
| DJL Serving | Best for Inferentia2 (Neuron SDK integration) | Limited community vs vLLM | Official DJL LMI containers |
For Llama 3 70B on SageMaker: vLLM with AWS’s optimized container is the recommended stack for GPU instances. DJL with Neuron SDK for Inferentia2.
4. EC2 vs SageMaker for Inference
When to Skip SageMaker
SageMaker’s managed layer adds 15–40% cost premium over raw EC2. The premium is justified when:
- Team lacks MLOps expertise to build health checks, rolling deploys, and auto-scaling
- Model Monitor is required for compliance/drift detection
- Using SageMaker Savings Plans (which don’t apply to EC2)
Skip SageMaker and run on EC2 when:
- You already operate Kubernetes + Karpenter for autoscaling
- You need Spot instances (SageMaker async inference supports Spot; real-time endpoints do not)
- You want to run vLLM with full control over GPU topology and NCCL settings
- Multi-node inference for 405B+ models (EC2 + EFA is simpler than SageMaker multi-node)
EC2 Pricing Reference (us-east-1, on-demand, Linux)
| Instance | GPUs | On-Demand $/hr | 1yr Reserved $/hr | Spot $/hr (typical) |
|---|---|---|---|---|
| g5.xlarge | 1× A10G | $1.006 | ~$0.63 | $0.30–0.45 |
| g5.12xlarge | 4× A10G | $5.672 | ~$3.55 | $1.70–2.50 |
| g5.48xlarge | 8× A10G | $16.288 | ~$10.20 | $4.90–7.20 |
| p4d.24xlarge | 8× A100 | ~$27.70 | ~$17.40 | $8–14 (low availability) |
| p5.48xlarge | 8× H100 | ~$98.32 | ~$61.70 | Rarely available on Spot |
| inf2.xlarge | 1 chip | $0.758 | ~$0.47 | $0.23–0.35 |
| inf2.48xlarge | 12 chips | $12.98 | ~$8.14 | Rarely Spot-available |
Spot Instance Risk for LLM Inference
Spot instances offer 50–70% discounts but carry preemption risk. For LLM inference:
- Short context requests (<2K tokens): Acceptable Spot risk — request completes in <200ms, preemption impact is minor
- Long context requests (32K+ tokens): Preemption mid-request loses all in-flight computation. Use On-Demand or Reserved for these workloads
- Async batch processing: Ideal for Spot — jobs can be retried on SQS re-queue with no user impact
- p4d/p5 Spot availability: Low. Spot capacity for P4d in us-east-1 is frequently unavailable at scale. Build for On-Demand and treat Spot as opportunistic savings, not baseline cost.
EFA for Multi-Node Inference
For 405B parameter models requiring tensor parallelism across nodes:
- EFA (Elastic Fabric Adapter) provides up to 400 Gbps bandwidth for node-to-node communication
- p4d.24xlarge and p5.48xlarge include EFA; g5 instances do not
- SageMaker multi-node inference (MNI) supports EFA but requires specific cluster configuration
- Alternative: EC2 with AWS ParallelCluster + NCCL over EFA — lower overhead, full control
- Typical setup for 405B: 4× p4d.24xlarge with EFA, vLLM tensor_parallel_size=32
Cost Delta: SageMaker Managed vs Raw EC2
For g5.12xlarge: $5.672/hr (SageMaker) vs $4.59/hr (EC2) = $1.082/hr premium = $779/month per endpoint.
At 10 endpoints running 24/7: $7,788/month in SageMaker overhead vs raw EC2. At this scale, the operational savings from managed infrastructure need to justify ~$93K/year in overhead.
Rule of thumb: SageMaker overhead pays for itself if it saves 1 FTE-week/quarter of MLOps work. At $200K fully-loaded eng cost, that’s ~$12.5K/quarter. The SageMaker overhead at 10 endpoints is ~$23K/quarter — marginal. At 3–5 endpoints, the math clearly favors SageMaker managed.
5. Operational Costs Beyond Compute
Model Storage
Llama 3 70B model weights:
- FP16: ~140GB → S3 Standard: $3.22/month (140GB × $0.023/GB)
- Q4 quantized: ~35GB → $0.81/month
- Multiple versions (3 checkpoints): ~$9.66/month FP16
This is trivially small relative to compute costs. Storage is not a meaningful cost driver.
Container Registry (ECR)
Custom inference containers:
- ECR storage: $0.10/GB/month. A custom vLLM container image is ~8–15GB compressed = $0.80–$1.50/month
- ECR data transfer to SageMaker within same region: free
- Cross-region replication for multi-region deployment: $0.08–0.09/GB/transfer
VPC Data Transfer
- SageMaker endpoint → same-region VPC: no transfer charge
- Cross-AZ traffic (e.g., application tier in us-east-1a, SageMaker in us-east-1b): $0.01/GB each way
- At 1M requests/day × 2KB average response: ~2GB/day cross-AZ = $0.60/day = $18/month
SageMaker Model Monitor
- Data capture: writes request/response logs to S3 at $0.023/GB
- At 1M requests/day × 4KB (request+response): ~4GB/day = $2.76/day = $83/month
- Monitor processing jobs: ~$0.10–$0.20/hour for the monitoring job compute (runs hourly by default)
- Monthly monitoring overhead: $125–200/endpoint at moderate traffic
CloudWatch Logging
- SageMaker endpoint logs (container logs): $0.50/GB ingestion
- At 1M requests/day, verbose logging at 500B/request: 500GB/day = $250/day if you log everything. Critical: set log level to WARNING in production or route to Firehose → S3 instead.
- Recommendation: log 1% of requests to CloudWatch, 100% to S3 via data capture. Monthly CloudWatch cost: $5–15/endpoint.
Engineering Overhead
This is the largest non-compute cost and the most underestimated:
| Activity | Self-hosted SageMaker | Bedrock Managed |
|---|---|---|
| Initial setup | 2–4 weeks FTE | 1–2 days |
| Model updates | 4–8 hours per major version | Automatic (with API version pinning) |
| Incident response (OOM, endpoint failures) | 2–4 hours/incident | Near-zero (AWS handles) |
| Security patching (container images) | Monthly, 4–8 hours | Automatic |
| Auto-scaling tuning | Ongoing, 2–4 hours/quarter | Managed |
| Ongoing maintenance FTE fraction | 0.15–0.25 FTE | ~0.01 FTE |
At $200K fully-loaded senior engineer cost:
- Self-hosted SageMaker: 0.20 FTE = $40K/year in maintenance overhead
- Bedrock: 0.01 FTE = $2K/year
$38K/year engineering delta must be factored into every break-even calculation. This alone shifts the break-even point significantly upward.
Adjusted break-even including engineering overhead:
At 0.20 FTE overhead for SageMaker maintenance ($3,333/month), the effective cost floor on a single g5.12xlarge endpoint with 1yr Savings Plan:
- Compute: $3.40/hr × 730 hr/month = $2,482/month
- Engineering overhead (prorated per endpoint): $3,333/month (if single endpoint)
- Total: $5,815/month
To break even against Bedrock Haiku at $3.00/1M (blended): need 1,938M tokens/month just to recover the fixed overhead. This improves dramatically when overhead is spread across multiple endpoints or high utilization.
6. Hybrid Architecture Patterns
The 80/20 Split Pattern
The highest ROI architecture in 2026: route 80% of tokens to self-hosted Llama (commodity extraction/classification/summarization), 20% to Bedrock Claude (complex reasoning, generation, agentic tasks).
Example: enterprise document processing at 10M tokens/day
| Component | Volume | Use Case | Unit Cost | Monthly Cost |
|---|---|---|---|---|
| Bedrock Claude Haiku 4.5 | 2M tokens/day (20%) | Complex extraction, entity resolution, quality check | $3.00/1M blended | $180/month |
| Self-hosted Llama 3 70B (SM + 1yr SP) | 8M tokens/day (80%) | Classification, chunk summarization, routing decisions | $5.39/1M at 70% util | $1,294/month |
| Total | 10M tokens/day | $1,474/month |
All-Bedrock equivalent:
- 2M/day Haiku: $180/month
- 8M/day Haiku: $720/month
- Total: $900/month
At 10M tokens/day, all-Bedrock Haiku is actually cheaper than hybrid — the self-hosted overhead dominates at this scale. The hybrid wins when:
- Volume reaches 50M+ tokens/day on the commodity tier
- The commodity tier uses a fine-tuned model (Bedrock can’t serve it without custom import)
- Latency requirements require <50ms p50 (Bedrock p50 is typically 80–200ms for Haiku)
Breakeven for hybrid (50M tokens/day commodity, 10M/day complex):
All-Bedrock Haiku: (60M × 30 × $3.00/1M) = $5,400/month
Hybrid:
- Bedrock Claude: 10M × 30 × $3.00 = $900
- SageMaker Llama (1yr SP, 70% util): 50M × 30 / 1M × $5.39 = $8,085 compute
- Engineering overhead: $3,333
- Total: $12,318/month
Still more expensive! The hybrid pattern is definitively cheaper only when:
- The commodity Llama model is fine-tuned (Bedrock equivalent would be Claude Sonnet at $9/1M output)
- Volume is very high and instance utilization exceeds 80%
Fine-tuned model hybrid (50M tokens/day commodity, quality match to Sonnet):
- All-Bedrock Sonnet: 50M × 30 × ($3+$15)/2 = $13,500/month commodity + $900 complex = $14,400/month
- Hybrid: $12,318/month → 15% savings
LiteLLM as Routing Layer
LiteLLM proxy (pip install litellm[proxy]) provides OpenAI-compatible endpoint with model routing:
# litellm proxy config for hybrid routing
model_list:
- model_name: complex-task
litellm_params:
model: bedrock/anthropic.claude-haiku-4-5-20251001-v1:0
aws_region_name: us-east-1
- model_name: commodity-task
litellm_params:
model: sagemaker/llama3-70b-finetuned
aws_region_name: us-east-1
router_settings:
routing_strategy: cost-based-routing # routes to cheapest available
fallback_model: complex-task # fallback if SageMaker unhealthy
Traffic routing logic:
- Rule-based routing: Classify requests by prompt template or metadata tag. Extraction/classification → Llama. Generation/reasoning → Claude.
- Cost-based routing: LiteLLM tracks token costs in real-time; routes to cheapest provider meeting latency SLA.
- Fallback routing: SageMaker endpoint failure → automatic Bedrock fallback. Critical for production reliability.
- Load-aware routing: LiteLLM checks queue depth; if SageMaker endpoint is saturated, overflow to Bedrock on-demand.
AWS’s Multi-Provider GenAI Gateway reference architecture (2026) formalizes this pattern with LiteLLM deployed on ECS Fargate behind API Gateway, with WAF and Cognito for auth.
Traffic Shaping by Request Type
Routing decision tree:
├── Has fine-tuning requirement → SageMaker (always)
├── Prompt length > 8K tokens → Bedrock Claude (better long-context)
├── Requires streaming < 50ms TTFT → SageMaker (lower cold-start on warm endpoint)
├── Batch / async job → SageMaker async inference or Bedrock Batch API
├── One-off / low volume → Bedrock on-demand (no idle compute)
└── High-volume classification → SageMaker with Savings Plan
Bedrock Batch API
For offline/async processing, Bedrock Batch API (GA 2024) provides up to 50% discount over on-demand:
- Claude Haiku batch: ~$0.50 input / $2.50 output per 1M tokens
- Minimum latency: hours (not real-time)
- Use case: nightly document processing, embedding pipelines, eval runs
- At scale, Bedrock Batch can undercut SageMaker on total cost for non-real-time workloads
7. Decision Framework
Primary Decision Matrix
| Criterion | Bedrock Wins | SageMaker Wins | EC2 Direct Wins |
|---|---|---|---|
| Monthly token volume | <200M | 200M–2B | >2B (with Savings Plans + own autoscaling) |
| Model customization | None needed | Fine-tuning required | Fine-tuning + custom serving infra |
| ML team size | 0–2 ML engineers | 2–5 ML engineers | 5+ ML/Platform engineers |
| Latency SLA (p99) | >200ms acceptable | 50–200ms required | <50ms required with full control |
| Latest model access | Critical | Less important | Not important |
| Data residency | Single-region Bedrock acceptable | VPC-isolated endpoint required | Air-gapped or on-prem required |
| Cost sensitivity | Low volume, cost not primary | High volume, Savings Plans viable | Extreme volume, every dollar counts |
| Operational risk tolerance | Low | Medium | High (you own the pager) |
When Bedrock is Always Right
- No ML team: Zero ops overhead for managed inference. Bedrock just works.
- Low volume (<200M tokens/month): Fixed infrastructure costs exceed token savings.
- Latest frontier models required: Bedrock gets Claude updates automatically. Self-hosting requires model upload and redeploy.
- Rapid iteration / prototyping: No endpoint provisioning latency. Call and go.
- Compliance-friendly managed service: Bedrock is SOC 2, ISO 27001, HIPAA-eligible. Self-hosting requires you to certify your own infrastructure.
- Unpredictable traffic patterns: Bedrock scales to zero and to burst without idle compute charges.
When Self-Hosting Definitively Wins
- Volume >500M tokens/month on commodity tasks: With SageMaker Savings Plans at 1yr all-upfront, self-hosted Llama on g5.12xlarge breaks even against Bedrock Haiku around 500M tokens/month including engineering overhead.
- Fine-tuned models are required: Bedrock Custom Model Import exists but carries compute overhead; your own SageMaker endpoint with your weights is more cost-efficient.
- Strict data residency: No third-party managed service acceptable. All inference must stay in your VPC.
- Sub-50ms P50 latency at high concurrency: Warm, dedicated endpoints with vLLM PagedAttention beat Bedrock cold starts for high-concurrency real-time serving.
- Offline/batch at extreme scale: SageMaker async inference + Spot instances for batch processing can achieve $1–3/1M tokens at high utilization, undercutting even Bedrock Batch API.
Savings Plans Strategy
- Never deploy SageMaker inference without a Savings Plan if you intend to run >6 months. The 1-year no-upfront plan (32% discount) requires no cash commitment and saves ~$1/hr on a g5.12xlarge — payback in weeks.
- Use AWS Cost Explorer’s Savings Plan recommendation tool quarterly. It analyzes your actual SageMaker usage and models the optimal commitment level.
- Do not over-commit: Savings Plans cover SageMaker instance hours, not specific instances. Unused hours don’t roll over. Model break-even with 70–80% forecast utilization confidence before committing to 3-year plans.
Inferentia2 Strategic Position
Inferentia2 is compelling for specific workloads:
- Best fit: High-volume, single-model, stable workload (e.g., 70B Llama for classification, running 24/7)
- Avoid for: Models that change frequently (each model version requires 1–2 hour Neuron compilation), multi-model serving, workloads needing CUDA-ecosystem integrations
- Cost advantage over GPU: inf2.48xlarge at $12.98/hr vs p4d.24xlarge at $32.77/hr for comparable Llama 70B throughput (~200–250 tok/sec on Inf2 vs 750+ tok/sec on p4d) — Inf2 wins on absolute throughput-per-dollar only for specific batch/throughput workloads, not latency-sensitive serving
Summary Thresholds (Rule of Thumb)
< 50M tokens/month → Bedrock on-demand, no self-hosting
50M–200M tokens/month → Bedrock + explore Bedrock Provisioned Throughput
200M–500M tokens/month → Evaluate hybrid; SageMaker viable if fine-tuning required
500M–2B tokens/month → SageMaker Savings Plans for commodity tier; Bedrock for complex tier
> 2B tokens/month → EC2 + custom autoscaling; SageMaker overhead becomes material
Sources and Data Quality Notes
- Bedrock pricing: Claude Haiku 4.5 at $1.00/$5.00 per 1M in/out confirmed as of April 2026 (20–35% increase from 2025 rates). Llama 3.3 70B on Bedrock at $0.72/1M symmetric.
- SageMaker instance pricing: ml.g5.12xlarge at $5.67/hr; ml.p4d.24xlarge at $32.77/hr; ml.inf2.48xlarge at $12.98/hr (us-east-1). P4/P5 pricing reflects June 2025 45% reduction.
- EC2 g5.12xlarge: $4.59/hr (economize.cloud, June 2026).
- Throughput figures: Llama 2 70B on g5.12xlarge (4× A10G) with llama.cpp: 2.07 tok/sec single-user, ~4.08 tok/sec Q4; batched vLLM throughput estimated from scaling factors. Inf2.48xlarge Llama 3.3 70B: ~223 tok/sec (HuggingFace Optimum Neuron benchmarks).
- SageMaker vs EC2 premium: 15–40% range from multiple sources; g5.12xlarge specific delta confirmed at ~23.5%.
- Savings Plans discount: Up to 64% for 3-year all-upfront; 32% for 1-year no-upfront (AWS documentation and Redress Compliance analysis).
- Engineering overhead estimates: Based on industry norms; actual costs vary significantly by team experience.
Pricing verified against AWS pricing pages and third-party cost analysis sources. Verify current prices before committing to infrastructure decisions — AWS pricing changes quarterly.