← Agent Frameworks 🕐 19 min read
Agent Frameworks

Bedrock Intelligent Model Routing for Cost Optimization (2026)

1. AWS Intelligent Prompt Routing (IPR) · March 2026

Model routing is the single highest-leverage cost control available to teams running LLM workloads at scale.

Model routing is the single highest-leverage cost control available to teams running LLM workloads at scale. The core insight: most production traffic is structurally simple — summarization, extraction, classification, short Q&A — yet teams default to a single capable model for everything. Routing 70–80% of that traffic to a 10–20× cheaper model while reserving frontier capability for the 20–30% that genuinely needs it produces 40–77% bill reductions with minimal quality regression when done with appropriate safeguards.

This note covers the full stack: AWS Bedrock’s managed Intelligent Prompt Routing (IPR), LiteLLM’s open-source router strategies, complexity classification methods, budget-pressure routing, A/B test methodology, fallback chains, and real-world patterns from production deployments.

How It Works

IPR provides a single serverless endpoint per configured model pair. For each incoming request, Bedrock:

  1. Analyzes the prompt for content and context
  2. Predicts response quality for each model in the configured pair
  3. Evaluates whether the quality difference exceeds the configured responseQualityDifference threshold
  4. Routes to the cheaper model when the quality difference falls below threshold; routes to the more capable model when the gap exceeds it
  5. Returns the response with model attribution in the response metadata

The routing decision logic is opaque — AWS does not publish the feature set used for prediction. The overhead added by the routing component is approximately 85ms at P90 latency, measured by AWS. This overhead is fixed and does not scale with prompt length.

IPR is optimized for English prompts only. It cannot incorporate application-specific performance data or fine-tuned model quality signals. For specialized workloads (legal, medical, domain-specific code), the generic routing predictor may underperform hand-tuned rules.

Pricing

  • $1.00 per 1,000 requests — flat fee, billed on top of underlying model inference costs
  • Token costs are billed at the rate of whichever model the request is routed to
  • The routing fee applies even when the cheaper model handles the request

Break-even analysis: If routing saves an average of $0.001 per request in model costs (reasonable for a Haiku/Sonnet split with ~500 tokens), the routing fee ($0.001/request) breaks even at 50% deflection rate. For high-volume workloads with mixed traffic, the savings margin typically exceeds the routing fee by 30–60×.

Supported Model Pairs (GA as of April 2025)

IPR supports any two models within the same family. GA-supported families and representative pairs:

Family Cheap Anchor Premium Model
Anthropic Claude 3 Haiku Claude 3.5 Sonnet / 3.5 Sonnet v2
Anthropic Claude 3.5 Haiku Claude 3.5 Sonnet v2
Amazon Nova Nova Lite Nova Pro
Meta Llama Llama 3.1 8B / 3.2 11B Llama 3.1 70B / 3.3 70B

Both single-region model IDs and cross-region inference profile IDs (prefixed us., eu., ap.) are supported. Cross-region profiles for the Nova family are available in 10 regions including all US, key EU, and AP regions.

Configuration API:

# Create a configured prompt router
aws bedrock create-prompt-router \
  --prompt-router-name "haiku-sonnet-prod" \
  --models '[{
    "modelArn": "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-5-haiku-20241022-v1:0"
  }]' \
  --fallback-model '[{
    "modelArn": "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0"
  }]' \
  --routing-criteria '{"responseQualityDifference": 0.1}'

The responseQualityDifference parameter (0.0–1.0) sets the threshold for upgrading. A value of 0.1 means: route to the fallback (more capable) model only when it would produce responses at least 10% better than the cheap model. Lower values route more aggressively to the capable model; higher values keep more traffic on the cheap model.

Reported Cost Savings by Family

From AWS benchmark data across representative workloads:

Family Avg Cost Reduction Notes
Anthropic (Haiku/Sonnet) 56% 60% measured on one internal test
Amazon Nova (Lite/Pro) 35% —
Meta Llama 16% Smaller capability gap limits savings
RAG-specific workloads 63.6% 87% of prompts routed to cheaper model

One DEV Community analysis of mixed-complexity workloads reported up to 94% cost reduction in cherry-picked scenarios where most traffic was structurally simple. That figure is not representative of a typical mixed production workload.

Reference benchmark: 100,000 requests/month, 500 input tokens, 400 output tokens, 70/30 simple/complex split on Nova Lite/Pro:

  • Without routing (all Nova Pro): ~$168/month
  • With IPR (70% Nova Lite, 30% Nova Pro + routing fee): ~$59/month + $100 routing = ~$159/month at this scale

At scale, the routing fee becomes relatively negligible. At 1M requests/month the routing fee is $1,000 but the deflection savings on the Nova pair at the same traffic profile would be ~$1,090/month — breakeven improves rapidly with volume.

CUR Appearance and Cost Attribution

Bedrock does not emit a separate line item for the IPR routing fee in CUR 2.0. The routing fee appears aggregated within Bedrock service charges. Individual inference costs appear at the model level — meaning a routed session that ends on Claude 3.5 Haiku charges at Haiku rates, not Sonnet rates.

CUR line_item_usage_type format:

{region}-{model-short-name}-{token-type}[-{service-tier}][-cross-region-global]

Examples from CUR 2.0:

USE1-Claude4.6Sonnet-input-tokens
USE1-Claude4.6Sonnet-output-tokens
USE1-Nova2.0Lite-input-tokens-flex
USE1-Claude4.6Sonnet-cache-read-input-token-count
USE1-Claude4.6Sonnet-output-tokens-cross-region-global

Key reconciliation rules:

  • Each request generates separate line items for input tokens, output tokens, and any cache read/write tokens. Summing only input+output undercounts cost on caching-heavy workloads.
  • In-region and cross-region inference carry different unit prices — apply matching rates per line item.
  • CUR 2.0 does not contain per-requestId line items. Requests are aggregated by usage type, operation, and hour/day. For per-prompt cost attribution, join model invocation logs to CUR at the model + usage-type grain.
  • Cost allocation tags from IAM principals (iamPrincipal/{key}) and application inference profiles (resourceTags/{key}) appear in CUR 2.0 only. Activate tags in AWS Billing console before expecting them in Cost Explorer; allow up to 24 hours.

Recommended CUR query for IPR savings analysis:

SELECT
  line_item_usage_type,
  SUM(line_item_usage_amount) AS tokens,
  SUM(line_item_blended_cost) AS cost
FROM bedrock_cur
WHERE line_item_usage_type LIKE '%Claude%'
   OR line_item_usage_type LIKE '%Nova%'
GROUP BY 1
ORDER BY cost DESC

Splitting by model name within usage_type reveals the actual cheap/expensive split that IPR is producing. A healthy IPR deployment shows the cheaper model handling materially more tokens.

2. LiteLLM Router Strategies for Cost Optimization

LiteLLM’s open-source proxy and router support five distinct routing strategies, each suited to different cost/performance tradeoffs. These operate independently of Bedrock IPR and can be layered over it.

Randomly selects from deployments weighted by rpm, tpm, or explicit weight values. No Redis dependency. Adds under 1ms overhead. Recommended for most production workloads due to lowest operational complexity.

model_list:
  - model_name: claude-haiku
    litellm_params:
      model: bedrock/anthropic.claude-3-5-haiku-20241022-v1:0
      aws_region_name: us-east-1
      rpm: 4000
      weight: 8   # 80% of traffic
  - model_name: claude-haiku
    litellm_params:
      model: bedrock/anthropic.claude-3-5-haiku-20241022-v1:0
      aws_region_name: us-west-2
      rpm: 4000
      weight: 2   # 20% of traffic (overflow region)

router_settings:
  routing_strategy: simple-shuffle

Weight-based traffic splitting under simple-shuffle is the recommended mechanism for controlled multi-region load balancing and for A/B testing model variants (see Section 6).

Strategy 2: cost-based-routing

Selects the deployment with the lowest estimated cost per request. Uses LiteLLM’s built-in cost map (litellm_model_cost_map) or custom input_cost_per_token / output_cost_per_token values. Models missing from the cost map default to $1/token (effectively always last in priority).

model_list:
  - model_name: smart-tier
    litellm_params:
      model: bedrock/anthropic.claude-3-5-haiku-20241022-v1:0
      input_cost_per_token: 0.0000008   # $0.80/1M
      output_cost_per_token: 0.000004   # $4.00/1M
  - model_name: smart-tier
    litellm_params:
      model: bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
      input_cost_per_token: 0.000003    # $3.00/1M
      output_cost_per_token: 0.000015   # $15.00/1M

router_settings:
  routing_strategy: cost-based-routing

Behavior: On each request, LiteLLM compares estimated cost (using token count if available from earlier in pipeline, or average token estimate) across healthy deployments (those below their rpm/tpm limits) and routes to lowest-cost. This strategy does not learn — it applies static cost rates without observing actual spend per call.

Use case: Multi-provider pools where you want to drain cheaper providers first. Not ideal for Haiku/Sonnet routing where you want quality-aware selection — use IPR for that, or a complexity classifier (Section 3).

Strategy 3: latency-based-routing

Routes to the deployment with the lowest cached response latency. Updates latency averages dynamically using a configurable TTL window.

router_settings:
  routing_strategy: latency-based-routing
  routing_strategy_args:
    ttl: 10                    # seconds — rolling latency window
    lowest_latency_buffer: 0.5 # 50% buffer — routes within 1.5× the fastest observed latency

Cost implication: Latency routing is cost-neutral by design but can produce indirect savings by reducing time-to-first-token, enabling tighter streaming timeouts and lower per-session infrastructure costs. The lowest_latency_buffer parameter prevents all traffic from piling onto a single fast endpoint — a 50% buffer means any deployment within 1.5× the fastest observed P50 is eligible.

Strategy 4: usage-based-routing-v2

Routes to the deployment with the lowest TPM usage in the current minute window. Requires Redis for distributed state. Adds meaningful latency overhead from async Redis calls (redis.incr, redis.mget).

model_list:
  - model_name: claude-pool
    litellm_params:
      model: bedrock/anthropic.claude-3-5-haiku-20241022-v1:0
      tpm: 100000
      rpm: 1000

router_settings:
  routing_strategy: usage-based-routing-v2
  redis_host: os.environ/REDIS_HOST
  redis_port: os.environ/REDIS_PORT
  redis_password: os.environ/REDIS_PASSWORD
  enable_pre_call_check: true

Production warning: Known bugs as of late 2025 cause this strategy to erroneously find no available deployment under certain load patterns (GitHub issue #16060). AWS recommends simple-shuffle with explicit tpm caps as the more stable alternative for rate-limit-aware routing. Use usage-based-routing-v2 only with dedicated Redis and active monitoring.

Strategy 5: least-busy

Routes to the deployment handling the fewest concurrent active requests. No Redis required — state is in-process. Best for workloads with high variance in per-request processing time (e.g., mixed short/long generation tasks) where you want to avoid stacking slow requests on a single endpoint.

router_settings:
  routing_strategy: least-busy

Cost relevance: Least-busy is latency-oriented, not cost-oriented. Relevant in Bedrock context where provisioned throughput units (PTU) have fixed cost regardless of utilization — maximizing PTU utilization minimizes effective cost per token.

Strategy Comparison

Strategy Redis Required Overhead Cost Savings Mechanism Recommended
simple-shuffle No <1ms Traffic split via weights ✓ Production default
cost-based-routing No <1ms Cheapest healthy deployment ✓ Multi-provider pools
latency-based-routing No <5ms Indirect via throughput Latency-critical
usage-based-routing-v2 Yes ~10ms+ Prevents TPM waste Caution — bugs
least-busy No <1ms PTU utilization PTU workloads

3. Request Complexity Classification and Routing Rules

The Core Problem with Prompt Length as Proxy

Length correlates weakly with complexity. A 50-token instruction to rewrite a contract clause is harder than a 500-token narrative summary request. Teams that build length-only routing discover this when short high-stakes prompts get silently downgraded to the cheap model. Length is a valid feature but must be combined with task type signals.

Routing Signal Taxonomy

Tier 1 — Fast heuristics (<1ms, no ML):

  • Token count (input length) — use as a weak prior, not a gate
  • Presence of task-type keywords
  • User tier / API key metadata
  • Explicit X-Model-Tier header from upstream services

Tier 2 — Embedding classification (~5–10ms):

  • Cosine similarity against pre-labeled routing exemplars
  • Sentence-transformer embeddings (e.g., all-MiniLM-L6-v2, 384d)
  • Threshold on similarity score to route to capability tier

Tier 3 — ML classifier (50–100ms):

  • Trained BERT/DeBERTa classifier on preference labels
  • RouteLLM-style quality-aware routing
  • Full LLM meta-router (expensive, use only for very high-value requests)

Task Type → Capability Tier Mapping

Task Category Route To Reasoning
Structured extraction (JSON, entities) Cheap tier Deterministic pattern completion
Classification / sentiment Cheap tier Low ambiguity, bounded output
Translation Cheap tier Well-established capability at Haiku level
Summarization (routine) Cheap tier Compression task, no novel synthesis
Summarization (synthesis across docs) Mid tier Cross-document inference required
Code generation (<100 lines) Mid tier Standard patterns
Q&A over retrieved context (RAG) Cheap tier Context provided, model fills template
Multi-step reasoning Premium tier Chain-of-thought depth matters
Code review / security audit Premium tier Adversarial thinking, subtle bugs
Long-horizon planning Premium tier Coherence over many steps
Ambiguous or multi-intent instructions Premium tier Disambiguation requires stronger model
High-stakes generation (legal, medical) Premium tier Hallucination cost is high

Keyword Signal Lists (Production-Calibrated)

CHEAP_SIGNALS = [
    "extract", "list", "classify", "categorize", "translate",
    "summarize", "format as", "convert to JSON", "find all",
    "what is", "define", "spell check"
]

PREMIUM_SIGNALS = [
    "analyze", "evaluate", "compare", "reason through",
    "debug", "review", "audit", "plan", "design",
    "why does", "how should", "recommend", "critique",
    "identify risks", "synthesize"
]

def classify_tier(prompt: str) -> str:
    prompt_lower = prompt.lower()
    token_count = len(prompt.split())  # rough proxy

    premium_hits = sum(1 for kw in PREMIUM_SIGNALS if kw in prompt_lower)
    cheap_hits = sum(1 for kw in CHEAP_SIGNALS if kw in prompt_lower)

    # Token length gates
    if token_count > 4000:
        return "premium"   # long context needs strong coherence
    if token_count < 100 and cheap_hits > 0 and premium_hits == 0:
        return "cheap"

    if premium_hits > cheap_hits:
        return "premium"
    if cheap_hits >= 2 and premium_hits == 0:
        return "cheap"

    return "mid"  # fallback to middle tier on ambiguity

Token Length Tier Thresholds

These are calibrated starting points, not universal rules. Validate against your actual workload distribution before deploying.

Input Token Range Default Routing Rationale
0–200 tokens Cheap (with keyword check) Short Q&A, extractions
200–1,000 tokens Mid (task-type dependent) Standard completions
1,000–8,000 tokens Mid → Premium (keyword check) Long context needs attention
8,000+ tokens Premium Long-horizon coherence risk on small models

Caveat: RouteLLM research found frontier models were necessary for only 14% of MT-Bench queries — far less than intuition suggests. Most teams over-provision their routing cutoffs on first deployment and leave savings on the table.

4. Haiku / Sonnet / Opus Tiered Routing

Current Bedrock Pricing (June 2026)

Model Input $/1M Output $/1M Notes
Claude 3 Haiku ~$0.25 ~$1.25 Oldest, fastest, cheapest
Claude 3.5 Haiku ~$0.80 ~$4.00 Materially better than 3 Haiku
Claude 3.5 Sonnet v2 $6.00 $30.00 GA via Bedrock extended access
Nova Lite $0.06 $0.24 Amazon’s cheapest capable model
Nova Pro $0.80 $3.20 Amazon’s mid-tier

The cost ratio between Claude 3.5 Haiku and Claude 3.5 Sonnet v2 is ~7.5× on input and ~7.5× on output. Every request correctly kept on Haiku instead of Sonnet saves 87% of that request’s model cost.

Three-Tier Architecture

Tier 1 (Cheap):    Claude 3.5 Haiku / Nova Lite
Tier 2 (Mid):      Nova Pro / Llama 3.3 70B
Tier 3 (Premium):  Claude 3.5 Sonnet v2 / Claude Opus

Routing up from Tier 1:

  • Detected multi-step reasoning signals
  • Input > 4,000 tokens with synthesis keywords
  • User tier = paid/enterprise and task = code review
  • Prior Tier 1 response failed quality gate (cascade pattern)

Staying on Tier 1:

  • Structured extraction, classification, translation
  • RAG completions where context is fully provided
  • Summary of a single document
  • Free-tier users regardless of task (cost enforcement)

Routing to Tier 3:

  • Explicit require_premium: true flag in request metadata
  • Security audit, legal review, medical content
  • Input > 8,000 tokens
  • Agent loop step N > 5 (complex task orchestration)

LiteLLM Three-Tier Config

model_list:
  # Tier 1 — Cheap
  - model_name: tier-cheap
    litellm_params:
      model: bedrock/anthropic.claude-3-5-haiku-20241022-v1:0
      aws_region_name: us-east-1
      input_cost_per_token: 0.0000008
      output_cost_per_token: 0.000004

  # Tier 2 — Mid
  - model_name: tier-mid
    litellm_params:
      model: bedrock/amazon.nova-pro-v1:0
      aws_region_name: us-east-1
      input_cost_per_token: 0.0000008
      output_cost_per_token: 0.0000032

  # Tier 3 — Premium
  - model_name: tier-premium
    litellm_params:
      model: bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
      aws_region_name: us-east-1
      input_cost_per_token: 0.000003
      output_cost_per_token: 0.000015

router_settings:
  routing_strategy: simple-shuffle
  fallbacks:
    - tier-cheap: [tier-mid, tier-premium]
    - tier-mid: [tier-premium]

The caller selects the tier before invoking LiteLLM — routing logic lives in the application layer (using the classifier from Section 3) rather than inside LiteLLM’s router. LiteLLM handles fallbacks when a tier is unavailable.

5. Semantic Routing via Embeddings

Architecture

Semantic routing classifies incoming requests by computing cosine similarity between the prompt embedding and a library of labeled exemplar embeddings. Each exemplar is tagged with a routing target. The highest-similarity exemplar determines the route.

Request → Embed (5ms) → Cosine Similarity → Route Label → Model Tier
                             ↑
                     Labeled Exemplar Library
                     (cheap/mid/premium examples)

Cost of the Classifier vs Savings

Embedding cost (OpenAI text-embedding-3-small): ~$0.02/1M tokens. A 500-token prompt costs $0.00001 to embed — effectively free relative to inference cost.

Self-hosted embedding (MiniLM-L6 via SentenceTransformers on CPU): ~5ms latency, $0 marginal cost. Recommended for high-volume routing.

Savings from accurate classification: If the classifier keeps 65% of traffic on Haiku instead of Sonnet (realistic for mixed workloads), and average request costs $0.003 on Sonnet vs $0.0004 on Haiku, the per-request savings is $0.00171. At 100,000 requests/day, that is $171/day = ~$62,000/year from a $0 classifier.

Exemplar Library Design

exemplars = [
    # Cheap tier examples
    {"text": "Extract all dates from this document", "tier": "cheap"},
    {"text": "Classify this email as spam or not spam", "tier": "cheap"},
    {"text": "Translate this paragraph to Spanish", "tier": "cheap"},
    {"text": "Summarize this article in 3 bullet points", "tier": "cheap"},
    {"text": "Convert this CSV row to JSON", "tier": "cheap"},

    # Mid tier examples
    {"text": "Write a product description for this item", "tier": "mid"},
    {"text": "Generate 5 test cases for this function", "tier": "mid"},
    {"text": "Explain this error message and suggest a fix", "tier": "mid"},

    # Premium tier examples
    {"text": "Review this contract for unusual indemnification clauses", "tier": "premium"},
    {"text": "Analyze the security implications of this authentication flow", "tier": "premium"},
    {"text": "Debug why this distributed system exhibits this failure mode", "tier": "premium"},
    {"text": "Compare these three architectural approaches and recommend one", "tier": "premium"},
]

Implementation Sketch

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")

# Pre-compute exemplar embeddings (do once at startup)
exemplar_texts = [e["text"] for e in exemplars]
exemplar_tiers = [e["tier"] for e in exemplars]
exemplar_embeddings = model.encode(exemplar_texts, normalize_embeddings=True)

def route_request(prompt: str, similarity_threshold: float = 0.45) -> str:
    prompt_embedding = model.encode([prompt], normalize_embeddings=True)[0]
    similarities = np.dot(exemplar_embeddings, prompt_embedding)
    best_idx = np.argmax(similarities)
    best_score = similarities[best_idx]

    if best_score < similarity_threshold:
        # Low confidence — default to mid tier rather than cheap
        return "mid"

    return exemplar_tiers[best_idx]

Accuracy vs Cost Tradeoff

Classifier Type Overhead Accuracy (MT-Bench) Cost
Keyword rules <1ms ~65–70% Free
Embedding cosine (MiniLM) ~5ms ~75–80% ~Free (self-hosted)
Fine-tuned BERT ~15ms ~85–90% $0 inference (self-hosted)
LLM meta-router (GPT-4o-mini) ~400ms ~92% $0.15/1M tokens
RouteLLM (trained) ~50ms ~85% @ 95% quality Open source

The embedding + cosine approach occupies the sweet spot: meaningfully better than keywords, negligible cost, no external API dependency.

Silent quality regression risk: The dominant failure mode is a well-calibrated classifier silently routing a hard request to the cheap model because the prompt superficially resembles a simple extraction. Mitigation: run a lightweight quality gate on a sample of cheap-tier responses (LLM-as-judge or groundedness check on 1–5% of requests), alert if quality drops below threshold.

6. A/B Testing Model Routing

Why A/B Test Routing

Routing changes are non-obvious in their quality impact. A classifier that looks accurate on benchmarks may underperform on your specific workload distribution. A/B testing provides ground-truth data on both cost and quality before committing to a routing change.

Traffic Splitting via LiteLLM Weights

model_list:
  # Control: all traffic to Sonnet
  - model_name: ab-control
    litellm_params:
      model: bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
      aws_region_name: us-east-1
    weight: 50   # 50% of requests to this group

  # Treatment: Haiku for 80% of traffic within this group
  - model_name: ab-treatment-haiku
    litellm_params:
      model: bedrock/anthropic.claude-3-5-haiku-20241022-v1:0
      aws_region_name: us-east-1
    weight: 40   # 50% * 80% = 40% of total

  - model_name: ab-treatment-sonnet
    litellm_params:
      model: bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
      aws_region_name: us-east-1
    weight: 10   # 50% * 20% = 10% of total

The cleaner pattern is to split at the application layer using a user/session hash, then call different model endpoints, and log the model used alongside the response quality metric.

Routing A/B Test Methodology

Phase 1: Shadow mode (0% traffic impact)

  • Route all requests to the primary model as normal
  • Simultaneously send to the candidate model (async, without serving response)
  • Compare outputs offline
  • Duration: 2–3 days minimum, 10,000+ requests

Phase 2: Controlled split

  • Route 5–10% of traffic to treatment routing policy
  • Track: cost per request, latency P50/P95, quality metrics
  • Duration: until statistical significance on primary quality metric

Phase 3: Ramp

  • 25% → 50% → 100% on weekly cadence if metrics hold

Statistical Significance for Cost and Quality

LLM quality distributions are non-normal — groundedness scores cluster toward high values (right-skewed), completeness scores are bimodal. Use bootstrap resampling rather than t-tests.

Bootstrap approach:

import numpy as np

def bootstrap_difference(control_scores, treatment_scores,
                         n_bootstrap=10000, alpha=0.05):
    """
    Returns (mean_diff, ci_lower, ci_upper, is_significant).
    Positive diff = treatment better than control.
    """
    diffs = []
    for _ in range(n_bootstrap):
        c_sample = np.random.choice(control_scores, len(control_scores), replace=True)
        t_sample = np.random.choice(treatment_scores, len(treatment_scores), replace=True)
        diffs.append(np.mean(t_sample) - np.mean(c_sample))

    ci_lower = np.percentile(diffs, alpha / 2 * 100)
    ci_upper = np.percentile(diffs, (1 - alpha / 2) * 100)
    mean_diff = np.mean(diffs)
    is_significant = ci_lower > 0 or ci_upper < 0  # CI doesn't straddle zero

    return mean_diff, ci_lower, ci_upper, is_significant

Sample size guidance:

Expected quality delta Required samples per arm
5% difference ~2,000
2% difference ~12,000
1% difference ~50,000

For most routing changes, a 5% quality regression is the minimum meaningful threshold. At typical production volumes of 10,000+ requests/day, a 48–72 hour test is sufficient.

Cost Comparison Methodology

Track at the per-request grain, not aggregate:

cost_per_request = {
    "control": [],
    "treatment": []
}

# For each request, log:
# - group (control/treatment)
# - input_tokens
# - output_tokens
# - model_used
# - cost = input_tokens * input_rate + output_tokens * output_rate
# - quality_score (from evaluator)

control_cost = np.mean(cost_per_request["control"])
treatment_cost = np.mean(cost_per_request["treatment"])
cost_reduction_pct = (control_cost - treatment_cost) / control_cost * 100

print(f"Cost reduction: {cost_reduction_pct:.1f}%")

Performance-per-dollar index: Divide quality score by blended cost per request. Winning routing policy maximizes this ratio, not just minimizes cost.

Gotchas

  • Session consistency: Users should see a consistent model within a session (hash on user ID, not request ID) to avoid mid-conversation quality regression.
  • Workload shift: If test period coincides with a product launch or weekend traffic pattern, results may not generalize. Run across at least one full weekly cycle.
  • Don’t use the same model to evaluate quality: An LLM-as-judge using Claude Sonnet to evaluate Haiku responses will show bias toward Sonnet-style outputs. Use a third independent evaluator or human judges on a sample.

7. Fallback Chains

Three-Tier Fallback Architecture

Primary (best quality/cost for task)
    ↓ [429, 5xx, timeout, context exceeded]
Secondary (same family, higher capacity or lower cost)
    ↓ [429, 5xx, timeout]
Emergency (cross-provider, high reliability, any cost)

LiteLLM Proxy Config

model_list:
  # Primary pool — Haiku on us-east-1
  - model_name: primary
    litellm_params:
      model: bedrock/anthropic.claude-3-5-haiku-20241022-v1:0
      aws_region_name: us-east-1

  # Fallback 1 — Haiku on us-west-2 (rate limit relief)
  - model_name: fallback-regional
    litellm_params:
      model: bedrock/anthropic.claude-3-5-haiku-20241022-v1:0
      aws_region_name: us-west-2

  # Fallback 2 — Nova Pro (different family, different rate limits)
  - model_name: fallback-family
    litellm_params:
      model: bedrock/amazon.nova-pro-v1:0
      aws_region_name: us-east-1

  # Emergency — Sonnet (higher cost, near-zero rate limit risk)
  - model_name: emergency
    litellm_params:
      model: bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
      aws_region_name: us-east-1

router_settings:
  routing_strategy: simple-shuffle
  fallbacks:
    - primary: [fallback-regional, fallback-family, emergency]
  context_window_fallbacks:
    - primary: [emergency]    # Haiku 200K → Sonnet 200K (same window but more capable)
  content_policy_fallbacks:
    - primary: [emergency]    # Content flagged → try Sonnet before surfacing error

  # Cooldown after 429 before retrying
  cooldown_time: 60   # seconds
  num_retries: 2
  timeout: 30

Error types handled by automatic fallback:

  • 429 RateLimitError — triggers fallback after cooldown_time seconds
  • 5xx ServiceUnavailability — immediate fallback
  • ContextWindowExceededError — routes to context_window_fallbacks
  • ContentPolicyViolationError — routes to content_policy_fallbacks
  • Timeout — triggers after timeout seconds

Cost Implications at Each Fallback Tier

Tier Scenario Cost Premium vs Primary
Primary → Regional Fallback Same model, different region 0% (same rates)
Primary → Family Fallback Haiku → Nova Pro ~6× higher output cost
Primary → Emergency (Sonnet) Rate limit extreme event ~37× higher output cost

Diagnostic threshold: If fallback-family or emergency is handling more than 3–5% of steady-state traffic, something is wrong with primary provisioning. Either increase Bedrock service quotas, enable cross-region inference profiles, or provision throughput units.

Latency impact: Fallback activation adds 200–400ms overhead (retry + new connection). In streaming workloads this is largely invisible. In synchronous workloads with tight SLOs, model the fallback latency into your p95 budget.

8. LiteLLM model_group_alias and weights for Controlled Traffic Splits

model_group_alias

Maps logical model names clients use to underlying model groups in the router. Enables transparent routing without client-side changes.

litellm_settings:
  model_group_alias:
    "gpt-4": "claude-sonnet-group"       # Route "gpt-4" calls to Sonnet pool
    "gpt-4o": "claude-haiku-group"       # Route "gpt-4o" calls to Haiku pool
    "gpt-3.5-turbo": "nova-lite-group"   # Route cheapest alias to Nova Lite

Use case: Migrating an existing application that hard-codes gpt-4 model names to Bedrock without changing application code.

Traffic Weights Within a Model Group

model_list:
  # Canary test: 90% stays on current Haiku, 10% routes to new Haiku 3.6
  - model_name: haiku-production
    litellm_params:
      model: bedrock/anthropic.claude-3-5-haiku-20241022-v1:0
      aws_region_name: us-east-1
    weight: 9   # 90%

  - model_name: haiku-production
    litellm_params:
      model: bedrock/anthropic.claude-3-6-haiku-v1:0  # hypothetical new version
      aws_region_name: us-east-1
    weight: 1   # 10%

Multiple deployments sharing a model_name form a group. Traffic distributes proportionally to weight. This is the primary mechanism for:

  • Canary rollouts of new model versions
  • Multi-region load balancing within a cost tier
  • A/B testing at the proxy level without application changes

Routing Groups (LiteLLM Pro Feature)

Routing groups allow different routing strategies per model group within the same proxy instance:

# Apply latency-based routing to premium group,
# simple-shuffle to cheap group
router_settings:
  routing_groups:
    premium-group:
      routing_strategy: latency-based-routing
    cheap-group:
      routing_strategy: simple-shuffle

9. Cost-Aware Routing Under Budget Pressure

Provider Budget Routing

LiteLLM’s provider_budget_config routes requests away from providers that have exhausted their daily/monthly budget.

router_settings:
  provider_budget_config:
    anthropic:
      budget_limit: 500     # $500 USD
      time_period: 1mo      # Monthly reset

    amazon-bedrock-nova:
      budget_limit: 2000    # $2000 USD
      time_period: 1mo

  # When provider exceeds budget, route to next available
  fallbacks:
    - anthropic-haiku: [bedrock-nova-lite, openai-gpt4o-mini]

Behavior: Redis tracks cumulative spend per provider. When budget_limit is hit, LiteLLM treats all deployments in that provider as unhealthy and routes to fallbacks. Resets at time_period boundary. Raises 429 if all providers are over budget.

Monitoring: Remaining budget available at /provider/budgets endpoint and via Prometheus metric litellm_provider_remaining_budget_metric. Set alerts at 80% depletion to give time to adjust allocations.

Team / Key Budget Routing

# Per-key budget enforcement
litellm_settings:
  default_team_settings:
    max_budget: 100     # $100/month per team
    budget_duration: 1mo
    tpm_limit: 1000000  # 1M TPM cap

Budget-aware routing can route high-spending teams to cheaper models as a soft enforcement mechanism rather than a hard block:

# Application layer: check remaining budget before routing
remaining = litellm_client.get_team_budget(team_id)
budget_pct_used = 1 - (remaining / max_budget)

if budget_pct_used > 0.80:
    model = "tier-cheap"    # Route to cheap tier at 80% budget consumed
elif budget_pct_used > 0.95:
    model = "tier-emergency-cheap"  # Most restrictive tier at 95%
else:
    model = classify_tier(prompt)   # Normal routing

Dynamic Model Tier Mapping by Budget State

Budget Used Routing Policy
0–70% Normal classifier routing
70–85% Cap at mid tier — no premium model unless require_premium: true
85–95% All requests to cheap tier regardless of classifier
>95% Serve from cache where possible; block new premium requests

This pattern — common in multi-tenant SaaS — prevents the scenario where a small number of complex requests from a single team exhaust the monthly budget in the first week.

10. Routing for Latency vs Cost: SLO-Constrained Routing

Defining Routing SLOs

A routing policy should have an explicit cost-latency tradeoff statement. Example SLO matrix:

Request Class Max P95 Latency Max Cost/Request Preferred Model
Interactive chat 2,000ms $0.003 Haiku (fast + cheap)
Background summarization 30,000ms $0.001 Haiku batch
Code review 10,000ms $0.020 Sonnet
Async report generation 120,000ms $0.005 Haiku or Sonnet batch

Latency-Constrained Cost Minimization

When latency is the binding constraint, route to the fastest-AND-cheapest model that can meet the SLO:

MODELS = [
    {"name": "haiku", "p95_ms": 800,  "cost_per_1k_out": 0.004},
    {"name": "nova-lite", "p95_ms": 600, "cost_per_1k_out": 0.00024},
    {"name": "nova-pro", "p95_ms": 1200, "cost_per_1k_out": 0.0032},
    {"name": "sonnet", "p95_ms": 2800, "cost_per_1k_out": 0.030},
]

def cheapest_within_slo(slo_p95_ms: int, task_tier: str) -> str:
    eligible = [m for m in MODELS if m["p95_ms"] <= slo_p95_ms]
    if not eligible:
        raise ValueError(f"No model meets SLO of {slo_p95_ms}ms")

    # Sort by cost ascending, return cheapest
    eligible.sort(key=lambda m: m["cost_per_1k_out"])
    return eligible[0]["name"]

Key insight from the latency data: Nova Lite is both cheaper and faster than Haiku for many workload types. The default choice of Haiku as the cheap tier is a habit from Anthropic-native deployments; on Bedrock, Nova Lite is often the correct cheap anchor.

P95 Latency Monitoring and Routing Adjustment

router_settings:
  routing_strategy: latency-based-routing
  routing_strategy_args:
    ttl: 30                      # 30-second latency window
    lowest_latency_buffer: 0.3   # Route within 1.3× of fastest observed P50

The lowest_latency_buffer is the key tuning parameter. Setting it too low (0.0) concentrates all traffic on the single fastest deployment, eventually saturating it. Setting it too high (1.0) distributes traffic evenly regardless of latency, defeating the purpose. 0.3–0.5 is a practical starting range.

Cross-Region Latency Routing

For applications with strict P95 SLOs and global traffic, routing to the lowest-latency Bedrock region reduces total latency:

model_list:
  - model_name: haiku-global
    litellm_params:
      model: bedrock/us.anthropic.claude-3-5-haiku-20241022-v1:0  # cross-region US
      aws_region_name: us-east-1
  - model_name: haiku-global
    litellm_params:
      model: bedrock/eu.anthropic.claude-3-5-haiku-20241022-v1:0  # cross-region EU
      aws_region_name: eu-central-1
  - model_name: haiku-global
    litellm_params:
      model: bedrock/ap.anthropic.claude-3-5-haiku-20241022-v1:0  # cross-region APAC
      aws_region_name: ap-northeast-1

router_settings:
  routing_strategy: latency-based-routing

Cross-region inference profiles carry a small additional cost vs single-region inference. For latency-sensitive interactive workloads, the quality-of-service benefit typically outweighs the cost premium.

11. Real-World Routing Architectures

Cursor: Tiered Model Selection by Context Complexity

Cursor (AI code editor, $29B valuation as of 2026) migrated from a fixed 500 fast requests/month model to a credit pool system in June 2025. The architecture uses “Auto” mode — automatic model selection based on query complexity — as the default, with users able to override to a specific model at the cost of additional credits.

Inferred architecture from public signals:

  • Auto mode routes short inline completions to a fast/cheap model (inferred: Haiku-class or GPT-4o-mini-class)
  • Complex multi-file edits and agent loops route to a frontier model (Sonnet or GPT-4o-class)
  • Credit consumption rate varies by model selected — the same task costs more if the user overrides Auto to a premium model
  • Users who stay in Auto mode stay closer to their plan’s expected spend; manual premium overrides drive bill overages

Routing signal: Editor context size (number of files in context window), command type (inline autocomplete vs agent edit vs chat), and user’s remaining credit balance.

Vercel: AI Gateway with Cost/Latency/Quality Sorting

Vercel’s AI Gateway (released 2025) provides a centralized interface to 100+ models with built-in routing modes. Routing criteria exposed to developers:

  • Cost sort: Selects cheapest provider for the logical model name
  • TTFT sort: Time-to-first-token — routes to lowest observed TTFT endpoint
  • Throughput sort: Tokens-per-second — routes to highest-throughput endpoint
  • Per-model analytics: Cost and latency visible per model in dashboard

Architecture pattern: Vercel abstracts routing via logical model names (e.g., anthropic/claude-3-5-haiku-latest) and routes at the gateway layer, allowing Next.js teams to add fallback and cost routing without application changes.

Replit: Effort-Based Billing with Model Routing

Replit’s agent (billing as of July 2025) uses an effort-based credit model: complex tasks (which require frontier model invocations) cost $0.60–$1.00+ per task while simple tasks cost $0.06–0.15. The agent internally routes sub-tasks to appropriate models based on estimated effort.

Routing signal inferred from pricing: Replit’s credit pricing scale implies the agent routes code generation and debugging to a cheaper model, but final synthesis and user-facing responses to a frontier model. The total task cost reflects the mix of models used within the agent loop.

NadirClaw / Open-Source Proxy Pattern

NadirClaw (open-source, GitHub) implements a drop-in OpenAI-compatible proxy that routes simple prompts to cheap/local models and complex ones to premium models. Reported 40–70% cost reduction depending on workload mix. The classifier uses embedding similarity against a labeled exemplar library — architecturally identical to the approach in Section 5.

RouteLLM Research Findings

RouteLLM (academic, open-source) is a trained router that predicts whether a request requires GPT-4-class capability. Key findings from MT-Bench evaluation:

  • Frontier model required for only 14% of queries — far less than team intuition suggests
  • A BERT-based classifier achieved 45% cost savings on MMLU with less than 5% quality degradation
  • The matrix factorization router achieved 85% cost savings while maintaining 95% of GPT-4 quality on MT-Bench
  • Best results come from routers trained on preference data (human or model-generated win/loss pairs), not just task type labels

Phase 1: Quick wins (days 1–7)

  1. Enable AWS IPR on your highest-volume Bedrock model pair (Nova Lite/Pro or Haiku/Sonnet). Default responseQualityDifference: 0.1. Expected savings: 35–56% on model costs, $1/1k requests routing fee.

  2. Add LiteLLM proxy with simple-shuffle and multi-region deployments for the same model. Eliminates single-region rate limit risk at zero cost increase.

  3. Tag IAM roles with Team and Project and activate as cost allocation tags in AWS Billing. Wait 24h, then verify tags appear in CUR 2.0.

Phase 2: Complexity routing (weeks 2–4)

  1. Implement keyword classifier (Section 3) routing to explicit cheap/mid/premium endpoints. Deploy in shadow mode first (log the routing decision, don’t act on it) for 48–72h.

  2. Validate shadow routing against actual quality: sample 200 requests that would have been downgraded and compare cheap-tier vs premium-tier responses with LLM-as-judge or human review.

  3. Activate routing if quality holds. A/B test at 10% traffic for 72h before full rollout.

Phase 3: Operational hardening (weeks 4–8)

  1. Add provider budget routing with 80% depletion alerts. Route budget-pressured teams to cheap tier before hitting hard budget ceiling.

  2. Add semantic routing (embedding classifier) to replace keyword classifier for higher accuracy on ambiguous requests.

  3. Implement fallback chain (Section 7) with at least regional fallback + emergency cross-family fallback. Monitor fallback trigger rate — above 5% indicates provisioning problem.

  4. Establish quarterly routing review: model pricing changes, new model releases (e.g., Nova 2.x with better cost/capability ratio), and routing accuracy drift from workload pattern shifts.

Key Numbers for FinOps Conversations

Metric Value
IPR routing fee $1.00 / 1,000 requests
IPR avg cost reduction, Anthropic family 56%
IPR avg cost reduction, Nova family 35%
Haiku vs Sonnet v2 cost ratio (output) ~7.5×
Nova Lite vs Nova Pro cost ratio (output) ~13×
Embedding classifier overhead ~5ms
Embedding classifier cost (OpenAI small) ~$0.00001/request
RouteLLM: % requests needing frontier model 14% (MT-Bench)
Practical cost reduction range (tiered routing) 40–77%
Fallback latency overhead 200–400ms
Bootstrap test sample size (5% delta detection) ~2,000/arm

Sources