← Agent Frameworks 🕐 19 min read

Agent Frameworks

Bedrock Inference Routing Optimization — CRIPs, Flex, LiteLLM Router (2026)

Executive Summary · March 2026

AWS Bedrock now exposes four distinct routing layers that compound when stacked correctly.

AWS Bedrock now exposes four distinct routing layers that compound when stacked correctly. Cross-Region Inference Profiles (CRIPs) absorb burst traffic with a latency overhead of ~10–30 ms and no cost surcharge for geographic profiles (global profiles carry ~10% savings over geographic). The Flex service tier provides a 50% token-rate discount for deferrable workloads — model eval, annotation, and multi-step agentic pipelines — at the cost of minutes-scale latency and best-effort queue priority. LiteLLM Router, deployed in front of Bedrock, adds per-deployment quota awareness, latency-based steering, and cost-based dispatch that Bedrock’s native routing does not expose. Application Inference Profiles (AIPs) bridge all three by allowing teams to tag and track traffic cost across organizational boundaries. Used together, the four layers can reduce blended inference cost by 40–65% versus naive single-region on-demand, while maintaining a 99.5% availability target for mission-critical paths.

1. Cross-Region Inference Profiles (CRIPs) — Architecture and GA Status

1.1 What CRIPs Are

A Cross-Region Inference Profile is a system-defined Bedrock resource that wraps a foundation model and a set of destination regions. When you invoke a CRIP instead of a bare model ID, Bedrock’s control plane dynamically selects the optimal destination region within the profile’s scope — optimizing for available capacity and, in the global tier, cost. The invoking account never directly chooses the destination; the selection is opaque to the caller.

There are two scopes:

Scope	Geographic Boundary	Cost vs. Single-Region	Typical Use Case
Geographic (US / EU / APAC / etc.)	Within a defined geography	Same price	Regulatory / data-residency constrained
Global	All commercial AWS regions worldwide	~10% cheaper than geographic	Maximum throughput, cost-optimized

As of June 2026, Global cross-region inference is supported on Anthropic Claude Sonnet 4 from five source regions: us-west-2, us-east-1, us-east-2, eu-west-1, and ap-northeast-1. Destination regions span all commercial AWS regions. Geographic profiles exist for Claude 3.x and Claude 4.x family models, Amazon Nova Pro/Premier, Llama 3.x, and Mistral variants. Embedding models do not support CRIPs.

1.2 Profile Identifiers

System-defined CRIPs use a structured ID pattern that encodes the geo scope as a prefix:

# Geographic — US scope
us.anthropic.claude-3-5-sonnet-20241022-v2:0

# Geographic — EU scope  
eu.anthropic.claude-3-5-sonnet-20241022-v2:0

# Geographic — APAC scope
ap.anthropic.claude-3-haiku-20240307-v1:0

# Global scope (Claude Sonnet 4 only as of June 2026)
us.anthropic.claude-sonnet-4-5-v1:0   # source: us-east-1, us-east-2, us-west-2

The same model ID may route to different destination regions depending on the source region. For example, invoking us.anthropic.claude-3-haiku-20240307-v1:0 from us-east-2 can route to us-east-1, us-east-2, or us-west-2; from us-west-2 it routes only to us-east-1 and us-west-2. Always call GetInferenceProfile to enumerate the exact destination set for your source region before building capacity assumptions into runbooks.

import boto3

bedrock = boto3.client("bedrock", region_name="us-east-1")
profile = bedrock.get_inference_profile(
    inferenceProfileIdentifier="us.anthropic.claude-3-5-sonnet-20241022-v2:0"
)
# profile["models"] is a list of ARNs — each encodes a destination region
for m in profile["models"]:
    print(m["modelArn"])
# arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0
# arn:aws:bedrock:us-east-2::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0
# arn:aws:bedrock:us-west-2::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0

1.3 Throughput and Quota Behavior

Using a CRIP raises your effective throughput ceiling. The quota math is straightforward:

Standard in-region on-demand quota: 1× baseline (e.g., 10K TPM for Claude Sonnet 4.6 in us-east-1)
Cross-region via geographic CRIP: up to 2× the in-region quota for the invoking (source) account in the source region
Cross-region via global CRIP: higher than geographic, as the destination pool expands to all commercial regions

The increased quota applies only to invocations via the CRIP model ID. Direct model ID invocations (anthropic.claude-3-5-sonnet-20241022-v2:0) still consume the standard in-region quota and do not benefit from the expanded pool.

SCP gotcha: If your organization uses Service Control Policies to restrict bedrock:InvokeModel* to specific regions, a CRIP request will fail if any destination region in the profile is blocked — even if other destinations remain allowed. The SCP check happens before Bedrock’s routing logic. The fix is to allow aws:RequestedRegion: "unspecified" for Bedrock actions in destination regions, or to enumerate all CRIP destinations explicitly.

{
  "Sid": "AllowBedrockCRIP",
  "Effect": "Allow",
  "Action": ["bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream"],
  "Resource": "*",
  "Condition": {
    "StringEquals": {
      "aws:RequestedRegion": [
        "us-east-1", "us-east-2", "us-west-2",
        "eu-west-1", "eu-west-2", "eu-central-1",
        "ap-northeast-1", "ap-southeast-1", "ap-southeast-2"
      ]
    }
  }
}

For global profiles, allow "unspecified" in addition to enumerated regions — global routing may select regions the SCP author did not anticipate.

1.4 Latency Trade-off

CRIP adds routing overhead when a request is deflected to a non-primary destination region. Observed overhead in production is 10–30 ms additional TTFT on deflected requests. This is measured server-side (Bedrock receive → first token) and excludes client network transit to the source region.

For streaming APIs, this means TTFT on a same-region hit might be 180 ms while a deflected cross-region hit runs 210–220 ms on the same model. For non-streaming (batch-style Converse calls), the overhead is swamped by total generation latency and is immaterial at typical output lengths above 100 tokens.

Decision rule: if your SLA requires TTFT < 200 ms, monitor the TimeToFirstToken CloudWatch metric per inference profile and build an alarm at 250 ms to detect when CRIP deflections are accumulating.

1.5 Failover Behavior

CRIP implements sequential failover across destination regions. When the primary region returns a throttling error, Bedrock retries the next destination in the profile. If all destinations are exhausted, the caller receives a standard ThrottlingException. This retry loop is transparent — the caller’s SDK sees a single API call succeed or fail. There is no retry budget consumed on the client side, and no cost for the failed attempts.

This behavior enables active-passive high availability without any client-side retry logic. The pattern fails when an organization’s SCP blocks one or more destination regions silently — the throttle appears to come from the source region, masking the underlying SCP misconfiguration.

1.6 CUR Billing Attribution — The Home Region Attribution Quirk

The core quirk: When a CRIP request is served from a destination region (e.g., us-west-2) but invoked from the source region (e.g., us-east-1), the cost appears in the source region’s CUR line items. The lineItemUsageType encodes the routing:

# In-region standard request — source=us-east-1, served=us-east-1
USE1-Claude4.6Sonnet-input-tokens

# Cross-region standard request — source=us-east-1, served=us-west-2
USE1-Claude4.6Sonnet-input-tokens-cross-region-global

# Priority tier cross-region  
USE1-Claude4.6Sonnet-input-tokens-cross-region-global-priority

# Flex tier cross-region
USE1-Claude4.6Sonnet-input-tokens-cross-region-global-flex

The region prefix (USE1) is always the source (billing) region. The -cross-region-global suffix is the only CUR signal that a request routed elsewhere. There is no column in standard CUR that shows the actual serving region. To find that, you must join CUR to model invocation logs (CloudWatch Logs or S3) on model and usage-type grain — logs carry the inferenceRegion field from CloudTrail.

Practical consequence for FinOps: A dashboard that filters CUR by region code will correctly attribute spend to the source region, but will not differentiate in-region from cross-region serving costs. Apply unit price logic per usage type pattern — *-cross-region-global has a different unit price than *-input-tokens (global is ~10% cheaper; geographic cross-region is same price as in-region). If you only count *-input-tokens, you will undercount because cross-region tokens have a different line item key.

CUR aggregation granularity: Neither legacy CUR nor CUR 2.0 carries a per-requestId identifier. Cost is aggregated by usage type and operation over 1-hour or 1-day windows. For per-prompt attribution, join model invocation logs to CUR at the (model, usage_type) grain. Use Application Inference Profile tags (see Section 4) to cost-allocate across teams without needing the log join.

IAM principal attribution (CUR 2.0 only): April 2026 GA: Bedrock records the invoking IAM user or role ARN in CUR 2.0 as iamPrincipal/* columns. This requires activating the tags in Billing console and CUR 2.0 (AWS Data Exports) — not available in legacy CUR format. After activation, allow up to 24 hours for backfill.

2. Flex Inference Tier — Routing Behavior and Cost Trade-offs

2.1 Tier Overview

As of 2026, Bedrock exposes four service tiers on a single service_tier parameter:

Tier	Pricing vs. Standard	Priority	SLA	Timeout	Best For
`reserved`	Fixed monthly TPM commitment	Highest	99.5% uptime	None	Mission-critical customer-facing
`priority`	Premium over standard	High	Best-effort fast	Standard	Real-time interactive, no reservation
`default` (standard)	1× baseline	Normal	Best-effort	Standard	General workloads
`flex`	50% discount	Lowest	Best-effort slow	60 minutes	Async, deferrable workloads

Setting the tier is a single parameter addition to any Converse or InvokeModel call:

response = bedrock_runtime.converse(
    modelId="us.anthropic.claude-3-5-sonnet-20241022-v2:0",
    messages=[{"role": "user", "content": [{"text": prompt}]}],
    additionalModelRequestFields={
        "service_tier": "flex"
    }
)

2.2 Flex Capacity Selection

Flex does not expose its region-selection logic. Internally, it draws from spare capacity across regions using the same pool as the CRIP infrastructure — which is why Flex and CRIP compose naturally. A request submitted as flex to a geographic CRIP profile may be served from any region in that profile’s destination set, prioritizing whichever region currently has excess capacity at the deepest discount. The caller has no visibility into which region served the request beyond the CloudTrail inferenceRegion field.

Flex tier requests sit at the bottom of the priority queue. If a region is saturated with Reserved and Priority traffic, Flex requests queue. Queue depth and wait time are not currently surfaced as CloudWatch metrics — the only observable signal is elapsed wall-clock time from request to first token (for streaming) or full response (for non-streaming).

60-minute timeout: Flex requests that have not begun processing within 60 minutes are rejected with a ServiceUnavailableException. For agentic workflows with many sequential steps, this means the per-step Flex timeout is 60 minutes — not the total workflow duration. Individual steps that exceed 60 minutes in the queue will fail; the application must handle retries or fallback to standard tier.

2.3 Flex-Eligible Models (as of June 2026)

Flex tier support varies by model. Confirmed Flex-capable:

Amazon Nova 2 Lite, Nova Pro, Nova Premier
OpenAI on Bedrock: gpt-oss-20b, gpt-oss-120b (via Bedrock Mantle)
DeepSeek V3.1
Qwen3 family (Qwen3-235B-A22B, Qwen3-32B)

Check the individual model card page (Models at a glance in Bedrock console) for up-to-date Flex availability — AWS is progressively enabling Flex for additional models through 2026.

2.4 Flex vs. On-Demand Routing Decision Table

Workload dimension          Flex        Standard     Priority    Reserved
────────────────────────────────────────────────────────────────────────
Response time requirement   Minutes     Seconds      Sub-second  Guaranteed
Token cost                  0.5×        1×           1.2–1.5×    Fixed/TPM
Queue priority              Lowest      Normal       High        Dedicated
Timeout risk                High (60m)  Low          Low         None
Best for                    Evals,      General      Customer    SLA-bound
                            annotation, use          chat,       enterprise
                            batch job                real-time   apps
Availability SLA            None        Best-effort  Best-effort 99.5%

Anti-pattern: Do not use Flex for user-facing requests where the user is waiting for a response. Do not use Flex for the orchestration step of an agentic loop — use it for the tool-call steps that do heavy lifting (summarization, extraction, classification) where the orchestrator can await the result asynchronously.

2.5 CUR Encoding for Flex

Flex token usage appears in CUR with a -flex suffix on the usage type:

USE1-Nova2.0Lite-input-tokens-flex
USE1-Nova2.0Lite-output-tokens-flex
USE1-Claude4.6Sonnet-input-tokens-cross-region-global-flex

The -flex suffix is present regardless of whether the request was also routed cross-region. When both cross-region and flex apply, the suffix chain is -cross-region-global-flex or -cross-region-{geo}-flex.

3. LiteLLM Router Strategies for Bedrock Cost Optimization

LiteLLM provides an open-source proxy layer that wraps multiple Bedrock deployments (and other providers) under a single OpenAI-compatible API. Its routing strategies are complementary to Bedrock’s native routing — they operate at the LiteLLM level before a request reaches Bedrock, whereas CRIPs and Flex are Bedrock-native.

3.1 Deployment Configuration — Bedrock with Multi-Region

# litellm_config.yaml — multi-region Bedrock with CRIP ARNs
model_list:
  # Primary: us-east-1 via CRIP (geographic US)
  - model_name: claude-sonnet
    litellm_params:
      model: bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
      modelId: us.anthropic.claude-3-5-sonnet-20241022-v2:0
      aws_region_name: us-east-1
      tpm: 50000
      rpm: 100

  # Secondary: us-west-2 via CRIP (geographic US)
  - model_name: claude-sonnet
    litellm_params:
      model: bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
      modelId: us.anthropic.claude-3-5-sonnet-20241022-v2:0
      aws_region_name: us-west-2
      tpm: 50000
      rpm: 100

  # Tertiary: EU region via EU CRIP (fallback for EU data-residency)
  - model_name: claude-sonnet-eu
    litellm_params:
      model: bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
      modelId: eu.anthropic.claude-3-5-sonnet-20241022-v2:0
      aws_region_name: eu-west-1
      tpm: 30000
      rpm: 60

router_settings:
  routing_strategy: usage-based-routing-v2
  redis_host: ${REDIS_HOST}
  redis_password: ${REDIS_PASSWORD}
  redis_port: 6379
  num_retries: 3
  timeout: 30
  retry_after: 10

3.2 Strategy Deep-Dive

usage-based-routing-v2

Tracks live TPM and RPM consumption per deployment using Redis. When a deployment’s consumption exceeds its configured limit, it is filtered from the candidate set. The remaining candidates are ranked by lowest current TPM usage.

Redis required: Async Redis incr/mget calls track TPM buckets per deployment per minute.
Production caveat: Redis round-trip adds ~1–3 ms overhead per routing decision. AWS documentation flags this as a concern for very high RPM workloads (>1,000 RPS). For typical enterprise usage (<100 RPS), the overhead is negligible.
Bedrock-CRIP interaction: Each LiteLLM “deployment” can point to a different CRIP source region. This means LiteLLM’s quota tracking stacks on top of Bedrock’s CRIP quota doubling — at 50K TPM per LiteLLM deployment across two regions, the effective ceiling is 100K TPM before Bedrock-level throttling applies.

from litellm import Router

router = Router(
    model_list=model_list,
    routing_strategy="usage-based-routing-v2",
    redis_host=os.environ["REDIS_HOST"],
    redis_password=os.environ["REDIS_PASSWORD"],
    redis_port=6379,
    enable_pre_call_checks=True,   # filter before request, not after failure
    num_retries=3,
    fallbacks=[{"claude-sonnet": ["claude-sonnet-eu"]}]
)

latency-based-routing

Maintains a rolling latency cache per deployment, updated on each completed request. Routes to the deployment with the lowest cached latency.

router_settings:
  routing_strategy: latency-based-routing
  routing_strategy_args:
    ttl: 30         # seconds; latency samples older than this are discarded
    lowest_latency_buffer: 0.3   # consider deployments within 30% of minimum

The lowest_latency_buffer is critical for multi-region setups: without it, all traffic concentrates on the single fastest deployment, which then becomes the slowest due to load. A buffer of 0.3 distributes load across deployments within 30% of the minimum observed latency, preserving throughput while favoring lower-latency regions.

Interaction with CRIP deflections: If us-east-1 is deflecting many requests cross-region (adding 20–30 ms TTFT), the latency cache will record the elevated latency and steer future requests to us-west-2 if it is answering faster. This creates a feedback loop that naturally corrects for regional capacity imbalances without manual intervention.

least-busy

Counts ongoing in-flight requests per deployment and routes to the deployment with the fewest open connections. This is the lowest-overhead strategy — no Redis, no cache — and works well for heterogeneous workload sizes (some requests generate 50 tokens, others 2,000).

router = Router(
    model_list=model_list,
    routing_strategy="least-busy"
)

Best suited when token output length varies significantly across requests and you cannot predict load from TPM metrics alone.

cost-based-routing

Selects the cheapest eligible deployment using litellm’s model cost map. Supports custom per-deployment pricing overrides.

model_list = [
    {
        "model_name": "summarizer",
        "litellm_params": {
            "model": "bedrock/amazon.nova-lite-v1:0",
            "aws_region_name": "us-east-1",
            # Nova Lite pricing as of 2026
            "input_cost_per_token": 0.00000006,
            "output_cost_per_token": 0.00000024,
        }
    },
    {
        "model_name": "summarizer",
        "litellm_params": {
            "model": "bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0",
            "aws_region_name": "us-east-1",
            "input_cost_per_token": 0.000003,
            "output_cost_per_token": 0.000015,
        }
    }
]
router = Router(model_list=model_list, routing_strategy="cost-based-routing")

Limitation: cost-based-routing is infrastructure-level — it always picks the cheapest healthy deployment without considering quality requirements. It is appropriate for homogeneous tasks (all calls have the same complexity profile) where the cheapest capable model is always the correct choice. For heterogeneous workloads, combine it with Bedrock’s Intelligent Prompt Routing (see Section 6) — IPR handles model-family selection based on prompt complexity, and LiteLLM handles regional load balancing within that model family.

simple-shuffle (recommended default)

Weight-based random selection. Lowest overhead, no Redis, no latency cache. Use weight to encode your preferred regional distribution (e.g., 70/30 primary/secondary split).

model_list:
  - model_name: claude-sonnet
    litellm_params:
      model: bedrock/...
      aws_region_name: us-east-1
      weight: 7    # 70% of traffic
  - model_name: claude-sonnet
    litellm_params:
      model: bedrock/...
      aws_region_name: us-west-2
      weight: 3    # 30% of traffic

3.3 Fallback Configuration

LiteLLM’s fallback system is independent of CRIP’s automatic failover. The two complement each other: CRIP handles transparent failover within a geography (invisible to LiteLLM), while LiteLLM fallbacks handle explicit model or region demotions you want to control.

router_settings:
  fallbacks:
    - claude-sonnet-us: ["claude-sonnet-eu", "claude-haiku-us"]
  context_window_fallbacks:
    - claude-sonnet: ["claude-sonnet-200k"]
  allowed_fails: 2      # failures before marking deployment unhealthy
  cooldown_time: 60     # seconds to wait before retrying failed deployment
  retry_after: 5        # seconds between retries on same deployment

A mature enterprise config stacks three levels: (1) LiteLLM routes within a CRIP-backed deployment, (2) CRIP deflects within the geography, (3) LiteLLM fallback to a different geography or model family. The result is fault isolation across all three layers with no single point of routing failure.

4. Application Inference Profiles (AIPs) — Multi-Region Routing with Tags

4.1 What AIPs Are

Application Inference Profiles are customer-created inference profile resources that wrap any Bedrock foundation model or system-defined CRIP. Unlike system CRIPs (which are read-only and AWS-managed), AIPs are mutable — you create, tag, and delete them. Their primary purpose is cost attribution: any invocation through an AIP ARN carries the AIP’s resource tags into CUR.

AIPs do not add routing intelligence. Routing is inherited from the underlying model or CRIP the AIP wraps. If you wrap a CRIP, routing is CRIP routing. If you wrap a bare model ID, there is no cross-region routing.

4.2 Creating an AIP via CLI

# Create AIP wrapping the US geographic CRIP
aws bedrock create-inference-profile \
  --inference-profile-name "prod-claude-sonnet-team-a" \
  --model-source '{"copyFrom": "arn:aws:bedrock:us-east-1::foundation-model/us.anthropic.claude-3-5-sonnet-20241022-v2:0"}' \
  --tags '{"CostCenter": "engineering", "Team": "platform", "Environment": "prod"}' \
  --region us-east-1

# Returns:
# {
#   "inferenceProfileArn": "arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/xxxxxxxx",
#   "status": "ACTIVE"
# }

4.3 Invoking via AIP ARN

Replace the model ID in any InvokeModel or Converse call with the AIP ARN:

response = bedrock_runtime.converse(
    modelId="arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/xxxxxxxx",
    messages=[{"role": "user", "content": [{"text": prompt}]}]
)

Bedrock resolves the AIP → underlying CRIP → destination region transparently. The caller’s code sees one ARN and never directly manages routing logic.

4.4 AIP ARN Anatomy

arn:aws:bedrock:{source-region}:{account-id}:application-inference-profile/{profile-id}

AIP creation is supported in 15 regions (as of June 2026): us-east-1, us-east-2, us-west-2, eu-west-1, eu-west-2, eu-west-3, eu-central-1, ap-northeast-1, ap-northeast-2, ap-southeast-1, ap-southeast-2, ap-south-1, ca-central-1, sa-east-1, us-gov-east-1. All other Bedrock-supported regions can invoke models but cannot create AIPs.

4.5 Multi-Team Cost Isolation Pattern

The canonical enterprise pattern is one AIP per team × environment × model tier:

prod-team-a-claude-sonnet    → wraps US CRIP Claude Sonnet 4.6   → tags: Team=A, Env=prod
prod-team-b-claude-sonnet    → wraps US CRIP Claude Sonnet 4.6   → tags: Team=B, Env=prod
prod-shared-nova-pro         → wraps US CRIP Nova Pro             → tags: Team=shared, Env=prod
dev-team-a-claude-haiku      → wraps US CRIP Claude Haiku 4.5    → tags: Team=A, Env=dev

CUR then carries the Team and Env tags as resourceTags/Team and resourceTags/Env columns, enabling per-team spend dashboards without any log-join work. Tags must be activated in the Billing console; allow 24 hours after activation before they appear in CUR.

4.6 AIP + LiteLLM Integration

LiteLLM supports AIP ARNs directly as the modelId:

model_list:
  - model_name: claude-sonnet-team-a
    litellm_params:
      model: bedrock/arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/prod-team-a-claude-sonnet
      aws_region_name: us-east-1
      tpm: 40000
      rpm: 80

This combines AIP-level cost tagging with LiteLLM’s quota-aware routing — the best of both layers.

5. Bedrock Gateway Patterns (aws-samples)

5.1 bedrock-access-gateway

The aws-samples/bedrock-access-gateway project provides an OpenAI-compatible proxy for Bedrock. It translates OpenAI API requests to Bedrock API calls, enabling any OpenAI SDK client to use Bedrock models without code changes.

Architecture options:

Deployment Mode	Backend	Latency	Cold Start	VPC Required
Serverless	API Gateway + Lambda (streaming)	Low	Yes (~200ms)	No
Containerized	ALB + Fargate	Lowest	None	Yes

The serverless mode uses API Gateway response streaming for SSE (Server-Sent Events) support without requiring a persistent container. The Fargate mode is preferred for high-throughput production workloads where Lambda cold starts degrade TTFT SLA.

Supported endpoints:

POST /chat/completions — chat inference (maps to Converse API)
POST /embeddings — embedding models
GET /models — model enumeration
Multimodal and tool-call passthrough

Token counting: The gateway exposes prompt cache token counts in the usage response (cached_tokens field), reflecting Bedrock’s prompt caching state. There is no pre-call token estimation endpoint exposed through the gateway, but Bedrock’s native CountTokens API is available directly for pre-call budgeting:

# Native Bedrock CountTokens — not proxied through gateway
count_response = bedrock_runtime.count_tokens(
    modelId="anthropic.claude-3-5-sonnet-20241022-v2:0",
    messages=[{"role": "user", "content": [{"text": prompt}]}]
)
estimated_input_tokens = count_response["inputTokenCount"]

Cross-region support: The gateway accepts an aws_region_name parameter at deployment time. To serve multiple regions, deploy multiple gateway stacks (one per region) behind a Route 53 latency-based routing policy, or use a single gateway in the primary region and pass CRIP model IDs — the gateway forwards them directly to Bedrock, which handles the cross-region routing.

5.2 sample-bedrock-proxy-gateway

The aws-samples/sample-bedrock-proxy-gateway adds enterprise access control features the basic access gateway omits:

Multi-account load distribution: Routes requests across multiple AWS accounts, each with independent Bedrock quotas. Effective combined throughput = N accounts × per-account quota.
Rate limiting: Token-based and request-based quotas per client identity.
Credential caching: Sub-10ms response times with Valkey (Redis-compatible) cache for AWS credential resolution.

# Multi-account configuration
accounts = [
    {"account_id": "111111111111", "role_arn": "arn:aws:iam::111111111111:role/BedrockAccessRole"},
    {"account_id": "222222222222", "role_arn": "arn:aws:iam::222222222222:role/BedrockAccessRole"},
    {"account_id": "333333333333", "role_arn": "arn:aws:iam::333333333333:role/BedrockAccessRole"},
]
# Gateway load-balances across accounts; each has independent TPM/RPM quota
# Effective throughput: 3× per-account quota

This pattern is the standard approach for enterprises that have hit the per-account TPM ceiling and cannot (or prefer not to) request quota increases for a single account.

5.3 Bedrock AgentCore Gateway

Distinct from the sample repos, Amazon Bedrock AgentCore Gateway (GA in 2026) is a managed service layer for agent tool invocation, not inference routing. It exposes existing API Gateway REST APIs and Lambda functions as MCP-compatible tool endpoints for Bedrock agents. It is relevant to routing in the sense that it is the routing layer for tool calls within an agent loop, but it does not affect foundation model inference routing. Do not conflate the two.

6. Intelligent Prompt Routing — Cost Impact Quantification

Bedrock’s Intelligent Prompt Routing (IPR) is a distinct feature from CRIPs and Flex. It routes within a model family (e.g., Claude Sonnet 4.6 vs. Claude Haiku 4.5) based on estimated prompt complexity, automatically selecting the cheaper model for simpler requests.

6.1 Quantified Cost Savings

AWS internal testing results (published, 2026):

Test Configuration	% Requests Routed to Cheaper Model	Cost Savings	Quality Impact
Claude family (Sonnet 4.6 / Haiku 4.5), hundreds of prompts	87%	63.6%	Baseline maintained vs. Sonnet
Amazon Nova family (Pro / Lite), mixed workload	~70%	35%	Baseline maintained
Real application: 70% simple / 30% complex, 100K req/month	70%	65%	Minimal (< 2% quality delta on evals)

Worked example:

Without IPR: 100,000 req/month × Claude Sonnet 4.6 → ~$168/month (500 input / 400 output tokens average)
With IPR (70% Haiku, 30% Sonnet): ~$59/month → 65% savings
IPR router cost: $1.00 per 1,000 requests → $100/month for 100K requests
Net savings: $168 − $59 − $100 = $9 net savings per $100 in baseline spend (but at 500K+ req/month the router cost becomes negligible)

Important nuance: IPR cost at $1.00/1,000 requests is fixed per request, not per token. At low request volumes with long prompts, the router overhead may exceed the savings. Break-even is approximately 8,000–10,000 requests/month for a typical enterprise workload; above that, IPR is always net-positive.

6.2 Stacking IPR with CRIP and Flex

These three mechanisms compose at the parameter level:

# Stack: IPR model selection + CRIP geographic routing + Flex tier
response = bedrock_runtime.converse(
    modelId="us.anthropic.claude-3-5-sonnet-20241022-v2:0",  # CRIP
    messages=[{"role": "user", "content": [{"text": prompt}]}],
    additionalModelRequestFields={
        "service_tier": "flex"   # Flex tier
    }
    # IPR is configured at the router resource level in Bedrock console,
    # not per-request. When enabled, Bedrock may downgrade to Haiku 4.5
    # transparently before applying the CRIP and Flex routing.
)

Stacking all three on deferrable tasks (batch annotation, eval runs, content summarization) produces the maximum cost reduction:

IPR: 35–65% savings from model downgrade
Flex: 50% discount on remaining token costs
CRIP: no cost increase (geographic) or 10% additional savings (global)
Combined net: 65–80% reduction vs. standard on-demand Sonnet on all tasks

7. Observability — CloudWatch, CUR, and OpenTelemetry

7.1 New CloudWatch Metrics (GA: March 2026)

Two new metrics require no opt-in and carry no additional cost:

TimeToFirstToken

Namespace: AWS/Bedrock
Unit: Milliseconds
APIs: ConverseStream, InvokeModelWithResponseStream only
Dimensions: ModelId, ServiceTier, ResolvedServiceTier
Aggregation: 1-minute
What it measures: Server-side time from Bedrock receiving the streaming request to first token generated. Excludes client-to-Bedrock network transit.

EstimatedTPMQuotaUsage

Namespace: AWS/Bedrock
Unit: Count (tokens)
APIs: All inference APIs
Additional dimension: ContextWindow (for requests >200K tokens)
Formula (on-demand): InputTokenCount + CacheWriteInputTokens + (OutputTokenCount × burndown_rate)

Critical quota gotcha: EstimatedTPMQuotaUsage reflects consumption after request completion — not the upfront max_tokens reservation used for throttling decisions. Bedrock reserves quota based on max_tokens at request time. If you set max_tokens=4096 but the model generates 200 tokens, the throttle check consumed 4096 tokens of quota capacity, but EstimatedTPMQuotaUsage only records 200 tokens after the fact. Right-size max_tokens to your expected output distribution, or you will observe throttling despite favorable EstimatedTPMQuotaUsage readings.

7.2 Recommended CloudWatch Alarm Configuration

import boto3

cw = boto3.client("cloudwatch", region_name="us-east-1")

# Alarm: TTFT degradation (CRIP deflection accumulating)
cw.put_metric_alarm(
    AlarmName="bedrock-crip-ttft-degraded",
    Namespace="AWS/Bedrock",
    MetricName="TimeToFirstToken",
    Dimensions=[
        {"Name": "ModelId", "Value": "us.anthropic.claude-3-5-sonnet-20241022-v2:0"}
    ],
    Statistic="p95",
    Period=300,          # 5-minute window
    EvaluationPeriods=3,
    Threshold=350.0,     # 350ms p95 = CRIP deflections accumulating
    ComparisonOperator="GreaterThanThreshold",
    AlarmActions=["arn:aws:sns:us-east-1:123456789012:bedrock-alerts"]
)

# Alarm: Quota approaching limit (pre-throttle warning)
cw.put_metric_alarm(
    AlarmName="bedrock-quota-80pct",
    Namespace="AWS/Bedrock",
    MetricName="EstimatedTPMQuotaUsage",
    Dimensions=[
        {"Name": "ModelId", "Value": "us.anthropic.claude-3-5-sonnet-20241022-v2:0"},
        {"Name": "ServiceTier", "Value": "default"}
    ],
    Statistic="Sum",
    Period=60,
    EvaluationPeriods=2,
    Threshold=40000,     # 80% of 50K TPM quota
    ComparisonOperator="GreaterThanThreshold",
    AlarmActions=["arn:aws:sns:us-east-1:123456789012:bedrock-alerts"]
)

7.3 CloudTrail for Cross-Region Attribution

Every Bedrock inference request generates a CloudTrail event with an additionalEventData.inferenceRegion field. For CRIPs, this field shows the destination region that actually served the request (e.g., us-west-2) even when the API call was made from us-east-1. This is the only authoritative source for serving-region attribution — CUR only shows the billing (source) region.

# Athena query to find CRIP deflection rate
# (requires CloudTrail log delivery to S3)
SELECT
    eventtime,
    requestparameters.modelid,
    additionaleventdata.inferenceregion as serving_region,
    awsregion as billing_region,
    count(*) as request_count
FROM cloudtrail_logs
WHERE eventsource = 'bedrock.amazonaws.com'
  AND eventname IN ('InvokeModel', 'Converse', 'ConverseStream')
  AND eventtime > current_date - interval '7' day
GROUP BY 1, 2, 3, 4
ORDER BY 1 DESC

7.4 OpenTelemetry Integration

Bedrock does not natively emit OTel traces. The canonical approach for distributed tracing across a multi-region Bedrock deployment uses one of three paths:

Path 1: LiteLLM + OpenTelemetry callback

from litellm.integrations.opentelemetry import OpenTelemetry

litellm.callbacks = [OpenTelemetry()]
# Emits spans per LiteLLM routing decision + per Bedrock call
# Span attributes include: model, region, input_tokens, output_tokens, latency

Path 2: AWS Distro for OpenTelemetry (ADOT) Deploy the ADOT collector as a sidecar to your inference service. Instrument with the AWS X-Ray SDK or the standard OTEL SDK — Bedrock SDK calls are automatically traced via X-Ray. ADOT forwards spans to both X-Ray and any OTLP-compatible backend (Grafana Tempo, Jaeger, Honeycomb, etc.).

Path 3: OpenInference / Arize Phoenix instrumentation

from openinference.instrumentation.bedrock import BedrockInstrumentor
BedrockInstrumentor().instrument()
# Wraps boto3 bedrock-runtime client; emits OTel spans with LLM semantic conventions
# span.attributes["gen_ai.system"] = "aws.bedrock"
# span.attributes["gen_ai.request.model"] = "us.anthropic.claude-..."
# span.attributes["gen_ai.response.model"] = actual served model
# span.attributes["aws.bedrock.inference_region"] = actual serving region

OpenInference’s Bedrock instrumentor (maintained by Arize) extracts inferenceRegion from the response metadata and attaches it as a span attribute, giving you per-request serving-region visibility in your trace backend without Athena queries.

7.5 CUR + Log Join for Per-Request Cost Attribution

For FinOps workflows requiring per-request cost (e.g., per-customer billing):

-- Athena: join CUR to invocation logs on (model, hour)
-- CUR is hourly aggregated; logs are per-request
SELECT
  logs.requestid,
  logs.model_id,
  logs.input_tokens,
  logs.output_tokens,
  (logs.input_tokens * cur.input_unit_price) +
  (logs.output_tokens * cur.output_unit_price) as estimated_cost_usd
FROM bedrock_invocation_logs logs
JOIN bedrock_cur cur
  ON logs.model_id = cur.model_id
  AND date_trunc('hour', logs.timestamp) = cur.usage_hour
  AND cur.usage_type LIKE '%input-tokens%'
WHERE logs.timestamp > current_timestamp - interval '24' hour

The join is imprecise when a single hour has both in-region and cross-region traffic (different unit prices). Add a filter on cur.usage_type pattern to separate -cross-region from in-region line items.

8. Enterprise Deployment Patterns

8.1 Active-Active vs. Active-Passive Decision Framework

Pattern	Description	When to Use	Trade-offs
CRIP passive failover	Single CRIP in one region; deflects automatically	Most teams; no quota management burden	Extra 10–30ms on deflections; no control over destination
LiteLLM + multi-region CRIPs	LiteLLM routes across CRIPs in multiple source regions	>50K RPM sustained; compliance isolation required	Redis required; operational overhead
Multi-account	Multiple AWS accounts, each with independent quotas	TPM ceiling exhausted in one account; financial isolation	IAM complexity; credential management
Active-active (global CRIP)	Single global CRIP; Bedrock selects destination globally	Maximum throughput, no geographic constraint	Data residency risk; SCP complexity
Reserved + on-demand overflow	Reserved capacity for baseline; on-demand for bursts	Predictable baseline with variable spikes	Reserved billing continues regardless of usage

8.2 Quota Management Across Regions

Default per-account TPM quotas for common models (June 2026, subject to change via AWS Service Quotas):

Model	us-east-1	us-west-2	eu-west-1
Claude Sonnet 4.6 (on-demand)	40K TPM	40K TPM	20K TPM
Claude Haiku 4.5	50K TPM	50K TPM	30K TPM
Amazon Nova Pro	40K TPM	40K TPM	20K TPM
Amazon Nova Lite	100K TPM	100K TPM	50K TPM

Verify current quotas via aws service-quotas list-service-quotas --service-code bedrock or the Bedrock console quota dashboard.

With a US geographic CRIP, effective throughput ceiling per account is up to 2× the source region quota (e.g., 80K TPM for Claude Sonnet 4.6 from us-east-1 via CRIP). This is the standard scale-out path before requesting quota increases.

April 2026: Amazon Bedrock expanded Service Quotas integration, allowing programmatic quota increase requests via the Service Quotas API without opening a support case. Use request_service_quota_increase for models that support it:

aws service-quotas request-service-quota-increase \
  --service-code bedrock \
  --quota-code L-XXXXXXXX \
  --desired-value 100000

8.3 Reserved Capacity in Multi-Region Deployments

Reserved tier requirements:

Minimum: 100K input TPM + 10K output TPM per reservation
Duration: 1 or 3 months; billing continues until deletion (contact AWS account team)
Overflow: Automatic overflow to Standard tier when reserved capacity is exhausted
Provisioned Throughput vs. Reserved tier: Provisioned Throughput is per-model per-account; Reserved tier is a service-level commitment with overflow protection

For multi-region, purchase separate Reserved tier capacity in each region where you need guaranteed throughput. Reserved capacity does not transfer across regions and does not interact with CRIP routing — a Reserved tier request in us-east-1 uses the us-east-1 Reserved pool regardless of CRIP configuration.

Anti-pattern: Do not size Reserved capacity at peak throughput. Reserved billing is fixed. Size at sustained baseline (P50 load), use Standard tier for daily variance (P50–P90), and CRIP for burst absorption (P90+). This three-tier stack minimizes the fixed cost component of the reserved commitment.

8.4 Enterprise Routing Architecture Reference

┌─────────────────────────────────────────────────────────────────┐
│                    Client Application                           │
└─────────────────────────┬───────────────────────────────────────┘
                          │ OpenAI-compatible API
                          ▼
┌─────────────────────────────────────────────────────────────────┐
│                LiteLLM Router (us-east-1)                       │
│  Strategy: usage-based-routing-v2                               │
│  ┌─────────────────┐  ┌─────────────────┐  ┌────────────────┐  │
│  │ AIP: Team-A     │  │ AIP: Team-B     │  │ AIP: Shared    │  │
│  │ Wraps US CRIP   │  │ Wraps US CRIP   │  │ Wraps EU CRIP  │  │
│  │ tpm: 40K        │  │ tpm: 40K        │  │ tpm: 20K       │  │
│  └────────┬────────┘  └────────┬────────┘  └───────┬────────┘  │
└───────────┼────────────────────┼───────────────────┼────────────┘
            │                    │                   │
            ▼                    ▼                   ▼
┌───────────────────────────────────────────────────────────────┐
│                 Amazon Bedrock (Regional API)                  │
│                                                               │
│  CRIP (US)                            CRIP (EU)               │
│  Primary: us-east-1 ──┐               Primary: eu-west-1      │
│  Failover: us-east-2  ├→ Bedrock       Failover: eu-central-1 │
│  Failover: us-west-2 ─┘  routing                              │
│                                                               │
│  Service Tier:                                                │
│  Reserved ← P50 baseline                                      │
│  Standard ← P50–P90 variance                                  │
│  Flex ← Async deferrable workloads                            │
└───────────────────────────────────────────────────────────────┘
            │
            ▼
┌───────────────────────────────────────────────────────────────┐
│               Cost Attribution (CUR 2.0)                      │
│                                                               │
│  AIP tags → resourceTags/Team, resourceTags/Env               │
│  IAM principal → iamPrincipal/arn                             │
│  Usage type → tier + routing signal                           │
│  CloudTrail → actual serving region (inferenceRegion)         │
└───────────────────────────────────────────────────────────────┘

8.5 Routing Strategy Decision Checklist

Use this decision sequence when designing a Bedrock inference routing architecture:

1. Do you have data-residency or geographic compliance constraints?
   YES → Use Geographic CRIP (US, EU, or APAC scope)
   NO  → Use Global CRIP for ~10% additional savings

2. Do you need per-team cost attribution without log-join work?
   YES → Create Application Inference Profiles with cost allocation tags
   NO  → Use system CRIPs directly

3. Is your peak throughput >2× the per-account CRIP ceiling?
   YES → Add LiteLLM Router with multi-region CRIP deployments OR
          use multi-account setup via sample-bedrock-proxy-gateway
   NO  → Single CRIP per region is sufficient

4. Do you have deferrable workloads (evals, annotation, batch)?
   YES → Set service_tier="flex" — 50% cost reduction
   NO  → Use service_tier="default"

5. Do you have mixed-complexity prompts (simple Q&A + complex reasoning)?
   YES → Enable Intelligent Prompt Routing for 35–65% model-level savings
   NO  → Route directly to the appropriate model tier

6. Do you need guaranteed throughput for customer-facing apps?
   YES → Purchase Reserved tier capacity for P50 baseline load
   NO  → On-demand (Standard) with CRIP overflow is sufficient

7. Are you running multi-step agentic pipelines?
   YES → Use Flex for inner tool-call steps, Standard for orchestration
          Set per-step timeout awareness for the 60-min Flex limit
   NO  → Standard tier throughout

9. Cost Impact Summary — Stacked Optimizations

Strategy	Cost Reduction	Latency Impact	Complexity
Geographic CRIP only	0% (throughput gain only)	+10–30ms on deflections	Low
Global CRIP	~10% vs. geographic	+10–30ms on deflections	Low
Flex tier (deferrable only)	50% on deferrable tokens	+minutes	Low
Intelligent Prompt Routing	35–65% on mixed workloads	-10–30ms (faster model)	Medium
LiteLLM latency routing	15% via reduced throttle-retries	Better P95	Medium
AIP tagging	$0 cost change (attribution only)	None	Low
Multi-account proxy	Unlocks higher quotas (cost shift)	None	High
Reserved capacity	20–40% vs. on-demand at scale	None	High
All stacked (deferrable)	65–80% vs. baseline	Varies	High
All stacked (real-time)	40–55% vs. baseline	Same or better P95	Medium