← Agent Frameworks 🕐 17 min read
Agent Frameworks

Bedrock Quota Management and TPM/RPM Scaling at Enterprise Scale (2026)

Overview · March 2026

Amazon Bedrock service quotas are among the most consequential operational constraints for enterprise AI platforms.

Amazon Bedrock service quotas are among the most consequential operational constraints for enterprise AI platforms. Unlike cloud compute quotas (which scale with spend), Bedrock TPM and RPM limits are set per-model, per-region, and require active management even for high-spend accounts. This note covers the quota architecture, monitoring tools, multi-region strategies, tier selection, and governance patterns that platform engineering and FinOps teams need to run Bedrock at scale in 2026.

The core problem: default quotas for new accounts are deliberately conservative. A fresh account calling anthropic.claude-sonnet-4-6 in us-east-1 may see as little as 10,000–50,000 TPM until a quota increase is approved. Production workloads processing hundreds of documents per minute or serving real-time user-facing APIs will saturate these defaults within hours of launch.

1. Quota Structure: What Is Actually Enforced

1.1 Quota Dimensions

Bedrock enforces quotas on the bedrock-runtime endpoint across four dimensions:

Quota Name Scope What It Counts
On-demand InvokeModel tokens per minute Per model, per region Input + output tokens, combined
Cross-Region InvokeModel tokens per minute Per model, per region Same, but for CRIP-routed traffic
Model invocation max tokens per day Per model, per region Cumulative daily cap
InvokeModel requests per minute (RPM) Per model, per region API call count; model-specific

Important: the bedrock-runtime endpoint combines input and output into a single TPM pool. The bedrock-mantle endpoint (used by the Responses and Chat Completions APIs) applies separate input TPM and output TPM quotas — a different constraint structure that matters if you’re routing through the OpenAI-compatible endpoint.

RPM quotas are model-specific. Some high-end models (Claude Opus 4.7, Claude Opus 4.8) have no RPM quota and are governed solely by token throughput. For Claude Sonnet and Haiku, RPM quotas exist and are enforced independently of TPM — you can exhaust RPM with many small requests before exhausting TPM.

1.2 Default Quota Values by Model (2026 Baselines)

AWS does not publish a static table of default quotas — they vary by account age, payment history, region, and model lifecycle status. The following are representative values observed in established accounts as of mid-2026. New accounts and GovCloud accounts start lower.

Model Model ID Default On-Demand TPM (us-east-1) Default RPM Notes
Claude Sonnet 4.6 anthropic.claude-sonnet-4-6 ~100,000–500,000 ~500–1,000 Geo/Global CRIP available; in-region not available in us-east-1
Claude Haiku 4.5 anthropic.claude-haiku-4-5 ~500,000–2,000,000 ~1,000–2,000 Highest default throughput in Claude family
Claude Sonnet 4.5 anthropic.claude-sonnet-4-5 ~200,000–1,000,000 ~500–1,000 GovCloud raised to 5,000,000 TPM Feb 2026
Nova Pro amazon.nova-pro-v1:0 ~800,000 ~200 Hard limit on RPM; no adjustable flag
Nova Lite amazon.nova-lite-v1:0 ~4,000,000 ~2,000 High default TPM
Nova Micro amazon.nova-micro-v1:0 ~4,000,000 ~2,000 Lowest latency, highest default quota

Always verify current limits for your specific account via the Service Quotas console. These numbers shift as AWS responds to demand and adjusts defaults for established accounts.

1.3 Soft vs. Hard Limits

The Service Quotas console lists an Adjustable column for each quota:

  • Adjustable = Yes (soft limit): You can submit a self-service increase request through Service Quotas. Most on-demand TPM and cross-region TPM quotas are adjustable.
  • Adjustable = No (hard limit): Cannot be raised via the console. These are typically service-level caps that require AWS architectural review or are not negotiable. Nova Pro’s 200 RPM limit is currently a hard limit for custom model deployments.

One important caveat: the daily token cap (Model invocation max tokens per day) is derived from TPM × 1,440 by default. When you increase your TPM, AWS will offer to raise the daily cap proportionally during the support interaction — request it explicitly, as it is not always raised automatically.

New accounts frequently receive 0 TPM / 0 RPM for models in certain regions. This is a provisioning issue, not the actual quota. Opening a support case referencing “quasi-zero quota” typically resolves within 24 hours.

1.4 Viewing Current Limits

Service Quotas Console:

AWS Console → Service Quotas → Amazon Bedrock → search model name

AWS CLI — list all Bedrock quotas:

aws service-quotas list-aws-default-service-quotas \
  --service-code bedrock \
  --region us-east-1 \
  --query "Quotas[?contains(QuotaName,'claude-sonnet-4-6')].[QuotaName,Value,Adjustable]" \
  --output table

AWS CLI — view applied quota (your account’s current value):

aws service-quotas list-service-quotas \
  --service-code bedrock \
  --region us-east-1 \
  --query "Quotas[?contains(QuotaName,'claude-sonnet-4-6')].[QuotaName,Value,Adjustable]" \
  --output table

Get a specific quota value by code:

# Quota code format: L-XXXXXXXX (find in console or list-aws-default-service-quotas)
aws service-quotas get-service-quota \
  --service-code bedrock \
  --quota-code L-XXXXXXXX \
  --region us-east-1

CloudWatch — view TPM utilization over last hour:

aws cloudwatch get-metric-statistics \
  --namespace AWS/Bedrock \
  --metric-name EstimatedTPMQuotaUsage \
  --dimensions Name=ModelId,Value=anthropic.claude-sonnet-4-6 \
  --start-time $(date -u -v-1H '+%Y-%m-%dT%H:%M:%SZ') \
  --end-time $(date -u '+%Y-%m-%dT%H:%M:%SZ') \
  --period 60 \
  --statistics Sum \
  --region us-east-1

2. Quota Request Process

2.1 Self-Service via Service Quotas Console

For any quota marked Adjustable = Yes:

  1. Navigate to Service Quotas → Amazon Bedrock in the target region
  2. Search for the specific model quota (e.g., “On-demand InvokeModel tokens per minute for anthropic.claude-sonnet-4-6”)
  3. Select the quota and click Request quota increase
  4. Enter the desired value and a justification

Critical note from AWS documentation (2026): Quota increase requests for TPM are prioritized for accounts that are actively consuming their existing quota allocation. If your account shows low usage relative to the current limit, AWS may deny the request. Submit requests when you are running at 60–80% of current capacity, not speculatively.

2.2 Bundled Increase Request

When you request an increase for the Cross-Region InvokeModel tokens per minute quota, AWS support will contact you and offer to also increase:

  • On-demand InvokeModel tokens per minute
  • Model invocation max tokens per day

Request all three simultaneously. This is the most efficient path to raising your effective ceiling across all traffic types.

AWS CLI — submit quota increase request:

aws service-quotas request-service-quota-increase \
  --service-code bedrock \
  --quota-code L-XXXXXXXX \
  --desired-value 2000000 \
  --region us-east-1

Track request status:

aws service-quotas list-requested-service-quotas-by-service \
  --service-code bedrock \
  --region us-east-1 \
  --query "RequestedQuotas[*].[QuotaName,DesiredValue,Status,Created]" \
  --output table

2.3 Justification That Accelerates Approval

Include in the justification field:

  1. Current usage data: “Our account averaged X TPM over the last 7 days (see CloudWatch metric EstimatedTPMQuotaUsage, model anthropic.claude-sonnet-4-6, region us-east-1)”
  2. Growth trajectory: “We are onboarding N new users / migrating Y workflows, projecting Z TPM within 30 days”
  3. Business impact of throttling: Quantify the cost of dropped requests — see Section 9 for the calculation framework
  4. Architecture description: State that you are using cross-region inference profiles, LiteLLM retry logic, and exponential backoff — this signals that your infrastructure handles throttling gracefully and will not create instability at higher quota levels

2.4 Approval Timelines (2026)

  • Self-service adjustable quotas (most Claude and Nova on-demand TPM): Automated approval is increasingly common for moderate increases (2–5x current limit). Resolution in minutes to hours.
  • Large increases (>10x) or new models: Routes to manual AWS review. 1–5 business days typical.
  • Claude Sonnet 4.5 / Haiku 4.5 (high-demand models): Some customers report 3–7 day review cycles during peak demand periods.
  • Models in Legacy or Deprecated lifecycle: Quota increases are denied. Verify model lifecycle status before submitting (Model lifecycle page in Bedrock docs).

Do not request quota increases for models nearing end-of-life. Migrate to the successor model and request quota on that model instead.

3. Cross-Region Quota Strategy

3.1 Why Cross-Region Quotas Multiply Effective Capacity

Each region has an independent TPM quota. A US Geo inference profile (CRIP) sources from one of your US regions but can route traffic to us-east-1, us-east-2, and us-west-2. The Cross-Region InvokeModel tokens per minute quota is set in your source region but backs compute capacity across the destination pool.

AWS documentation states that on-demand cross-region usage can yield up to 2x the in-region allocated quota due to pooled capacity, and global inference provides the highest available throughput due to routing across 20+ regions worldwide.

3.2 Three-Tier Cross-Region Architecture

Profile Type Inference ID Reaches Use When
In-Region anthropic.claude-sonnet-4-6 Single region Strict data residency + latency requirements
Geo (US) us.anthropic.claude-sonnet-4-6 us-east-1, us-east-2, us-west-2 US data residency required; higher throughput
Global global.anthropic.claude-sonnet-4-6 All commercial regions worldwide Maximum throughput; no residency constraints

For Claude Sonnet 4.6 specifically, in-region is not available in us-east-1 (as of mid-2026) — Geo and Global are the access paths. This makes the cross-region quota the operative limit for most US deployments.

3.3 Using Global Inference for Maximum Throughput

Global inference profiles (introduced as GA in 2025, expanded to more models in 2026) offer approximately 10% cost savings versus standard pricing in addition to higher throughput. For workloads without geographic compliance constraints, Global is the correct default choice.

import boto3

client = boto3.client('bedrock-runtime', region_name='us-east-1')

# Use Global inference ID instead of bare model ID
response = client.converse(
    modelId='global.anthropic.claude-sonnet-4-6',
    messages=[{'role': 'user', 'content': [{'text': 'Process this document'}]}]
)

# CloudTrail will log additionalEventData.inferenceRegion showing where request was processed

3.4 Multi-Source-Region Strategy with LiteLLM

When Global inference is not sufficient (regulated workloads, or you need explicit control over quota pools), deploy a LiteLLM proxy that distributes requests across multiple source regions. Each region has its own TPM/RPM quota, and LiteLLM handles load balancing and failover.

LiteLLM config for multi-region Bedrock load balancing:

# litellm_config.yaml
model_list:
  # US East pool
  - model_name: claude-sonnet-46-pool
    litellm_params:
      model: bedrock/us.anthropic.claude-sonnet-4-6
      aws_region_name: us-east-1
      tpm: 500000
      rpm: 1000
  
  # US West pool
  - model_name: claude-sonnet-46-pool
    litellm_params:
      model: bedrock/us.anthropic.claude-sonnet-4-6
      aws_region_name: us-west-2
      tpm: 500000
      rpm: 1000

  # EU pool (for EU-resident traffic)
  - model_name: claude-sonnet-46-eu
    litellm_params:
      model: bedrock/eu.anthropic.claude-sonnet-4-6
      aws_region_name: eu-west-1
      tpm: 300000
      rpm: 600

litellm_settings:
  num_retries: 5
  retry_after: 5
  request_timeout: 120
  fallback_models:
    - claude-haiku-45-pool  # fallback to Haiku if Sonnet pool exhausted

router_settings:
  routing_strategy: least-busy
  enable_pre_call_checks: true

Effective combined TPM in this config: 500,000 (us-east-1) + 500,000 (us-west-2) = 1,000,000 TPM before EU. With Global inference profiles, Bedrock handles this routing internally, but explicit multi-region routing gives you transparent quota accounting per region.

3.5 Request Routing in CloudTrail

All cross-region requests are logged in CloudTrail in the source region (where you called the API), not the destination region where the request was processed. The additionalEventData.inferenceRegion field identifies the actual processing region. Set up a CloudTrail Lake query to audit destination region distribution:

aws cloudtrail-data start-query \
  --query-statement "SELECT eventTime, additionalEventData.inferenceRegion AS dest_region, COUNT(*) AS requests FROM <trail-arn> WHERE eventSource='bedrock.amazonaws.com' AND eventName='InvokeModel' GROUP BY dest_region ORDER BY requests DESC"

4. Quota Monitoring: CloudWatch Metrics and Alarms

4.1 The EstimatedTPMQuotaUsage Metric — What It Actually Measures

Released in March 2026, EstimatedTPMQuotaUsage is the most important Bedrock metric for quota management. It is not a raw token counter — it applies the same burndown logic that Bedrock’s quota enforcement engine uses.

On-demand formula:

EstimatedTPMQuotaUsage = InputTokenCount + CacheWriteInputTokens + (OutputTokenCount × burndown_rate)

Burndown rates by model family (2026):

  • Claude Sonnet 4.x, Haiku 4.x: 5x on output tokens
  • Claude Opus 4.x: 5x on output tokens
  • Amazon Nova Pro, Lite, Micro: 1x (no output multiplier for Nova family)

Example for Claude Sonnet 4.6: A request with 1,000 input tokens, 200 cache write tokens, and 100 output tokens consumes:

1,000 + 200 + (100 × 5) = 1,700 quota tokens

Even though the actual token count is 1,300, the quota usage is 1,700. A pipeline that generates long outputs hits its TPM ceiling much faster than raw token counts suggest.

Why this matters for capacity planning: If your workloads generate high output-to-input ratios (document generation, code synthesis, summarization into long formats), apply the 5x multiplier to all output tokens when sizing quota requests. A 500,000 TPM quota with Claude Sonnet 4.6 effectively supports about 83,000 output tokens per minute if requests are pure output — not 500,000.

Provisioned Throughput formula (different):

EstimatedTPMQuotaUsage = InputTokenCount + (CacheWriteInputTokens × 1.25) + (CacheReadInputTokens × 0.1) + OutputTokenCount

Provisioned Throughput does not apply the 5x output burndown — one reason it is more cost-efficient for high-output workloads.

4.2 The max_tokens Reservation Problem

Before EstimatedTPMQuotaUsage was available (pre-March 2026), the de-facto monitoring approach was tracking raw input + output tokens. This understated quota consumption because Bedrock’s enforcement logic reserves quota equal to max_tokens at request submission, then reconciles at completion.

If your requests set max_tokens: 4096 but typically generate 200 output tokens, Bedrock reserves 4,096 × 5 = 20,480 quota tokens at request start and releases the excess (3,896 × 5 = 19,480 tokens) upon completion. Under high concurrency, your “in-flight reservation” can temporarily exhaust quota even if your actual throughput is well below the limit.

Mitigation: Set max_tokens to realistic values based on observed P95 output length. The aws-samples/sample-quota-dashboard-for-amazon-bedrock dashboard tracks both initial reservation (InputTokens + CacheWriteTokens + MaxTokens) and actual consumption side-by-side to surface this gap.

4.3 InvocationThrottles — The Lagging Indicator

InvocationThrottles counts requests that received a HTTP 429 response. It is a lagging indicator — by the time this metric rises, your users are already experiencing errors. Use it to:

  1. Measure throttling rate during incidents
  2. Set alarms that page if throttling rate exceeds 1% of total invocations
  3. Correlate with EstimatedTPMQuotaUsage trends to understand which quota pool is saturated

4.4 CloudWatch Alarms

Alarm: TPM approaching quota (pre-throttle warning, fires at 80%)

{
  "AlarmName": "Bedrock-TPM-Quota-Warning-Sonnet46",
  "AlarmDescription": "Claude Sonnet 4.6 TPM usage is at 80% of quota in us-east-1. Request quota increase before throttling begins.",
  "Namespace": "AWS/Bedrock",
  "MetricName": "EstimatedTPMQuotaUsage",
  "Dimensions": [
    {
      "Name": "ModelId",
      "Value": "us.anthropic.claude-sonnet-4-6"
    },
    {
      "Name": "Region",
      "Value": "us-east-1"
    }
  ],
  "Period": 300,
  "Statistic": "Sum",
  "ComparisonOperator": "GreaterThanThreshold",
  "Threshold": 400000,
  "EvaluationPeriods": 3,
  "DatapointsToAlarm": 2,
  "TreatMissingData": "notBreaching",
  "ActionsEnabled": true,
  "AlarmActions": ["arn:aws:sns:us-east-1:123456789012:bedrock-quota-alerts"],
  "OKActions": ["arn:aws:sns:us-east-1:123456789012:bedrock-quota-alerts"]
}

(Set Threshold to 80% of your current TPM quota. 400,000 assumes a 500,000 TPM quota.)

Alarm: Active throttling — critical (fire on first throttled request)

{
  "AlarmName": "Bedrock-InvocationThrottles-Critical-Sonnet46",
  "AlarmDescription": "Claude Sonnet 4.6 is actively throttling requests. Check EstimatedTPMQuotaUsage and request quota increase.",
  "Namespace": "AWS/Bedrock",
  "MetricName": "InvocationThrottles",
  "Dimensions": [
    {
      "Name": "ModelId",
      "Value": "us.anthropic.claude-sonnet-4-6"
    },
    {
      "Name": "Region",
      "Value": "us-east-1"
    }
  ],
  "Period": 60,
  "Statistic": "Sum",
  "ComparisonOperator": "GreaterThanThreshold",
  "Threshold": 0,
  "EvaluationPeriods": 1,
  "DatapointsToAlarm": 1,
  "TreatMissingData": "notBreaching",
  "ActionsEnabled": true,
  "AlarmActions": ["arn:aws:sns:us-east-1:123456789012:bedrock-quota-critical"]
}

Deploy alarm via CLI:

aws cloudwatch put-metric-alarm \
  --cli-input-json file://bedrock-tpm-warning-alarm.json \
  --region us-east-1

Dashboard widget — dual quota view (reservation vs. consumption):

{
  "type": "metric",
  "properties": {
    "title": "Sonnet 4.6 TPM: Reservation vs Consumption",
    "view": "timeSeries",
    "metrics": [
      ["AWS/Bedrock", "EstimatedTPMQuotaUsage", "ModelId", "us.anthropic.claude-sonnet-4-6", {"label": "Quota Consumption (post-request)", "color": "#ff7f0e"}],
      ["AWS/Bedrock", "InputTokenCount", "ModelId", "us.anthropic.claude-sonnet-4-6", {"label": "Input Tokens", "color": "#1f77b4"}],
      ["AWS/Bedrock", "OutputTokenCount", "ModelId", "us.anthropic.claude-sonnet-4-6", {"label": "Output Tokens", "color": "#2ca02c"}],
      ["AWS/Bedrock", "InvocationThrottles", "ModelId", "us.anthropic.claude-sonnet-4-6", {"label": "Throttled Requests", "color": "#d62728", "yAxis": "right"}]
    ],
    "period": 60,
    "stat": "Sum",
    "annotations": {
      "horizontal": [
        {
          "label": "TPM Quota (500K)",
          "value": 500000,
          "color": "#d62728"
        },
        {
          "label": "80% Warning",
          "value": 400000,
          "color": "#ff7f0e"
        }
      ]
    }
  }
}

5. Handling Throttling Gracefully

5.1 HTTP 429 Response Structure

Bedrock returns a ThrottlingException with HTTP 429 when either TPM or RPM quota is exceeded. The response body distinguishes the cause:

  • "Too many tokens per minute" — TPM quota exhausted
  • "Too many requests per minute" — RPM quota exhausted
  • "Too many tokens per day" — Daily token cap reached (rare with properly sized quotas)
  • "Service unavailable" — Capacity constraint, not a quota violation; retry with backoff

5.2 Exponential Backoff with Jitter

The critical rule: backoff intervals must be at least 60 seconds at maximum depth, because Bedrock quota windows refresh on a 60-second cycle. Backing off for only 2–4 seconds on repeated throttles is ineffective — you will re-hit the limit on the next retry.

import boto3
import random
import time
from botocore.exceptions import ClientError

def invoke_with_backoff(client, model_id, messages, max_retries=7):
    """
    Exponential backoff with full jitter for Bedrock ThrottlingException.
    
    Retry schedule (approximate):
      Attempt 1: wait 1–2s
      Attempt 2: wait 2–4s
      Attempt 3: wait 4–8s
      Attempt 4: wait 8–16s
      Attempt 5: wait 16–32s
      Attempt 6: wait 32–64s (crosses 60s window boundary)
      Attempt 7: wait 64–128s
    """
    base_delay = 1.0
    max_delay = 120.0
    
    for attempt in range(max_retries):
        try:
            response = client.converse(
                modelId=model_id,
                messages=messages
            )
            return response
        except ClientError as e:
            error_code = e.response['Error']['Code']
            
            if error_code == 'ThrottlingException':
                if attempt == max_retries - 1:
                    raise  # exhausted all retries
                
                # Full jitter: sleep random value in [0, min(max_delay, base * 2^attempt)]
                cap = min(max_delay, base_delay * (2 ** attempt))
                sleep_time = random.uniform(0, cap)
                
                print(f"Throttled (attempt {attempt + 1}/{max_retries}). "
                      f"Sleeping {sleep_time:.1f}s before retry.")
                time.sleep(sleep_time)
                continue
            
            elif error_code == 'ServiceUnavailableException':
                # Not a quota issue — shorter backoff
                sleep_time = random.uniform(0, min(30.0, base_delay * (2 ** attempt)))
                time.sleep(sleep_time)
                continue
            
            else:
                raise  # non-retryable error
    
    raise RuntimeError(f"Failed after {max_retries} attempts")

5.3 LiteLLM Retry and Fallback Configuration

LiteLLM proxy handles retry logic at the gateway layer, keeping application code clean:

# litellm_config.yaml

model_list:
  - model_name: claude-sonnet-46
    litellm_params:
      model: bedrock/us.anthropic.claude-sonnet-4-6
      aws_region_name: us-east-1
      tpm: 480000    # 96% of 500K quota — leave headroom for retries
      rpm: 950

  - model_name: claude-sonnet-46-fallback
    litellm_params:
      model: bedrock/us.anthropic.claude-sonnet-4-6
      aws_region_name: us-west-2
      tpm: 480000
      rpm: 950

  - model_name: claude-haiku-45-emergency
    litellm_params:
      model: bedrock/us.anthropic.claude-haiku-4-5
      aws_region_name: us-east-1
      tpm: 1900000
      rpm: 1950

litellm_settings:
  # Retry settings
  num_retries: 6
  retry_after: 5          # minimum seconds before first retry
  
  # Fallback routing when primary model pool is exhausted
  fallbacks:
    - claude-sonnet-46:
        - claude-sonnet-46-fallback
        - claude-haiku-45-emergency
  
  # Context window overflow fallback (separate from quota fallback)
  context_window_fallbacks:
    - claude-sonnet-46: claude-sonnet-46-fallback

router_settings:
  routing_strategy: least-busy
  retry_policy:
    ThrottlingException:
      num_retries: 6
      retry_after: 5
    ServiceUnavailableException:
      num_retries: 3
      retry_after: 3
  
  # Enable pre-call quota checks (prevents requests that would certainly throttle)
  enable_pre_call_checks: true

# Priority reservation: prod gets 90% of capacity, dev gets 10%
priority_reservation:
  "prod": 0.9
  "dev": 0.1

5.4 Circuit Breaker Pattern

A circuit breaker prevents thundering-herd retries when an entire quota pool is exhausted. When the circuit is “open,” requests fail fast with a 503 rather than queuing behind exponential backoff delays.

import time
import threading
from enum import Enum
from collections import deque

class CircuitState(Enum):
    CLOSED = "closed"      # Normal operation
    OPEN = "open"          # Failing fast
    HALF_OPEN = "half_open"  # Testing recovery

class BedrockCircuitBreaker:
    """
    Circuit breaker for Bedrock quota exhaustion.
    
    CLOSED → OPEN when throttle_rate > threshold over window
    OPEN → HALF_OPEN after recovery_timeout seconds
    HALF_OPEN → CLOSED on first success, OPEN on first failure
    """
    
    def __init__(
        self,
        failure_threshold: float = 0.5,  # 50% throttle rate opens circuit
        window_size: int = 20,            # rolling window of N requests
        recovery_timeout: int = 60,       # seconds before trying recovery
        min_requests: int = 5             # minimum requests before evaluating
    ):
        self.failure_threshold = failure_threshold
        self.window_size = window_size
        self.recovery_timeout = recovery_timeout
        self.min_requests = min_requests
        
        self.state = CircuitState.CLOSED
        self.results = deque(maxlen=window_size)
        self.open_time = None
        self._lock = threading.Lock()
    
    def call(self, func, *args, **kwargs):
        with self._lock:
            if self.state == CircuitState.OPEN:
                elapsed = time.time() - self.open_time
                if elapsed >= self.recovery_timeout:
                    self.state = CircuitState.HALF_OPEN
                else:
                    raise RuntimeError(
                        f"Circuit OPEN. Bedrock quota pool exhausted. "
                        f"Recovery in {self.recovery_timeout - elapsed:.0f}s"
                    )
        
        try:
            result = func(*args, **kwargs)
            self._on_success()
            return result
        except Exception as e:
            if 'ThrottlingException' in str(type(e).__name__):
                self._on_failure()
            raise
    
    def _on_success(self):
        with self._lock:
            self.results.append(True)
            if self.state == CircuitState.HALF_OPEN:
                self.state = CircuitState.CLOSED
    
    def _on_failure(self):
        with self._lock:
            self.results.append(False)
            if self.state == CircuitState.HALF_OPEN:
                self.state = CircuitState.OPEN
                self.open_time = time.time()
                return
            
            if len(self.results) >= self.min_requests:
                failure_rate = self.results.count(False) / len(self.results)
                if failure_rate >= self.failure_threshold:
                    self.state = CircuitState.OPEN
                    self.open_time = time.time()

6. Reserved Service Tier vs. Quota Increase

6.1 The Four On-Demand Tiers (2026)

As of 2026, Bedrock inference is split across four service tiers. Understanding the tier stack is prerequisite to the Reserved vs. quota-raise decision:

Tier Parameter Value Priority Cost Use Case
Standard "default" (or omit) Lowest Base rate Everyday workloads
Flex "flex" Below Standard Discount Batch, async, cost-sensitive
Priority "priority" Above Standard Premium Customer-facing, latency-sensitive
Reserved "reserved" Highest + dedicated Fixed monthly fee Mission-critical, predictable workloads

All on-demand tiers (Standard, Flex, Priority) share the same TPM/RPM quota pool. Raising your quota ceiling benefits all three tiers simultaneously. Reserved tier is a separate capacity reservation — it does not draw from your on-demand quota, and does not count against it.

6.2 Requesting a Specific Tier at Runtime

# Priority tier — customer-facing real-time API
response = client.converse(
    modelId='us.anthropic.claude-sonnet-4-6',
    messages=messages,
    additionalModelRequestFields={'service_tier': 'priority'}
)

# Flex tier — background batch processing  
response = client.converse(
    modelId='us.anthropic.claude-sonnet-4-6',
    messages=messages,
    additionalModelRequestFields={'service_tier': 'flex'}
)

# Reserved tier — after purchasing reservation with AWS account team
response = client.converse(
    modelId='anthropic.claude-sonnet-4-6',
    messages=messages,
    additionalModelRequestFields={'service_tier': 'reserved'}
)

Check ResolvedServiceTier in the response to confirm which tier actually served the request (Reserved requests may resolve to Standard if reserved capacity overflows).

6.3 Decision Framework: Reserved Tier vs. Quota Increase

Use a quota increase when:

  • Your traffic is bursty or variable (Reserved tier is billed for capacity, not usage)
  • You need higher throughput but can tolerate occasional throttling at peaks
  • Your workload is not revenue-critical (SLA penalty for dropped requests is low)
  • You are still validating throughput requirements and don’t want to commit to Reserved pricing

Use Reserved tier when:

  • You have predictable, sustained TPM requirements (e.g., a document processing pipeline running at consistent load)
  • SLA penalties for throttled requests are material — Reserved targets 99.5% uptime
  • Your workload has an asymmetric token ratio: summarization (high input, low output) benefits from separate input/output TPM allocation in Reserved
  • You have validated throughput needs and are ready for a 1- or 3-month commitment
  • You are already hitting on-demand quota ceiling despite approved increases

The overflow behavior: Reserved tier automatically spills to Standard when it exceeds the reserved capacity. This is a safety net, not a design target — size your reservation to cover P95 load.

Minimum reservation requirements:

  • Input TPM: 100,000 minimum
  • Output TPM: 10,000 minimum

Access via your AWS account team (not self-service).

6.4 Reserved Tier Sizing Calculation

# Sizing worksheet for Reserved tier

# Observed from CloudWatch over trailing 30 days
observed_p95_input_tpm = 180_000   # InputTokenCount P95, 1-minute window
observed_p95_cache_write_tpm = 20_000  # CacheWriteInputTokenCount P95
observed_p95_output_tpm = 15_000    # OutputTokenCount P95

# Reserved tier counts: input + cache_write toward input TPM
# output counts at 1x (no burndown multiplier in Reserved)
required_input_reservation = observed_p95_input_tpm + observed_p95_cache_write_tpm
required_output_reservation = observed_p95_output_tpm

print(f"Recommended input TPM reservation: {required_input_reservation:,}")
print(f"Recommended output TPM reservation: {required_output_reservation:,}")

# Add 20% headroom for growth
headroom_factor = 1.2
print(f"With 20% headroom — Input: {required_input_reservation * headroom_factor:,.0f} TPM")
print(f"With 20% headroom — Output: {required_output_reservation * headroom_factor:,.0f} TPM")

7. Application Inference Profiles: Cost Tracking, Not Quota Isolation

7.1 What AIPs Do and Do Not Provide

Application Inference Profiles (AIPs) are frequently misunderstood as a quota isolation mechanism. They are not. Key facts:

  • AIPs provide cost attribution, not quota enforcement. All AIPs share the same underlying TPM/RPM quota of the model they reference.
  • AIPs attach cost allocation tags to billing records, flowing to Cost Explorer and CUR 2.0.
  • They are model-specific: one AIP per unique combination of model + tag set.
  • They replace the modelId in API calls with an AIP ARN.

For actual quota isolation per team, the recommended approach is separate AWS accounts. Separate accounts receive independent quota allocations and eliminate contention entirely.

7.2 Creating and Using an AIP for Team Cost Tracking

# Create an AIP for the data-engineering team
aws bedrock create-inference-profile \
  --inference-profile-name "data-engineering-sonnet46" \
  --description "Claude Sonnet 4.6 usage for data-engineering team" \
  --model-source copyFrom="anthropic.claude-sonnet-4-6" \
  --tags team=data-engineering,cost-center=CC-1042,env=production \
  --region us-east-1

# Application code uses the profile ARN instead of model ID
import boto3

client = boto3.client('bedrock-runtime', region_name='us-east-1')

# AIP ARN obtained from create-inference-profile output
aip_arn = "arn:aws:bedrock:us-east-1:123456789012:inference-profile/data-engineering-sonnet46"

response = client.converse(
    modelId=aip_arn,
    messages=[{'role': 'user', 'content': [{'text': 'Summarize this document'}]}]
)

7.3 AIPs with Cross-Region Profiles

AIPs can wrap a system-defined cross-region inference profile:

# Wrap the US Geo CRIP with team-specific cost tags
aws bedrock create-inference-profile \
  --inference-profile-name "ml-platform-sonnet46-us" \
  --model-source copyFrom="us.anthropic.claude-sonnet-4-6" \
  --tags team=ml-platform,cost-center=CC-2001 \
  --region us-east-1

When used this way, the AIP inherits all cross-region routing behavior from the underlying CRIP, and cost is tagged at the team level. This is the recommended pattern for organizations that want both cross-region throughput and team-level billing attribution.

7.4 Monitoring AIP Usage in CloudWatch

AIPs surface as a dimension in CloudWatch Bedrock metrics. You can isolate EstimatedTPMQuotaUsage per AIP:

aws cloudwatch get-metric-statistics \
  --namespace AWS/Bedrock \
  --metric-name EstimatedTPMQuotaUsage \
  --dimensions \
    Name=InferenceProfileArn,Value=arn:aws:bedrock:us-east-1:123456789012:inference-profile/data-engineering-sonnet46 \
  --start-time "$(date -u -v-1d '+%Y-%m-%dT%H:%M:%SZ')" \
  --end-time "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" \
  --period 3600 \
  --statistics Sum \
  --region us-east-1

8. Enterprise Quota Governance

8.1 Central Quota Pool Management

The platform engineering team owns the AWS account-level TPM/RPM quota. Individual teams consume from this pool. The governance challenge: prevent one team’s batch job from crowding out another team’s real-time user-facing API.

Three-layer governance model:

Layer 1: AWS Account quota (hard ceiling)
  ↓
Layer 2: LiteLLM virtual key limits (enforced by proxy)
  ↓
Layer 3: Application-level rate limiting (self-enforced)

8.2 LiteLLM Virtual Key Quota Allocation

LiteLLM Enterprise supports per-key and per-team TPM/RPM limits. Each team receives a virtual key with a quota ceiling that the proxy enforces before requests reach Bedrock.

Create a virtual key with quota limits via API:

# Generate a virtual key for the data-engineering team
curl -X POST "http://litellm-proxy:4000/key/generate" \
  -H "Authorization: Bearer sk-1234" \
  -H "Content-Type: application/json" \
  -d '{
    "team_id": "data-engineering",
    "key_alias": "data-eng-prod",
    "models": ["claude-sonnet-46"],
    "tpm_limit": 200000,
    "rpm_limit": 400,
    "max_budget": 500.00,
    "budget_duration": "monthly",
    "metadata": {
      "cost_center": "CC-1042",
      "env": "production",
      "priority": "prod"
    }
  }'

Allocate quota across teams (total must not exceed 90% of account quota):

Team Allocation TPM RPM Budget/Month Priority
ml-platform (real-time) 250,000 500 $5,000 prod
data-engineering (batch) 150,000 300 $3,000 dev
product-ai (internal tools) 75,000 150 $1,500 dev
emergency reserve 25,000 50 $500 prod
Total 500,000 1,000 $10,000

Reserve 10–15% for headroom. When an EstimatedTPMQuotaUsage alarm fires at 80% of account quota, the platform team can temporarily reallocate from lower-priority keys before the quota increase request is approved.

Dynamic priority reservation in LiteLLM (Enterprise feature):

# litellm_config.yaml — priority reservation
priority_reservation:
  "prod": 0.85    # 85% of proxy quota reserved for prod keys
  "dev": 0.10     # 10% for dev/batch
  # 5% unallocated — available to any priority via borrowing
  
litellm_settings:
  callbacks: ["dynamic_rate_limiter_v3"]
  database_url: "postgresql://litellm:password@rds-host:5432/litellm"

8.3 Per-Team Quota Monitoring Dashboard

Surface per-team usage back to team leads so they self-manage toward their allocation:

# CloudWatch Insights query — top consumers by virtual key (requires LiteLLM to emit custom metrics)
aws logs start-query \
  --log-group-name /litellm/requests \
  --start-time $(date -d '1 hour ago' +%s) \
  --end-time $(date +%s) \
  --query-string 'fields @timestamp, team_id, model, input_tokens, output_tokens
    | stats sum(input_tokens + output_tokens * 5) as quota_tokens_consumed by team_id
    | sort quota_tokens_consumed desc
    | limit 20'

8.4 Separating Production and Dev/Test Traffic

Anti-pattern: Dev and test environments share the same AWS account and quota pool as production. A developer running a batch evaluation job at 3 PM can spike TPM and throttle the production API.

Pattern A — account isolation (recommended):

  • Production account: dedicated TPM quota, independent from dev
  • Development account: separate quota; throttling in dev does not affect prod

Pattern B — LiteLLM quota separation within one account:

  • Prod virtual keys: high TPM ceiling, priority: prod
  • Dev virtual keys: low TPM ceiling (force batching), priority: dev
  • Reserved tier capacity (if purchased) can be restricted to prod keys via IAM condition

IAM policy — restrict Reserved tier to production role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowStandardTierOnly",
      "Effect": "Deny",
      "Action": "bedrock:InvokeModel",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "bedrock:ServiceTier": "reserved"
        },
        "StringNotEquals": {
          "aws:PrincipalTag/environment": "production"
        }
      }
    }
  ]
}

9. Cost of Throttling: Justifying Quota Increases with Business Data

9.1 Calculating the Business Cost of Dropped Requests

Quota increase requests are more likely to be approved — and approved faster — when they include a business cost calculation. AWS support prioritizes accounts that demonstrate economic impact.

Framework:

Hourly throttle cost = (throttled_requests_per_hour / total_requests_per_hour)
                     × requests_per_hour
                     × revenue_per_successful_request
                     × sla_penalty_multiplier

Example calculation for a document processing platform:

  • 10,000 requests/hour during business hours
  • At peak, 8% throttle rate (800 throttled requests/hour)
  • Each successful request processes a document worth $0.50 in analyst time saved
  • SLA: 4-hour processing window; throttled requests miss SLA and require manual re-submission at 2× cost
requests_per_hour = 10_000
throttle_rate = 0.08
revenue_per_request = 0.50   # analyst time saved per document
resubmission_cost_multiplier = 2.0  # manual resubmission cost
sla_penalty_per_miss = 25.00  # contractual SLA penalty per missed document

throttled_per_hour = requests_per_hour * throttle_rate  # 800
direct_revenue_loss = throttled_per_hour * revenue_per_request  # $400/hour
resubmission_overhead = throttled_per_hour * (revenue_per_request * (resubmission_cost_multiplier - 1))  # $400/hour
sla_penalties = throttled_per_hour * sla_penalty_per_miss  # $20,000/hour

total_hourly_cost = direct_revenue_loss + resubmission_overhead + sla_penalties
# $400 + $400 + $20,000 = $20,800/hour

# Monthly cost (8 business hours/day, 22 days)
monthly_cost = total_hourly_cost * 8 * 22
print(f"Monthly throttling cost: ${monthly_cost:,.0f}")  # $3,660,800

Include this calculation in your quota increase request justification. The business case dwarfs the cost of a quota increase (which is free — you only pay for tokens consumed).

9.2 SLA Impact Calculation for CloudWatch Alarm Bodies

When setting up PagerDuty/OpsGenie integrations on the InvocationThrottles alarm, include calculated business impact in the alert body:

# Lambda function triggered by CloudWatch alarm → SNS → PagerDuty
import json
import boto3
from datetime import datetime, timedelta

def calculate_throttle_impact(model_id: str, region: str, window_minutes: int = 5) -> dict:
    cw = boto3.client('cloudwatch', region_name=region)
    
    end = datetime.utcnow()
    start = end - timedelta(minutes=window_minutes)
    
    # Get throttle count
    throttle_resp = cw.get_metric_statistics(
        Namespace='AWS/Bedrock',
        MetricName='InvocationThrottles',
        Dimensions=[{'Name': 'ModelId', 'Value': model_id}],
        StartTime=start, EndTime=end, Period=window_minutes * 60,
        Statistics=['Sum']
    )
    
    # Get total invocations
    total_resp = cw.get_metric_statistics(
        Namespace='AWS/Bedrock',
        MetricName='Invocations',
        Dimensions=[{'Name': 'ModelId', 'Value': model_id}],
        StartTime=start, EndTime=end, Period=window_minutes * 60,
        Statistics=['Sum']
    )
    
    throttled = throttle_resp['Datapoints'][0]['Sum'] if throttle_resp['Datapoints'] else 0
    total = total_resp['Datapoints'][0]['Sum'] if total_resp['Datapoints'] else 1
    
    throttle_rate = throttled / max(total, 1)
    revenue_at_risk_per_minute = throttled / window_minutes * 0.50  # $0.50/request
    
    return {
        'throttled_requests': int(throttled),
        'throttle_rate_pct': round(throttle_rate * 100, 1),
        'estimated_revenue_at_risk_per_minute': round(revenue_at_risk_per_minute, 2),
        'recommended_action': 'Request quota increase immediately' if throttle_rate > 0.1 else 'Monitor'
    }

10. 2026 Quota Landscape Changes

10.1 Key Quota Increases in 2026

Change Date Details
Claude Sonnet 4.5 default quota increase in GovCloud February 2026 Default TPM raised to 5,000,000 TPM for GovCloud US-West and US-East
Reserved Tier for Claude Sonnet 4.5 in GovCloud January 2026 Reserved tier availability extended to GovCloud regions
New CloudWatch metrics: EstimatedTPMQuotaUsage + TimeToFirstToken March 2026 Automated, no opt-in; zero additional cost
Bedrock Service Quotas expansion May 2026 More model quotas now visible and adjustable directly via Service Quotas console without opening support tickets
Global cross-region inference expanded to SE Asia / Pacific 2025–2026 Thailand, Malaysia, Singapore, Indonesia, Taiwan, New Zealand added as destination regions

10.2 Flex Tier Quota Considerations

The Flex tier (introduced in 2025, expanded in 2026) has a separate quota from Standard tier in some configurations. Check the Service Quotas console for “Flex InvokeModel tokens per minute” entries for your target models. Flex tier is designed for batch workloads that can tolerate queuing delays — appropriate for nightly report generation, evaluation runs, and data processing pipelines where latency is not a constraint.

10.3 Batch Inference Quotas

For very high-volume offline workloads, Batch Inference bypasses on-demand TPM quotas and operates under separate job-level quotas:

Quota Default
Batch inference input file size per job 1 GB
Batch inference cumulative job size per account 5 GB
Concurrent batch jobs 5 (adjustable)

Batch jobs are priced at a discount vs. on-demand and are the correct architecture for pre-processing large document corpora, nightly model evaluations, or bulk classification tasks where results are not needed in real time.

10.4 bedrock-mantle Endpoint Quota Structure (Separate from bedrock-runtime)

The bedrock-mantle endpoint (OpenAI-compatible Chat Completions and Responses APIs) applies separate input and output TPM quotas — distinct from bedrock-runtime. Teams migrating from OpenAI SDK to Bedrock via the OpenAI-compatible endpoint need to check mantle-specific quotas in the Service Quotas console. The mantle endpoint supports different models and has different quota increase pathways (documented at quotas-mantle.md in the Bedrock userguide).

11. Operational Runbook: Responding to a Quota Incident

Triage sequence when InvocationThrottles alarm fires:

Step 1 — Identify which quota is exhausted (90 seconds)

# Check TPM consumption
aws cloudwatch get-metric-statistics \
  --namespace AWS/Bedrock \
  --metric-name EstimatedTPMQuotaUsage \
  --dimensions Name=ModelId,Value=us.anthropic.claude-sonnet-4-6 \
  --start-time $(date -u -v-5M '+%Y-%m-%dT%H:%M:%SZ') \
  --end-time $(date -u '+%Y-%m-%dT%H:%M:%SZ') \
  --period 60 --statistics Sum --region us-east-1

# Check RPM consumption
aws cloudwatch get-metric-statistics \
  --namespace AWS/Bedrock \
  --metric-name Invocations \
  --dimensions Name=ModelId,Value=us.anthropic.claude-sonnet-4-6 \
  --start-time $(date -u -v-5M '+%Y-%m-%dT%H:%M:%SZ') \
  --end-time $(date -u '+%Y-%m-%dT%H:%M:%SZ') \
  --period 60 --statistics Sum --region us-east-1

Step 2 — Immediate mitigation (under 5 minutes)

  • If LiteLLM is deployed: verify fallback routing is active; check proxy logs for queue depth
  • Reduce max_tokens on non-critical requests to free reservation headroom
  • Temporarily downgrade non-urgent batch traffic to Haiku (much higher default quota)
  • If one team’s batch job caused the spike: kill or throttle that job via virtual key suspension in LiteLLM

Step 3 — Open quota increase request (under 15 minutes)

# Get current quota code from console, then:
aws service-quotas request-service-quota-increase \
  --service-code bedrock \
  --quota-code L-XXXXXXXX \
  --desired-value 2000000 \
  --region us-east-1 \
  --query "RequestedQuota.Id" --output text

Include in justification: current CloudWatch EstimatedTPMQuotaUsage data, number of throttled requests in the incident window, and business impact calculation.

Step 4 — Post-incident (within 24 hours)

  • Lower alarm threshold from 80% to 70%
  • Review max_tokens distribution across active request types
  • Evaluate whether Reserved tier is appropriate for the affected workload
  • Update quota allocation table in team documentation

References