← Agent Frameworks 🕐 24 min read
Agent Frameworks

AI Gateway Patterns for Bedrock — API Management and Cost Governance (2026)

1. Amazon API Gateway Costs for Bedrock Proxying · March 2026

Enterprise teams proxying Amazon Bedrock through an API management layer face a compounding cost problem: every governance feature adds infrastructure spend on top of Bedrock's already-variable token

Enterprise teams proxying Amazon Bedrock through an API management layer face a compounding cost problem: every governance feature adds infrastructure spend on top of Bedrock’s already-variable token pricing. This note maps the full stack — from AWS-native API Gateway and VPC endpoints through third-party gateways (LiteLLM, Portkey) and gateway-level optimization techniques (semantic caching, cross-region routing) — with concrete pricing and architectural decision criteria at each layer.

API Gateway is the default proxy choice for teams already in the AWS ecosystem. The cost model has three components: per-request charges, data transfer out, and optionally caching or WebSocket connection time.

REST vs HTTP API

Dimension REST API HTTP API
Per million requests $3.50 $1.00
Caching Yes (additional cost) No
Usage plans + API keys Yes No
WAF integration Yes No
Request transformation Yes No
JWT authorizer No Yes (native)

HTTP API is 71% cheaper and covers the majority of Bedrock proxy patterns: route the request to Bedrock runtime, pass through the response. REST API is only warranted when you need usage plans for per-team throttle enforcement, WAF for prompt-injection blocking, or request transformation to normalize different client payload shapes before they reach Bedrock.

At 10M requests/month: REST API costs $35/month in request charges; HTTP API costs $10. The $25 gap looks trivial until you add data transfer. At 100M requests/month — a realistic scale for an enterprise platform team serving dozens of product teams — the gap widens to $250/month: $350 for REST API vs $100 for HTTP API. That $250/month differential alone exceeds the monthly cost of a Redis instance for rate limiting.

When REST API is mandatory for Bedrock proxying:

  • Usage Plans + API keys for per-team throttle enforcement (HTTP API requires a separate Lambda authorizer to replicate this)
  • WAF integration for blocking malicious prompts or IP-based abuse
  • Caching at the API Gateway layer for identical request bodies (usually superseded by gateway-level semantic caching — see Section 7)
  • Stage-level access logging with execution logging to CloudWatch (HTTP API supports only access logging, not execution logging)

Data Transfer

Bedrock responses are token-heavy. A Claude 3.5 Sonnet response averaging 500 output tokens is approximately 2KB of JSON. At 10M requests/month, that is 20GB out. Data transfer charges through API Gateway apply at the standard AWS internet egress rate: first 10TB/month at $0.09/GB. That adds $1.80/month at this scale — small, but it scales linearly. At 100M requests/month and 2KB average response, data transfer is $18/month, negligible. The cost inflates when you log full request/response bodies at the gateway layer (see Section 9).

WebSocket for Streaming

Bedrock’s Converse API and InvokeModel both support streaming responses via Server-Sent Events (SSE). API Gateway WebSocket adds a layer for bidirectional streaming, charged at:

  • $1.00 per million messages (32KB increments — a 64KB chunk counts as two messages)
  • $0.25 per million connection minutes for persistent connections

For typical streaming workloads (10–30-second completions, 30–50 chunks per response), WebSocket API cost runs $1–$2 per million completions on top of the per-request charge. For most Bedrock workloads, HTTP API with SSE pass-through is preferable: it avoids WebSocket connection management overhead and the per-minute connection charge.

Practical threshold: Use WebSocket only when clients need to send mid-stream interrupts (rare in enterprise internal tooling). SSE over HTTP API handles one-directional streaming at lower cost and complexity.

2. VPC Endpoints for Bedrock

Interface Endpoint vs Gateway Endpoint

Bedrock uses PrivateLink-backed interface endpoints, not Gateway endpoints. Gateway endpoints (S3, DynamoDB) are free. Interface endpoints are not.

Interface endpoint pricing:

  • $0.01/hour per Availability Zone = $7.30/AZ/month
  • Data processing: $0.01/GB for the first petabyte

A 3-AZ production VPC with a Bedrock interface endpoint costs $21.90/month in endpoint charges alone, plus data processing on all traffic routed through it.

When VPC Endpoints Pay Off vs Public Internet

The break-even analysis depends on what you are avoiding:

Scenario A: NAT Gateway replacement If your Bedrock clients are in private subnets, traffic to Bedrock over the public internet requires a NAT Gateway. NAT Gateway costs $0.045/hour ($32.85/month) plus $0.045/GB processed. At even modest traffic (50GB/month), NAT Gateway costs $32.85 + $2.25 = $35.10/month per AZ. A Bedrock interface endpoint at $7.30/AZ + $0.01/GB ($0.50) = $7.80/month — a 77% cost reduction at this traffic level.

Scenario B: Security/compliance requirement (no internet egress) If your VPC has no internet gateway by policy (common in regulated industries), interface endpoints are mandatory regardless of cost.

Scenario C: Low-traffic internal tooling For workloads generating <5GB/month of Bedrock traffic from public subnets, public internet access with an Internet Gateway (free) is cheaper than $21.90/month in endpoint charges across 3 AZs.

Scenario D: Multi-AZ high-availability Deploying interface endpoints in 3 AZs for HA does not triple your Bedrock traffic volume — it triples your endpoint availability. If AZ-a’s endpoint is unhealthy, traffic fails over to AZ-b’s endpoint transparently. The per-AZ endpoint charge is the cost of that availability. Compare: a NAT Gateway with 3 AZs costs $32.85 × 3 = $98.55/month plus data processing; 3-AZ Bedrock interface endpoints cost $21.90/month plus data processing. The VPC endpoint path is cheaper even before network savings.

Recommendation: Deploy Bedrock interface endpoints in any VPC where Bedrock is a production dependency and NAT Gateway is already in use, or where security posture requires no public egress. The cost is typically dominated by NAT Gateway savings, not endpoint charges. Single-AZ deployments for non-production environments can use one endpoint at $7.30/month to minimize cost during development.

3. LiteLLM as AI Gateway — Infrastructure TCO

LiteLLM is the leading open-source AI gateway, maintained by BerriAI, with native AWS Bedrock support. The software itself is Apache 2.0 licensed; enterprise features ($250/month license) add SSO, advanced rate limiting, and audit logs.

Required Infrastructure Components

A production LiteLLM deployment for Bedrock requires:

Proxy Server (ECS Fargate):

  • Minimum viable: 0.5 vCPU / 1GB RAM per task for routing logic only
  • Production (with caching/logging): 1 vCPU / 2GB RAM, 2+ tasks for HA
  • Fargate cost at us-east-1: $0.04048/vCPU-hour, $0.004445/GB-hour
  • 2 tasks × (1 vCPU + 2GB) running 24/7 = ~$87/month compute

PostgreSQL (RDS or Aurora Serverless):

  • Stores SpendLogs: per-request cost tracking, virtual key metadata, team budgets
  • Minimum: db.t3.micro ($15/month) for low-volume; db.t3.small ($30/month) for >1M requests/month
  • Aurora Serverless v2 (minimum 0.5 ACU): ~$43/month base, scales with load
  • Recommended for production: Aurora Serverless v2 to handle usage spikes without pre-provisioning

Redis (ElastiCache):

  • Required for: rate limit counters, exact-match response caching, distributed lock for budget enforcement
  • cache.t3.micro ($12/month) sufficient for rate limiting only
  • cache.t3.small ($24/month) if using response caching with significant hit volume
  • For semantic caching: cache.r7g.large ($130/month) — vector storage requires more memory

Network and Misc:

  • ALB in front of Fargate: ~$16/month + $0.008/LCU-hour
  • CloudWatch logging: $0.50/GB ingested (see Section 9 for how quickly this scales)
  • Secrets Manager for key rotation: $0.40/secret/month

Full TCO Estimate (Monthly)

Component Conservative Production
ECS Fargate compute $44 $87
RDS PostgreSQL $15 $43 (Aurora Serverless)
ElastiCache Redis $12 $24–$130
ALB $16 $16
CloudWatch Logs $10–$50 $50–$200
LiteLLM Enterprise license $0 (OSS) $250
Total infrastructure $97–$137 $470–$726
DevOps labor (0.1–0.2 FTE) $1,000–$2,000 $1,500–$3,000
All-in monthly TCO $1,097–$2,137 $1,970–$3,726

Labor is the dominant cost. Teams often underestimate the ongoing burden: schema migrations for SpendLogs, Redis memory tuning as usage grows, Fargate task definition updates when LiteLLM releases break changes, incident response when rate limit enforcement blocks a production workload.

AWS publishes a reference architecture (aws-solutions-library-samples/guidance-for-multi-provider-generative-ai-gateway-on-aws) that combines LiteLLM with Bedrock on ECS Fargate — this is the recommended starting point rather than building the deployment from scratch.

4. Portkey AI Gateway — Pricing and Enterprise Positioning

Portkey was acquired by Palo Alto Networks in April 2026. The open-source gateway core (Apache 2.0) is available for self-hosting; the managed platform adds persistent observability and compliance certifications.

Pricing Tiers (June 2026)

Tier Price Logs/Month Retention Notes
Developer Free 10,000 3 days Prototyping only
Production $49/month 100,000 30 days +$9/100K additional requests
Enterprise Custom Unlimited Custom SSO, VPC hosting, HIPAA, SOC2, RBAC

At 1M requests/month, Production tier: $49 + (9 × $9) = $130/month for the managed platform. Enterprise custom pricing typically runs $2,000–$10,000+/month for large-volume deployments.

Self-Hosted vs Managed

Self-hosted (open-source): Core routing, fallbacks, load balancing, and guardrails with no per-request fee. You manage infrastructure (similar TCO to LiteLLM, minus the enterprise license). No persistent log storage, no managed observability dashboard.

Managed cloud: Portkey hosts the gateway and provides: persistent request logs with cost/latency breakdowns, semantic caching layer, RBAC for team spend governance, audit trails, SOC2/HIPAA compliance. For regulated industries or teams without DevOps bandwidth, the managed platform cost ($130–$2,000+/month) is usually justified.

LiteLLM vs Portkey — Decision Matrix

Criteria LiteLLM Portkey
Bedrock native support Yes Yes
Self-hosted option Yes (full features) Yes (routing only)
Spend tracking granularity Per virtual key, per team Per key, per user, per model
Rate limiting Yes (Redis-backed) Yes (managed)
Semantic caching Plugin (requires config) Built-in (managed tier)
PII redaction / guardrails Limited (enterprise) Yes (all paid tiers)
Compliance certifications None SOC2, HIPAA (enterprise)
DevOps burden High Low (managed)
Lock-in risk Low (Apache 2.0) Medium (Palo Alto acquisition)

Rule of thumb: LiteLLM for teams with DevOps capacity who prioritize control and cost. Portkey managed for teams where compliance, observability, and low operational overhead outweigh platform costs. The Palo Alto acquisition introduces roadmap and pricing risk for Portkey enterprise customers; evaluate in 2027.

Cross-Account Architecture

The canonical enterprise pattern: a central “AI Platform” AWS account owns Bedrock endpoints, rate limit infrastructure, and cost governance tooling. Application accounts access Bedrock through this central account via PrivateLink.

Components:

  1. Central account creates a VPC Endpoint Service backed by an NLB in front of the AI gateway (LiteLLM or custom proxy)
  2. Consumer accounts create interface endpoints pointing to the central account’s endpoint service
  3. Cross-account IAM roles grant bedrock:InvokeModel to the gateway role only; application accounts never hold direct Bedrock credentials
  4. AWS Resource Access Manager (RAM) can share endpoint services across an AWS Organization

Cost Implications

Cross-account PrivateLink endpoint service pricing:

  • $0.05/hour per remote region with attached VPCEs = $36/month per consumer account connecting from another region
  • $0.01/GB data processing on cross-account traffic
  • Plus the consumer-side interface endpoint: $0.01/hour/AZ

For 5 application accounts in the same region accessing a central AI Platform account, total PrivateLink overhead: 5 × $7.30 (consumer endpoints, 1 AZ each) = $36.50/month + data processing. Cross-region adds the $0.05/hour endpoint service charge per region.

vs. Cross-Region Bedrock Routing: If application accounts are multi-region and invoking Bedrock directly in each region, cross-region data transfer ($0.02/GB inter-region) typically exceeds centralized PrivateLink costs for workloads above ~500GB/month. Below that threshold, direct per-region Bedrock access is cheaper.

6. Rate Limiting and Throttling Patterns

Token Bucket vs Leaky Bucket

Token bucket is standard for LLM rate limiting because it allows short bursts (accommodating prompt variation) while enforcing a long-run average. A team budget of 100K tokens/minute with a burst capacity of 200K tokens handles occasional large prompts without blocking the team.

LiteLLM implements token bucket rate limiting via Redis sorted sets: each virtual API key gets a bucket, requests decrement it, a background job refills at the configured rate. Redis is mandatory — in-memory rate limiting breaks under multi-instance deployments.

Team-Level Quota Enforcement

LiteLLM’s virtual key system maps API keys to teams, each with configurable:

  • max_budget: hard spend cap in USD
  • tpm_limit: tokens per minute
  • rpm_limit: requests per minute
  • model_list: which Bedrock models the team can invoke

When a team hits its budget cap, LiteLLM returns HTTP 429. The PostgreSQL SpendLogs table accumulates per-request costs (input tokens × price + output tokens × price), and the proxy evaluates against the team’s budget on each request.

Database load consideration: At 1M requests/day, SpendLogs generates 1M rows/day. After 30 days, that is 30M rows. Without partitioning or archival, query performance for budget lookups degrades significantly. Production LiteLLM deployments should use table partitioning by date and archive rows older than 90 days to cold storage (S3).

Sliding Window vs Fixed Window

Token bucket (sliding window) is preferable to fixed window for LLM rate limiting. Fixed window counters reset at a hard boundary (e.g., top of each minute), which means a burst at 0:59 and 1:01 effectively doubles the allowed throughput for a brief period. Sliding window avoids this by tracking the count of requests in the trailing N seconds, but requires Redis sorted sets rather than simple counters — slightly higher Redis CPU cost but more accurate enforcement.

LiteLLM implements sliding window via Redis sorted sets for rpm_limit and a separate token counter for tpm_limit. Both checks happen synchronously on each request, adding two Redis round-trips to every request path. At p50 latency with a local Redis cluster, this overhead is 2–4ms total.

Rate Limit Infrastructure Cost

Redis for rate limiting only (no semantic caching): cache.t3.micro at $12/month handles up to ~5,000 requests/second of counter operations. For >10,000 RPS, move to cache.r7g.medium ($85/month) or cluster mode.

The hidden cost is not Redis — it is the latency added by the rate limit check. A Redis round-trip adds 1–3ms. At p99 latency budgets for interactive applications, this is acceptable. For batch inference pipelines where latency is not a constraint, even a 10ms rate limit check is irrelevant.

Budget Exhaustion Handling

When a team’s max_budget is hit, LiteLLM returns HTTP 429 with a body indicating budget exhaustion (distinct from rate limit 429). Application code must handle both 429 subtypes differently: a rate limit 429 should retry with exponential backoff; a budget exhaustion 429 should surface a user-facing error and alert the team’s engineering lead. Conflating the two failure modes is a common integration bug that causes silent request drops when teams are unaware their monthly budget has been consumed.

Alert pattern: configure a CloudWatch alarm on LiteLLM’s logged budget_exceeded events, with SNS notification to the team’s Slack channel at 80% and 100% of monthly budget.

7. Gateway-Level Semantic Caching

Architecture

Semantic caching intercepts requests before they reach Bedrock and returns a cached response if the incoming prompt is semantically similar to a prior prompt. The pipeline:

  1. Embed the incoming prompt (e.g., text-embedding-3-small at $0.02/1M tokens)
  2. Vector similarity search against the cache index (Redis Vector Similarity Search or a dedicated vector DB)
  3. If similarity > threshold (typically 0.95 cosine similarity): return cached response, log cache hit
  4. If below threshold: send to Bedrock, cache the response with its embedding

Dual-layer architecture improves hit rate: exact-match cache (Redis string lookup by prompt hash) handles repeated identical prompts at zero embedding cost; semantic cache handles paraphrases.

Cache Hit Rate Economics

Published benchmarks (2026) report 61–73% cache hit rates in high-repetition enterprise workloads (internal helpdesks, FAQ bots, code review tooling where similar questions recur). Customer-facing applications with more diverse queries see 20–40% hit rates.

Break-even at 40% hit rate, Claude Sonnet 4.5 pricing ($3/1M input, $15/1M output):

Assume 500 input tokens + 500 output tokens average per request, 1M requests/month:

  • Without caching: (500 × $3 + 500 × $15) / 1,000,000 × 1,000,000 = $9,000/month
  • With 40% hit rate: 600K requests reach Bedrock = $5,400/month
  • Embedding cost: 1M × 500 tokens × $0.02/1M = $10/month
  • Redis vector cache (r7g.large): $130/month
  • Net savings: $3,460/month against $140/month in caching infrastructure = 25× ROI

At 70% hit rate, savings are $6,300/month against the same $140 infrastructure, 45× ROI.

Cost vs Accuracy Tradeoff

The threshold parameter is the key lever. At 0.95 cosine similarity, you occasionally return cached answers to subtly different questions — this is the main quality risk. For factual internal tooling (HR policies, IT help desk), 0.95 is safe. For legal analysis or financial modeling, use 0.99 or disable semantic caching entirely; a wrong cached answer costs more than the inference savings.

LiteLLM supports semantic caching via GPTCache integration. Portkey’s managed tier includes semantic caching as a first-class feature with configurable similarity thresholds and cache TTL. The open-source Bifrost gateway (Go-based) ships semantic caching as a built-in plugin.

Cache Invalidation and TTL Strategy

LLM responses can become stale if the underlying model changes (e.g., Claude Sonnet 3.7 → 4.5 rollout) or if the data the model was asked about changes (e.g., “what is today’s price of X”). Cache TTL must reflect the temporal sensitivity of the queries being served:

Query Type Recommended TTL
Static policy / HR FAQ 7–30 days
Product documentation 24–72 hours
Market data commentary 1–4 hours
Real-time analytics Do not cache

Set TTL at the prompt-category level using LiteLLM’s cache_params — inject a ttl field in the request metadata based on a routing rule that classifies the prompt type. For most enterprise internal tooling, a 24-hour TTL on semantic cache entries captures the cost savings while keeping responses fresh.

Embedding Model Cost for Semantic Cache

The semantic cache pipeline requires embedding each incoming prompt. Choices:

  • Amazon Titan Text Embeddings V2 (via Bedrock): $0.00002/1K input tokens — a 500-token prompt costs $0.00001 per lookup. At 1M lookups/month: $10/month. Keeps the embedding call in-VPC if using interface endpoints.
  • OpenAI text-embedding-3-small: $0.02/1M tokens — comparable cost, requires external API call
  • Local sentence-transformers (running on the gateway host): zero per-call cost, adds ~50ms CPU latency per lookup, requires sufficient Fargate CPU allocation (add 0.5 vCPU to task definition = ~$18/month)

For cost and latency balance, Titan Text Embeddings V2 via Bedrock is the recommended choice for AWS-native deployments: no external egress, low per-call cost, and the embedding call benefits from the same VPC endpoint as model invocations.

8. Load Balancing Across Bedrock Regions

AWS Native: Global Cross-Region Inference

Bedrock’s built-in Global cross-Region inference (GCRI) profiles automatically route to regions with available capacity. Critically: billing is based on the source region rate, not the destination region where compute runs. There are no routing surcharges. GCRI for Claude Sonnet 4.5 pricing is approximately 10% cheaper than single-region on-demand pricing due to AWS’s ability to optimize compute allocation globally.

GCRI profiles:

  • Geographic profiles (US, EU, Japan, Australia): route within a geography, maintaining data residency
  • Global profile: route across all commercial regions for maximum throughput and lowest cost

For most enterprise workloads, GCRI geographic profiles are the right default. They eliminate the need for custom cross-region routing logic in the gateway layer.

LiteLLM Load Balancing for Custom Routing

When GCRI is insufficient (e.g., mixing Bedrock with other providers, or implementing custom cost-weighted routing), LiteLLM’s router_strategy settings offer:

  • least-busy: routes to the endpoint with the lowest current request queue
  • latency-based: routes based on measured p50 response time per endpoint
  • cost-based: routes to the cheapest model/region combination for the token count

For Bedrock specifically, cost-based routing is most useful when an application can accept Claude Haiku for short requests and escalates to Sonnet only above a token threshold. This requires the gateway to inspect the prompt before routing — LiteLLM’s rpm_limit per model configuration handles this.

Model Tiering via Gateway Routing

A tiered routing pattern reduces cost by matching model capability to request complexity:

Simple queries (< 200 input tokens, no tool use) → Claude Haiku 3.5 ($0.80/1M in, $4/1M out)
Standard queries (200–2K tokens) → Claude Sonnet 4.5 ($3/1M in, $15/1M out)
Complex reasoning (> 2K tokens, extended thinking) → Claude Opus 4 ($15/1M in, $75/1M out)

At 1M requests/month distributed 60/35/5 across tiers:

  • Without tiering (all Sonnet): 1M × avg $9 = $9,000/month
  • With tiering: 600K × $1.44 + 350K × $9 + 50K × $45 = $864 + $3,150 + $2,250 = $6,264/month
  • Savings: 30% on Bedrock spend, gateway adds 0 marginal cost for the routing decision

LiteLLM implements this via model_alias routing rules with token count thresholds extracted from the request body before dispatch. The token count check adds <1ms to request processing time.

Cross-region routing cost consideration: If you implement custom cross-region load balancing (bypassing GCRI) and your clients are in us-east-1 but you route to eu-west-1, inter-region data transfer adds $0.02/GB. For typical Bedrock responses, this is $0.00004 per request — negligible until scale exceeds hundreds of millions of requests.

9. Request/Response Logging Overhead

CloudWatch Logs Cost at Scale

Full request/response logging at the API Gateway layer means storing complete prompt text and completion text. At average 500 input + 500 output tokens (~4KB per request):

Monthly Volume Log Data/Month CloudWatch Ingestion Cost
1M requests 4GB $2.00
10M requests 40GB $20.00
100M requests 400GB $200.00
1B requests 4TB $2,000.00

CloudWatch Logs storage adds $0.03/GB/month after the first 15 days. For 400GB at 100M requests/month: $12/month storage (assuming 30-day retention).

These numbers look manageable until you factor in CloudWatch Logs Insights queries for debugging and cost attribution: $0.005/GB scanned. A single debugging session scanning 400GB costs $2. At 50 debugging sessions/month: $100/month in query costs alone.

Sampling Strategies

For workloads above 10M requests/month, log sampling is essential:

Tiered sampling approach:

  1. Metadata always: timestamp, request ID, virtual key/team ID, model, input token count, output token count, latency, cost, cache hit/miss — ~200 bytes per request, logged at 100%
  2. Error traces always: any 4xx/5xx response, logged at 100% with full request/response body
  3. Full body sampling: 5–10% of successful requests, randomly sampled, for quality monitoring
  4. Structured traces: send to X-Ray or OpenTelemetry collector instead of CloudWatch for lower cost at high volume (OpenTelemetry to S3 via Kinesis Firehose runs ~$0.029/GB vs $0.50/GB for CloudWatch ingestion)

LiteLLM supports S3 logging as an alternative to CloudWatch for SpendLogs and request bodies — this reduces logging costs by approximately 90% for high-volume deployments.

Structured Metadata Logging Architecture

The recommended pattern separates lightweight metadata (always logged at 100%, low volume) from heavyweight payload logging (sampled, high volume):

Tier 1 — Metrics stream (always-on, low volume): Send to CloudWatch Metrics via EMF (Embedded Metric Format) from the gateway: request_count, input_tokens, output_tokens, cost_usd, latency_ms, cache_hit, model_id, team_id. EMF bypasses Logs ingestion cost for metric data — metrics are free to store in CloudWatch, only alarms and dashboards add cost.

Tier 2 — Structured access logs (always-on, medium volume): Log one JSON line per request with metadata fields only (no prompt/response body): ~200 bytes/request. At 10M requests/month: 2GB = $1.00/month ingestion. This tier supports cost attribution queries (Section 11) without payload storage costs.

Tier 3 — Full payload logs (10% sample): Store to S3 via Kinesis Firehose in gzip-compressed Parquet for cost efficiency. Athena queries on S3 for debugging run at $5/TB scanned — far cheaper than CloudWatch Insights for high-volume forensic analysis. At 10M requests/month with 10% sampling and 4KB average payload: 4GB/month to S3 = ~$0.09/month storage.

This three-tier architecture delivers full observability at <$5/month in logging infrastructure costs for most production workloads.

10. Auth and API Key Management Costs

API Gateway API Keys vs Cognito

API Gateway native API keys (REST API only):

  • No per-key cost
  • Managed via Usage Plans with rate/quota limits per key
  • No authentication — API keys in API Gateway are for throttling and quota tracking, not security
  • Key rotation: manual via API/Console; no automated rotation natively

Cognito User Pools as authorizer:

  • $0.015/MAU (Essentials tier) above 10,000 free MAUs
  • For an internal API with 50 teams (each a “user”): stays within free tier
  • For enterprise deployments with 500+ API consumers: $0.015 × 490 = $7.35/month — negligible
  • Supports JWT-based machine-to-machine (M2M) clients at separate pricing: $0.015/M token requests after 100K free
  • Automatic token expiry and rotation built-in; no Secrets Manager overhead for token rotation

Secrets Manager for API key storage:

  • $0.40/secret/month
  • 50 team API keys: $20/month
  • Automatic rotation supported via Lambda

Recommendation: For internal Bedrock gateway, use Cognito M2M clients with client credentials flow. Cost is negligible (<$10/month for most enterprise scales), rotation is automated, and JWT validation is native to HTTP API’s JWT authorizer — no Lambda authorizer needed, which eliminates the additional invocation cost (~$0.20/1M authorizer invocations for Lambda).

API Key Rotation Overhead

API Gateway native API keys have no automated rotation mechanism. Key rotation requires:

  1. Generate new key (API call or Terraform apply)
  2. Distribute to all clients — the non-trivial step in large organizations
  3. Deprecate old key after a cutover window
  4. Remove old key

For 50 teams with quarterly rotation cycles, this is a 200-operation/year process. At 10 minutes per rotation (generating + communicating + confirming receipt): 33 hours/year of engineering time. At $150/hour fully loaded: $5,000/year in rotation overhead.

Cognito M2M with short-lived tokens (1-hour TTL) eliminates rotation ceremonies entirely. Client applications call Cognito’s token endpoint to get a fresh JWT before each session or periodically, and the gateway validates the JWT signature without a database lookup. Rotation reduces to Cognito client secret rotation (AWS CLI, once per year, zero distribution overhead because the client secret rotates in Secrets Manager and the application fetches it automatically).

Secrets Manager vs Parameter Store for API key storage:

  • Secrets Manager: $0.40/secret/month, built-in rotation Lambda support, cross-account sharing. Best for production API keys requiring automated rotation.
  • Parameter Store SecureString: $0.05/parameter/month (Advanced tier), no native rotation. Best for non-rotating configuration values.

At 50 secrets (team API keys): $20/month for Secrets Manager vs $2.50/month for Parameter Store. The $17.50/month difference is justified by automated rotation support for sensitive credentials.

11. Cost Attribution Through the Gateway Stack

Tracing a single user request from API key to AWS Cost and Usage Report (CUR):

User/Team API Key
    ↓
API Gateway (REST) — tagged: team=engineering, env=prod
    ↓ (Authorization: Usage Plan maps key → team)
LiteLLM Proxy (ECS Fargate)
    ↓ — SpendLogs: virtual_key, team_id, model, input_tokens, output_tokens, cost, timestamp
    ↓ (Optional: semantic cache hit → return cached, log cache_hit=true)
Amazon Bedrock Runtime
    ↓ — AWS Cost and Usage Report: resource tag bedrock-team=engineering
CUR aggregation

Tagging strategy for CUR attribution:

  • Tag all Bedrock API calls via the x-amzn-bedrock-additional-model-request-fields header (available in Bedrock’s Converse API)
  • AWS Cost Explorer can then filter Bedrock costs by tag
  • LiteLLM’s SpendLogs provide finer granularity (per-key, per-request) than CUR (per-account, per-service aggregated hourly)

Cost attribution gap: CUR reports Bedrock costs in aggregate per account; it does not break down by virtual team key. LiteLLM’s PostgreSQL SpendLogs are the ground truth for intra-account cost attribution. Build a monthly reconciliation process: sum SpendLogs costs per team, compare to the account-level CUR Bedrock line, and flag discrepancies >5% (which indicate uncaptured direct Bedrock calls bypassing the gateway).

Enforcement pattern for preventing gateway bypass: Attach a Service Control Policy (SCP) at the AWS Organizations level that denies bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream for all principals except the gateway’s IAM role ARN. This makes bypassing the gateway structurally impossible without explicit SCP exceptions, ensuring 100% of Bedrock spend flows through the attribution chain.

{
  "Effect": "Deny",
  "Action": ["bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream"],
  "Resource": "*",
  "Condition": {
    "StringNotEquals": {
      "aws:PrincipalARN": "arn:aws:iam::ACCOUNT_ID:role/ai-gateway-execution-role"
    }
  }
}

This SCP pattern, combined with LiteLLM’s SpendLogs, closes the attribution gap completely. CUR becomes a verification layer; SpendLogs become the billing source of truth for chargeback to business units.

12. Build vs Buy Decision Framework

Option A: Thin Custom Gateway (Lambda + API Gateway)

Appropriate when:

  • Single model provider (Bedrock only)
  • <5 teams sharing the gateway
  • No semantic caching or advanced routing needed
  • Team has API Gateway expertise already

Cost: Near-zero beyond API Gateway charges. Lambda invocation adds $0.20/1M requests. Total overhead: ~$0.60–$1.20/1M requests (HTTP API + Lambda).

Limitations: No built-in rate limiting by team, no spend tracking, no semantic caching, no fallback routing.

Option B: LiteLLM Self-Hosted

Appropriate when:

  • Multiple model providers (Bedrock + OpenAI + Anthropic direct)
  • 5–50 teams requiring spend governance
  • DevOps capacity to manage ECS/Fargate + RDS + Redis
  • Budget constraint on SaaS gateway tooling

Cost: $470–$726/month infrastructure (excluding DevOps labor). TCO including labor: $2,000–$3,700/month.

Sweet spot: 10–50M requests/month across 5–20 teams.

Option C: Portkey Managed

Appropriate when:

  • Compliance requirements (SOC2, HIPAA) are non-negotiable
  • DevOps bandwidth is limited
  • Observability and RBAC are higher priority than cost minimization

Cost: $130–$2,000+/month depending on volume. Zero infrastructure management overhead.

Risk factor (2026): Palo Alto Networks acquisition introduces pricing and roadmap uncertainty. Evaluate contract terms carefully; avoid multi-year lock-in until integration direction is clearer.

Option C.5: Helicone

Helicone (not covered in depth above) occupies a similar space to Portkey: a managed observability + proxy layer that sits between your application and Bedrock. As of 2026, Helicone’s pricing is request-volume based ($0.000080/request above 10K free), making it cost-competitive with Portkey Production tier at low-to-mid volumes. At 1M requests/month, Helicone costs $80/month vs Portkey’s $130/month. Helicone’s differentiator is prompt experimentation and A/B testing tooling; its cost governance features (team budgets, rate limiting) are thinner than both LiteLLM and Portkey. Best fit: teams that prioritize prompt optimization and evaluation over spend governance.

Option D: AWS Multi-Provider Generative AI Gateway (Reference Architecture)

AWS’s published reference architecture (aws-solutions-library-samples) combines LiteLLM + ECS Fargate + Amazon SageMaker AI as a supported, documented pattern. This is the recommended starting point for AWS-native teams over building from scratch.

Appropriate when: AWS-only deployment, Bedrock as primary provider, need for AWS Support coverage on the architecture.

Decision Matrix Summary

Scenario Recommended Option
Single team, Bedrock only, <1M req/month Thin Lambda proxy (Option A)
5+ teams, cost attribution required, DevOps available LiteLLM self-hosted (Option B)
Regulated industry, compliance certs required Portkey managed (Option C)
Multi-provider, AWS-native, want AWS reference arch AWS Multi-Provider Gateway (Option D)
>50M req/month with high query repetition Option B or D + semantic caching layer

Cost Summary: Full Gateway Stack at 10M Requests/Month

Layer Conservative With Optimization
API Gateway (HTTP API) $10 $10
VPC Interface Endpoints (3 AZ) $21.90 $21.90 (required)
LiteLLM infrastructure (OSS) $137 $137
CloudWatch Logs (10% sampling) $2 $2
Cognito M2M auth $0–$10 $0–$10
Total gateway overhead $171–$181/month $171–$181/month
Per-request gateway overhead $0.017/1K requests $0.017/1K requests

Against Claude Sonnet 4.5 at ~$9/1K requests (500 in/500 out tokens), gateway overhead is 0.2% of total spend. The ROI case for the gateway stack rests entirely on what it enables: team-level spend caps that prevent budget overruns, semantic caching that reduces Bedrock costs by 40–70% for repetitive workloads, and cross-region routing that captures AWS’s GCRI discount automatically.

ROI summary for a 10M-requests/month deployment:

  • Bedrock spend without gateway optimization: ~$90,000/month (Sonnet, all requests)
  • Model tiering (30% savings): saves ~$27,000/month
  • Semantic caching at 40% hit rate (44% savings on uncached requests): saves ~$17,500/month
  • GCRI routing (10% savings): saves ~$4,500/month
  • Total potential Bedrock spend reduction: ~$49,000/month
  • Gateway infrastructure TCO: ~$500–$800/month
  • Net ROI: 60–100× on gateway infrastructure investment

The numbers are directional — actual hit rates and tiering splits vary by workload — but the order of magnitude is consistent with published enterprise deployments. The gateway pays for itself within the first week of operation at this scale.

Key Decision Thresholds — Quick Reference

Volume Gateway Recommendation Caching Auth
<1M req/month Lambda proxy + HTTP API None API Gateway keys
1–10M req/month LiteLLM OSS on ECS Exact match only Cognito M2M
10–50M req/month LiteLLM + semantic cache Semantic cache (r7g.large Redis) Cognito M2M + SCP
>50M req/month LiteLLM enterprise or Portkey managed Semantic cache + model tiering Cognito M2M + SCP + SIEM
Regulated industry Portkey managed (enterprise tier) Per compliance policy Cognito + MFA

VPC Endpoints: Deploy whenever NAT Gateway is already present (saves money). Mandatory for regulated workloads.

Semantic Caching: Only if query repetition rate >15%. Below that, the embedding cost and Redis overhead exceeds savings. Customer-facing chat with diverse queries: likely below threshold. Internal helpdesk, FAQ bots, code review: likely above threshold.

Cross-Region: Use GCRI geographic profiles by default. Only implement custom LiteLLM cross-region routing if you need to mix Bedrock with non-Bedrock providers or enforce specific fallback sequences.

Sources