← Agent Frameworks 🕐 22 min read
Agent Frameworks

LiteLLM Production Deployment Patterns — 2026

1. Docker / Docker Compose · March 2026

LiteLLM is an open-source proxy/SDK providing a unified OpenAI-compatible API across 100+ providers (Anthropic, OpenAI, Azure, Bedrock, Vertex, Cohere, etc.).

See also (wiki): ai-gateway · llm-cost-attribution · agentic-token-governance · enterprise-agent-runtime-infrastructure

LiteLLM is an open-source proxy/SDK providing a unified OpenAI-compatible API across 100+ providers (Anthropic, OpenAI, Azure, Bedrock, Vertex, Cohere, etc.). The proxy layer adds ~3.25ms overhead per call and benchmarks at 4,200+ requests per minute on a modest 4-core instance. This document covers production-grade deployment patterns, enterprise features, observability realities, and known failure modes.

The canonical starting point. Use the -database image variant when you need virtual keys, spend tracking, or the admin UI — it bundles PostgreSQL migrations.

docker pull docker.litellm.ai/berriai/litellm-database:main-stable

docker run \
  -v $(pwd)/litellm_config.yaml:/app/config.yaml \
  -e LITELLM_MASTER_KEY=sk-my-master-key \
  -e DATABASE_URL=postgresql://user:pass@postgres:5432/litellm \
  -e REDIS_URL=redis://redis:6379 \
  -e LITELLM_SALT_KEY=<random-32-char-string> \
  -p 4000:4000 \
  docker.litellm.ai/berriai/litellm-database:main-stable \
  --config /app/config.yaml

Key environment variables:

Variable Purpose
LITELLM_MASTER_KEY Admin credential; must start with sk-
DATABASE_URL PostgreSQL for keys, spend, teams
LITELLM_SALT_KEY Encrypts stored provider API keys at rest — set before adding any model
REDIS_URL Rate limiting and cross-instance router state
LITELLM_MODE=PRODUCTION Disables .env file loading
LITELLM_LOCAL_MODEL_COST_MAP=true Skip live pricing pull on startup (eliminates one cold-start network call)

Minimal litellm_config.yaml:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: azure/gpt-4o-prod
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
      rpm: 300

general_settings:
  database_connection_pool_limit: 10
  allow_requests_on_db_unavailable: true

router_settings:
  redis_host: os.environ/REDIS_HOST
  redis_port: 6379
  redis_password: os.environ/REDIS_PASSWORD
  routing_strategy: simple-shuffle   # avoid usage-based in prod — ~80 RPS slower

Health endpoints: GET /health/liveliness and GET /health/readiness on port 4000. Wire both into load balancer checks.

Horizontal scaling: Run multiple containers behind a load balancer. Each container needs REDIS_URL set — Redis is the shared state for rate limiting and router metrics. Without it, per-key rate limits are per-instance only.

2. Kubernetes / Helm

2.1 Official Helm Chart

LiteLLM ships an official Helm chart via OCI registry at docker.litellm.ai/berriai/litellm-helm. As of June 2026 the chart is at approximately v1.86–1.89 (mirrors the application version). It is not listed on ArtifactHub (open feature request: GitHub issue #12420).

helm pull oci://docker.litellm.ai/berriai/litellm-helm --version 1.86.2
helm install litellm ./litellm-helm -f values-prod.yaml

HA features the official chart supports:

Feature Default Notes
replicaCount 1 Must be raised manually
HPA (CPU) disabled autoscaling.enabled: true, targetCPUUtilizationPercentage: 80
HPA (memory) commented out Must uncomment and set threshold
KEDA autoscaling disabled Prometheus-based; mutually exclusive with HPA
PDB disabled Supports both minAvailable and maxUnavailable
Topology spread empty Example commented in values.yaml for multi-zone
terminationGracePeriodSeconds 90 Too low — must raise to 620+ (see below)
Redis disabled Optional caching; standalone architecture only
PostgreSQL Bitnami standalone No native HA postgres dependency
Resource limits none No defaults set; must specify explicitly

Critical chart gap — terminationGracePeriodSeconds: The official chart defaults to 90 seconds, but LiteLLM’s default request timeout is 600 seconds. Pods will be killed mid-request during rolling updates unless you override:

# values-prod.yaml
terminationGracePeriodSeconds: 620

HA production values overlay:

replicaCount: 3

autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 10
  targetCPUUtilizationPercentage: 60   # docs recommend 60% CPU, 80% memory

podDisruptionBudget:
  enabled: true
  minAvailable: 2

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: litellm

terminationGracePeriodSeconds: 620

resources:
  requests:
    cpu: "1"
    memory: "4Gi"
  limits:
    cpu: "2"
    memory: "4Gi"   # set limits == requests for predictable scheduling

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1

readinessProbe:
  httpGet:
    path: /health/readiness
    port: 4000
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 3

livenessProbe:
  httpGet:
    path: /health/liveliness
    port: 4000
  initialDelaySeconds: 120   # DB migrations run on startup
  periodSeconds: 15
  failureThreshold: 3

2.2 Community / Third-Party Helm Charts

RichardoC/litellm_helm-chart — the most active community fork.

  • ArtifactHub: litellm-helm/litellm (latest: v1.89.0 as of mid-2026; also published at oci://ghcr.io/richardoc/litellm_helm-chart/litellm)
  • Adds: progressive rollout support (Argo Rollouts-compatible), PDB enabled by default, draws patterns from Argo Helm and Bitnami
  • Install: helm repo add litellm-helm https://RichardoC.github.io/litellm_helm-chart/
  • Maturity: small single-maintainer project; tracks upstream application versions closely but lacks enterprise support

Unique-AG litellm chart (unique/litellm on ArtifactHub) — now deprecated. Do not use.

Official experimental microservices chart (docs.litellm.ai/docs/proxy/microservices_helm) — deploys four independent services with per-service HPA:

Service Min→Max Replicas HPA CPU Target Purpose
Gateway 1→10 70% LLM inference plane /chat/completions
Backend 1→4 70% Key/user/spend management API
UI 1→3 disabled Next.js admin dashboard
Migrations job — One-shot Prisma schema setup

This chart is experimental — values schema may change between releases; pin the version and review diffs before upgrading. Architectural benefit: a load spike on the inference plane does not affect dashboard performance, and dashboard query slowness does not consume inference threads.

2.3 HA Architecture: Redis

Redis is required for accurate cross-pod rate limiting. Without it, per-key RPM/TPM limits are enforced per-pod, not globally.

LiteLLM supports three Redis topologies:

1. Standalone (default — not HA):

router_settings:
  redis_host: redis.default.svc.cluster.local
  redis_port: 6379
  redis_password: os.environ/REDIS_PASSWORD
  # NEVER use redis_url in K8s — ~80 RPS slower and breaks connection pooling

2. Redis Sentinel (recommended HA mode for most deployments):

litellm_settings:
  cache: true
  cache_params:
    type: redis
    sentinel_nodes: '[["redis-sentinel-0.redis-sentinel.default.svc", 26379], ["redis-sentinel-1.redis-sentinel.default.svc", 26379], ["redis-sentinel-2.redis-sentinel.default.svc", 26379]]'
    service_name: "mymaster"

Or via env: REDIS_SENTINEL_NODES='[["host1", 26379], ["host2", 26379]]'

Deploy with Bitnami Redis chart in Sentinel mode:

helm install redis bitnami/redis \
  --set sentinel.enabled=true \
  --set sentinel.quorum=2 \
  --set replica.replicaCount=3

3. Redis Cluster (for >2,000 QPS): Use redis_startup_nodes in cache_params with multiple shard node addresses, or use ElastiCache (cluster mode enabled) / Upstash.

HA-critical Redis settings in LiteLLM config:

general_settings:
  use_redis_transaction_buffer: true   # required for 10+ instances; buffers DB writes via Redis queue
  use_shared_health_check: true        # cross-pod health state shared via Redis

litellm_settings:
  cache_params:
    max_connections: 100   # fix for GitHub issue #15794 — async client ignored this before v1.77.7
    enable_redis_auth_cache: true   # caches virtual-key auth in Redis; reduces DB round trips

Redis circuit breaker (default-on since v1.82.0): Trips after 5 failures, fast-fails at 0ms for 60 seconds, then probes for recovery. Prevents the catastrophic pattern of 100+ pods each hanging 30 seconds on auth checks simultaneously.

REDIS_CIRCUIT_BREAKER_FAILURE_THRESHOLD=5
REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT=60

2.4 HA Architecture: PostgreSQL

Connection budget formula:

total_connections = replicas × workers_per_pod × database_connection_pool_limit

Example: 5 pods × 1 worker × 10 pool limit = 50 connections. Standard RDS db.t3.medium allows ~170. Plan before scaling.

Recommended options by scale:

Scale PostgreSQL Option Notes
Dev/staging Bitnami PostgreSQL chart (single) Simple; no HA
Production ≤500 RPS CloudNativePG (CNCF sandbox, May 2025) Declarative, streaming replication, built-in PgBouncer pooler
Production >500 RPS or AWS RDS Aurora Multi-AZ + PgBouncer sidecar IAM auth supported natively in LiteLLM Helm chart
Enterprise/regulated CrunchyData PGO PgBouncer native, pgBackRest, patroni failover

PgBouncer (transaction mode) is required when running 5+ pods. Wire it between LiteLLM and PostgreSQL:

  • LiteLLM sets database_connection_pool_limit to the PgBouncer pool size, not the raw Postgres limit
  • Set database_connection_pool_timeout to detect pool exhaustion quickly rather than hanging
  • CloudNativePG and CrunchyData PGO both include integrated PgBouncer configuration

High-write-rate pattern (use_redis_transaction_buffer: true): Instead of every pod writing spend/rate-limit updates directly to PostgreSQL, all pods write to a Redis queue. A single elected pod (determined by Redis lock) drains the queue to Postgres in batches. Eliminates the deadlock pattern that occurs when 10+ pods simultaneously UPDATE the same user_id or team_id row.

DB connection pool settings in config:

general_settings:
  database_connection_pool_limit: 10       # per worker; default 10
  database_connection_pool_timeout: 30     # seconds before pool exhaustion error
  proxy_batch_write_at: 60                 # batch spend updates every 60s to reduce write pressure
  allow_requests_on_db_unavailable: true   # only safe in private VPC

2.5 Known Failure Modes (K8s-specific)

DB migration race condition on multi-pod startup (GitHub issue #16190, postmortem #22807):

  • Root cause: the Helm chart migration job launches alongside the application pods without waiting for PostgreSQL readiness. When the job fails (DB not yet up), Kubernetes marks it “Completed” despite errors, so the tables are never created.
  • Symptom from issue #22807: pod starts, /health returns 200, but required tables are absent — no indication of failure.
  • Fix: add an initContainer running prisma db push (or prisma migrate deploy) before the main container starts:
initContainers:
  - name: db-migrate
    image: docker.litellm.ai/berriai/litellm-database:main-stable
    command: ["sh", "-c", "prisma db push --schema /app/schema.prisma"]
    env:
      - name: DATABASE_URL
        valueFrom:
          secretKeyRef:
            name: litellm-secrets
            key: DATABASE_URL

Issue #16190 is closed as “not planned” — the official chart has not fixed this. Use the initContainer workaround.

Redis async client ignores max_connections (GitHub issue #15794, fixed in v1.77.7-stable via PR #15797):

  • Root cause: max_connections in cache_params was added to kwargs but never passed to the async Redis client initialization, leaving connections unmanaged.
  • Symptom: “ERR max number of clients reached” under load; unbounded connection growth.
  • Fix: upgrade to v1.77.7-stable or later; explicitly set max_connections: 100 in cache_params.

False readiness on separate health app (GitHub issue #18415):

  • Root cause: the health endpoint becomes ready in ~1 second while the main application needs ~10 seconds to fully initialize, and when Redis is down or the cache is unavailable the readiness probe can return healthy incorrectly.
  • Fix: increase initialDelaySeconds on readiness probe to 30+, add Redis availability check if using shared health state.

readOnlyRootFilesystem permission error on migrations (GitHub issue #19859):

  • Root cause: prisma generate writes files to the container filesystem; fails when readOnlyRootFilesystem: true.
  • Fix: add emptyDir volumes mounted at the Prisma output path and the UI assets directory.

Budget leak / phantom BudgetExceededError (GitHub issue #27639, affects v1.84.0):

  • Root cause: reserve_budget_for_request() leaks Redis spend counters when using use_redis_transaction_buffer. Phantom budget exceeded errors appear despite actual spend being below limit.
  • Status: reported June 2026; workaround is to restart pods to clear Redis counters.

OpenShift / HTTPS proxy migration failure (GitHub issue #7173):

  • The migration Job pod cannot run on OpenShift (security context constraints) or behind an HTTPS proxy.
  • Fix: use the initContainer pattern above instead of the separate migration job.

2.6 Secret Management

LiteLLM Helm chart values support three patterns in order of security preference:

1. External Secrets Operator (recommended for enterprise):

# ExternalSecret CR pointing to AWS Secrets Manager / Vault
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: litellm-secrets
spec:
  secretStoreRef:
    name: aws-secretsmanager
    kind: ClusterSecretStore
  target:
    name: litellm-secrets
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: /litellm/prod/database-url
    - secretKey: LITELLM_MASTER_KEY
      remoteRef:
        key: /litellm/prod/master-key
    - secretKey: LITELLM_SALT_KEY
      remoteRef:
        key: /litellm/prod/salt-key

2. Vault Secrets Operator: Syncs Vault secrets to K8s secrets natively; preferred for existing Vault deployments.

3. Kubernetes Secrets (minimum viable): Mount as envFrom. Never store in litellm_config.yaml or values.yaml — reference via os.environ/VAR_NAME syntax in the config file.

Critical secret hygiene notes:

  • LITELLM_SALT_KEY must be set before any model credentials are stored. Changing it after the fact requires re-encrypting all stored provider keys.
  • Master key bypasses all budget/rate limits — treat as a root credential, not an application key.
  • Issue virtual keys per team/service with scoped model access and budget caps. Never share the master key with application code.

2.7 HA Control Plane (Beta / Enterprise)

The HA Control Plane is a separate enterprise feature (not the same as multi-replica deployment). It deploys a single admin UI managing multiple fully independent LiteLLM worker instances, each with its own database, Redis, and master key.

# Control plane config
general_settings:
  worker_registry:
    - worker_id: "worker-us-east"
      name: "US East Worker"
      url: "http://litellm-us-east:4000"
    - worker_id: "worker-eu-west"
      name: "EU West Worker"
      url: "http://litellm-eu-west:4000"

Use case: regional data residency, blast-radius isolation across business units, or active-active multi-region deployments where a shared PostgreSQL/Redis would be a single point of failure.

# values-ha-prod.yaml — official chart
replicaCount: 3

autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 10
  targetCPUUtilizationPercentage: 60

podDisruptionBudget:
  enabled: true
  minAvailable: 2

terminationGracePeriodSeconds: 620

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1

resources:
  requests:
    cpu: "1"
    memory: "4Gi"
  limits:
    cpu: "2"
    memory: "4Gi"

securityContext:
  readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 101

extraVolumes:
  - name: tmp
    emptyDir: {}
  - name: prisma-cache
    emptyDir: {}

extraVolumeMounts:
  - name: tmp
    mountPath: /tmp
  - name: prisma-cache
    mountPath: /app/.prisma

initContainers:
  - name: db-migrate
    image: docker.litellm.ai/berriai/litellm-database:main-stable
    command: ["sh", "-c", "prisma migrate deploy --schema /app/schema.prisma"]
    envFrom:
      - secretRef:
          name: litellm-secrets

env:
  - name: LITELLM_MODE
    value: "PRODUCTION"
  - name: LITELLM_LOCAL_MODEL_COST_MAP
    value: "true"
  - name: REDIS_CIRCUIT_BREAKER_FAILURE_THRESHOLD
    value: "5"
  - name: REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT
    value: "60"

# proxy_config.yaml content (passed as configmap):
# general_settings:
#   database_connection_pool_limit: 10
#   database_connection_pool_timeout: 30
#   proxy_batch_write_at: 60
#   use_redis_transaction_buffer: true
#   use_shared_health_check: true
#   allow_requests_on_db_unavailable: true
#
# router_settings:
#   redis_host: redis-sentinel.default.svc.cluster.local
#   redis_port: 6379
#   redis_password: os.environ/REDIS_PASSWORD
#   routing_strategy: simple-shuffle
#
# litellm_settings:
#   cache: true
#   cache_params:
#     type: redis
#     sentinel_nodes: '[["redis-s-0:26379","redis-s-1:26379","redis-s-2:26379"]]'
#     service_name: "mymaster"
#     max_connections: 100
#   set_verbose: false
#   json_logs: true

Chart maturity assessment:

  • Official chart (docker.litellm.ai/berriai/litellm-helm): ships with every LiteLLM release, actively maintained, but has significant HA gaps out-of-the-box (terminationGracePeriodSeconds too low, no PDB, no initContainer for migrations, no resource defaults). Production use requires substantial values override.
  • RichardoC community chart: better production defaults (PDB on by default, progressive rollouts), but single-maintainer and lagging on the experimental microservices architecture.
  • Experimental microservices chart: best isolation model for high-traffic production, but schema instability between releases makes it risky to adopt without version pinning.

Recommended chart selection:

  • Teams with <500 RPS starting out: official chart with the HA values overlay above.
  • Teams running >500 RPS or needing inference/management plane isolation: experimental microservices chart, pinned version.
  • Teams on OpenShift or with progressive rollout requirements: RichardoC community chart.

3. Serverless / Managed

AWS ECS Fargate: No cold starts (tasks remain running); scaling new tasks takes 30–60 seconds. Official Terraform reference at github.com/BerriAI/litellm-ecs-deployment. Fargate is the better choice when consistent latency matters more than cost-at-zero-traffic.

GCP Cloud Run: Scales to zero, but cold starts add 8–15 seconds when a new instance initializes — the proxy must connect to PostgreSQL, run any pending migrations, and optionally pull model pricing. Mitigate with --min-instances=1. Appropriate for dev/staging or bursty batch workloads where cold starts are acceptable.

Railway / Render: Viable for smaller teams. Both support persistent containers (not function-as-a-service), so there are no cold start penalties. PostgreSQL and Redis add-ons available on both platforms. Not suitable for >500 RPS — managed Redis connection limits become the ceiling.

Cold start note: Disable live pricing pulls (LITELLM_LOCAL_MODEL_COST_MAP=true) in any environment where cold start time matters. The default behavior fetches a remote pricing JSON file on startup.

4. Enterprise Features

LiteLLM OSS covers the core gateway. Enterprise tier (commercial license) adds SSO (Okta, Azure AD, Google Workspace, OIDC/SAML), JWT auth, and role-based access controls.

Virtual keys: Generated at /key/generate, scoped to specific models, with hard budget caps:

curl http://localhost:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -d '{
    "models": ["gpt-4o", "claude-3-5-sonnet"],
    "max_budget": 50.00,
    "budget_duration": "30d",
    "rpm_limit": 100,
    "tpm_limit": 500000
  }'

Spend tracking: Automatic via completion_cost() on every response. Tracked at key / user / team level, queryable via /key/info, /user/info, /team/info. All spend aggregates to the LiteLLM_VerificationTokenTable in PostgreSQL.

Budget controls: max_budget (total cap), budget_duration (rolling window), temp_budget_increase (temporary overrides with expiry). Budget exhaustion returns a hard 429.

Rate limiting: rpm_limit, tpm_limit, and max_parallel_requests per key. Redis is required for accurate cross-instance enforcement — without it, limits are per-pod only.

Model fallbacks:

model_list:
  - model_name: primary-gpt4
    litellm_params:
      model: azure/gpt-4o-eastus
  - model_name: fallback-gpt4
    litellm_params:
      model: openai/gpt-4o

router_settings:
  fallbacks: [{"primary-gpt4": ["fallback-gpt4"]}]
  num_retries: 3
  retry_after: 5

Load balancing: The routing_strategy options are simple-shuffle (recommended for production), least-busy, latency-based-routing (38% lower p95 vs. round-robin in multi-region deployments, but ~50ms overhead on cold proxy startup and ~5% extra memory for per-endpoint metrics).

Config reload: LiteLLM does not support zero-downtime config reload. Rolling restarts via Kubernetes RollingUpdate strategy (with maxUnavailable: 0) are the standard approach.

5. Observability Integration

Prometheus: /metrics endpoint exposed by default. Covers request count, latency histograms, error rates, and token usage per model.

OpenTelemetry: Set via environment:

OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4317
OTEL_SERVICE_NAME=litellm-proxy

What IS captured in OTEL traces:

  • gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.total_tokens
  • gen_ai.client.response.time_to_first_token (streaming only)
  • gen_ai.client.response.duration (vendor latency, excludes LiteLLM overhead)
  • gen_ai.client.operation.duration (end-to-end including proxy overhead)
  • gen_ai.cost.total_cost, gen_ai.cost.input_cost, gen_ai.cost.output_cost
  • Model name, provider, operation type, team/org/key metadata
  • Prompt and completion content via gen_ai.input.messages / gen_ai.output.messages — only when OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_AND_EVENT

What is NOT captured by default: Response content is suppressed unless CAPTURE_MESSAGE_CONTENT is explicitly set. Raw provider request/response body attributes (llm.{provider}.*) are suppressed in semconv mode. Per-turn token usage for multi-turn agent trajectories lives only in AgentResult.trajectory and must be manually correlated with proxy logs by timestamp.

Callback integrations: Langfuse, Helicone, Arize, MLflow, PromptLayer, and Splunk are supported as pluggable callbacks. Configuration in litellm_config.yaml:

litellm_settings:
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]

6. Security

Master key vs. virtual keys: The master key is a superadmin credential — it bypasses all budget and rate limits. Never expose it to application code. Issue virtual keys with scoped model access and budget caps for every consumer.

Network isolation: Run the proxy in a private VPC with no public endpoint. Expose only via an internal load balancer or service mesh. The only inbound port needed is 4000.

Secret rotation: Use LITELLM_KEY_ROTATION_ENABLED=true with a grace_period (e.g., "24h") so old and new keys overlap during rotation. Set LITELLM_SALT_KEY once at initialization — changing it after models are stored requires re-encrypting all stored credentials.

Audit logging: Enable json_logs: true and disable verbose mode (set_verbose: False) in production. Log ingestion (Splunk, Datadog, CloudWatch) should consume stdout JSON for audit trails.

Container hardening: readOnlyRootFilesystem: true is fully supported. Add emptyDir volumes for /tmp and the UI assets directory.

7. Performance Benchmarks

Metric Value
Proxy overhead (p50) ~3.25ms vs. direct API call
Throughput (4-core instance) 4,200+ RPM
Latency-based routing p95 improvement 38% lower vs. round-robin
Redis redis_url vs. separate params ~80 RPS slower with redis_url
Recommended minimum spec (production) 4 CPU, 8 GB RAM
High-traffic spec (1,000+ RPS) Redis required; managed cluster (ElastiCache/Upstash) beyond 2,000 QPS

8. Known Production Failure Modes

K8s-specific failure modes with GitHub issue references are detailed in §2.5. Summary of all failure modes:

Database connection exhaustion: The most common production incident. Formula: replicas × workers × database_connection_pool_limit. Standard RDS db.t3.medium max is ~170. Deploy PgBouncer in transaction mode for 5+ pods. Set database_connection_pool_timeout: 30 to surface exhaustion quickly rather than hanging silently.

Redis connection pool exhaustion (GitHub issue #15794, fixed v1.77.7): Triggers “ERR max number of clients reached” under ~300 concurrent requests with default config because the async client ignored max_connections. Set max_connections: 100 in cache_params and upgrade to v1.77.7+. Move to Redis Sentinel or ElastiCache cluster mode beyond 2,000 QPS.

Slow Redis more dangerous than dead Redis: 100+ pods hanging 30 seconds on auth checks fill threadpools silently, cascading into PostgreSQL. Circuit breaker (default-on since v1.82.0) resolves this — tune REDIS_CIRCUIT_BREAKER_FAILURE_THRESHOLD=5 and REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT=60.

DB migration race condition on pod startup (GitHub issue #16190, #22807): Migration job marks “Completed” even when Postgres wasn’t ready; missing tables are invisible at startup because /health still returns 200. Use the initContainer workaround in §2.5 — the official chart has not fixed this (issue closed as “not planned”).

Phantom budget exceeded errors (GitHub issue #27639, v1.84.0): Redis spend counters leak when using use_redis_transaction_buffer. Pods report budget exceeded despite actual spend being below limit. Workaround: rolling pod restart clears counters.

Config reload requires restart: No hot reload. Rolling restarts are necessary for model list or routing changes. Use maxUnavailable: 0 and maxSurge: 1 with terminationGracePeriodSeconds: 620+.

Worker memory growth: Under sustained load, Uvicorn workers accumulate memory. Set MAX_REQUESTS_BEFORE_RESTART to recycle workers after a fixed request count (e.g., 10,000) before RSS grows unbounded.

Decision Matrix

Deployment Best For Hard Limit
Docker Compose Single-team internal gateway, dev/staging No HA; single failure domain
Kubernetes + Helm Multi-team enterprise, >100 RPS Operational complexity
AWS ECS Fargate AWS-native shops, latency-sensitive 30–60s scale-out lag
GCP Cloud Run Bursty/batch, cost optimization 8–15s cold start without min-instances
Railway / Render Startups, <500 RPS Managed Redis connection ceiling

For Fortune 500 deployments: Kubernetes with the official Helm chart, PostgreSQL with PgBouncer, managed Redis (ElastiCache/Upstash), virtual keys per team, and OTEL to your existing observability stack. Enable OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_AND_EVENT only if your data governance policy permits storing prompt/completion content in traces.

9. AWS Bedrock Cost Attribution via Inference Profiles

9.1 The Three Profile Types

AWS Bedrock has two conceptually distinct profile types, plus a third “global” variant:

Profile Type Who Creates It Purpose ARN Prefix
System inference profile (cross-region) AWS (predefined) Automatic geographic load balancing across regions in a named zone (US, EU, AP, JP) arn:aws:bedrock:<region>::inference-profile/<zone>.<model-id> — note double-colon, no account ID
Application inference profile (AIP) You (per account) Cost attribution via AWS cost allocation tags; wraps any foundation model or system profile arn:aws:bedrock:<region>:<account-id>:application-inference-profile/<id>
Global inference profile AWS (predefined) Routes to optimal commercial region globally; newer variant of cross-region arn:aws:bedrock:<region>:<account-id>:inference-profile/global.<model-id>

Key distinction for enterprises: System/global inference profiles solve throughput and latency (request goes to least-loaded region). Application inference profiles solve billing (request shows up in Cost Explorer under your tag). You can nest them: create an AIP whose source model is a system inference profile to get both geographic load balancing and cost attribution.

For LiteLLM routing to Bedrock, the ARN you pass in config is always the application inference profile ARN (application-inference-profile/...). Do not pass the system profile ARN directly if you need cost tags — system profiles have no tagging mechanism.

9.2 ARN Formats

# System (cross-region) inference profile — AWS managed, no account ID
arn:aws:bedrock:us-east-1::inference-profile/us.anthropic.claude-3-5-sonnet-20241022-v2:0

# Application inference profile — customer-created, has account ID
arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/a1b2c3d4e5f6

# Global inference profile — AWS managed, routes to best worldwide region
arn:aws:bedrock:us-east-2:111122223333:inference-profile/global.anthropic.claude-sonnet-4-20250514-v1:0

Model request ID (not ARN): The application-inference-profile/<id> suffix is an opaque alphanumeric identifier returned in the inferenceProfileArn field when you create the profile. It is not the model ID or a human-readable name.

9.3 Creating Application Inference Profiles with Cost Allocation Tags

# Step 1: Create the profile, wrapping a foundation model or system profile
aws bedrock create-inference-profile \
  --inference-profile-name "fraud-detection-prod" \
  --description "Fraud detection team production workload" \
  --model-source 'copyFrom=arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0' \
  --tags '[
    {"key":"Project","value":"fraud-detection"},
    {"key":"Team","value":"data-science"},
    {"key":"CostCenter","value":"CC-8821"},
    {"key":"Environment","value":"production"}
  ]' \
  --region us-east-1

# Output:
# {
#   "inferenceProfileArn": "arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/a1b2c3d4e5f6",
#   "status": "ACTIVE"
# }

# Step 2: Add additional tags post-creation if needed
aws bedrock tag-resource \
  --resource-arn arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/a1b2c3d4e5f6 \
  --tags key=Owner,value=alice@example.com

# Step 3: Verify
aws bedrock list-inference-profiles --type-equals APPLICATION --region us-east-1

# Step 4: Activate tags as cost allocation tags in Billing console
# (Manual step — AWS Console → Billing → Cost allocation tags → activate)
# Tags take up to 24 hours to appear in Cost Explorer after activation.
# Cost allocation tags are NOT retroactive — only future costs are tagged.

To wrap a cross-region system profile instead (get both load balancing + cost attribution):

aws bedrock create-inference-profile \
  --inference-profile-name "fraud-detection-multi-region" \
  --model-source 'copyFrom=arn:aws:bedrock:us-east-1::inference-profile/us.anthropic.claude-3-5-sonnet-20241022-v2:0' \
  --tags '[{"key":"Team","value":"data-science"}]' \
  --region us-east-1

Profile proliferation caveat: Each AIP is model-specific. One AIP per model per tag combination. For 10 teams × 5 models = 50 profiles minimum. AWS recommends tagging at the team or cost-center level (not per-user) to limit proliferation. For per-user attribution without profiles, use IAM principal tracking (separate feature in CUR 2.0 that attributes by IAM role/user automatically).

9.4 LiteLLM Configuration for Bedrock Inference Profiles

Working config as of LiteLLM v1.79+:

model_list:
  # Application Inference Profile via model_id parameter (RECOMMENDED — works reliably)
  - model_name: claude-fraud-detection
    litellm_params:
      model: bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0  # base model name tells LiteLLM the provider
      model_id: arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/a1b2c3d4e5f6
      aws_region_name: us-east-1

  # Alternative: explicit converse route with ARN in model string
  - model_name: claude-fraud-detection-alt
    litellm_params:
      model: bedrock/converse/arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/a1b2c3d4e5f6
      aws_region_name: us-east-1

  # Cross-region system inference profile (load balancing, no cost tags)
  - model_name: claude-us-distributed
    litellm_params:
      model: bedrock/us.anthropic.claude-3-5-sonnet-20241022-v2:0
      aws_region_name: us-east-1

Do NOT use this (broken as of GitHub issue #18258, filed Dec 2025, not yet fixed):

# BROKEN — fails with "Unknown provider=None"
model: bedrock/arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/a1b2c3d4e5f6

Root cause (GitHub issue #18258): LiteLLM’s parser extracts the provider (Anthropic, Amazon, etc.) from the model string. Application inference profile ARNs contain no provider information, so provider=None and the request fails. The model_id workaround routes the call to the AIP while the base model string provides provider identification.

GitHub issue #8911 (older report, also unfixed as of mid-2026) — same class of bug for image generation models with AIPs.

Python SDK equivalent:

from litellm import completion

response = completion(
    model="bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0",
    model_id="arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/a1b2c3d4e5f6",
    messages=[{"role": "user", "content": "Analyze this transaction for fraud."}],
    aws_region_name="us-east-1"
)

Multi-team setup pattern: Create one LiteLLM virtual key per team, one AIP per team per model. Map each virtual key to the model alias that points to the team’s AIP:

model_list:
  - model_name: claude-team-fraud
    litellm_params:
      model: bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
      model_id: arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/fraud-team-profile-id
  - model_name: claude-team-riskops
    litellm_params:
      model: bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
      model_id: arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/riskops-team-profile-id

Then issue virtual keys scoped to models: ["claude-team-fraud"] for the fraud team, models: ["claude-team-riskops"] for risk ops. AWS billing shows per-team spend via AIP tags; LiteLLM virtual key tracking provides the spend view within the proxy UI.

9.5 IAM Policy for Application Inference Profiles

The LiteLLM pod’s IAM role (or the credentials it uses) needs these permissions. The official AWS prerequisite policy (from docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles-prereq.html):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "InvokeViaProfile",
      "Effect": "Allow",
      "Action": [
        "bedrock:InvokeModel",
        "bedrock:InvokeModelWithResponseStream",
        "bedrock:CreateInferenceProfile"
      ],
      "Resource": [
        "arn:aws:bedrock:*::foundation-model/*",
        "arn:aws:bedrock:*:*:inference-profile/*",
        "arn:aws:bedrock:*:*:application-inference-profile/*"
      ]
    },
    {
      "Sid": "ManageProfiles",
      "Effect": "Allow",
      "Action": [
        "bedrock:GetInferenceProfile",
        "bedrock:ListInferenceProfiles",
        "bedrock:DeleteInferenceProfile",
        "bedrock:TagResource",
        "bedrock:UntagResource",
        "bedrock:ListTagsForResource"
      ],
      "Resource": [
        "arn:aws:bedrock:*:*:inference-profile/*",
        "arn:aws:bedrock:*:*:application-inference-profile/*"
      ]
    }
  ]
}

Least-privilege hardening for LiteLLM runtime (inference only, no profile creation/deletion from the app pod):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "InvokeViaNamedProfiles",
      "Effect": "Allow",
      "Action": [
        "bedrock:InvokeModel",
        "bedrock:InvokeModelWithResponseStream"
      ],
      "Resource": [
        "arn:aws:bedrock:us-east-1::foundation-model/*",
        "arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/*"
      ]
    },
    {
      "Sid": "DenyDirectModelAccess",
      "Effect": "Deny",
      "Action": ["bedrock:InvokeModel*"],
      "Resource": "arn:aws:bedrock:*::foundation-model/*",
      "Condition": {
        "Null": {
          "bedrock:InferenceProfileArn": "true"
        }
      }
    }
  ]
}

The Deny statement with bedrock:InferenceProfileArn null condition forces all model invocations through an inference profile — direct model calls without a profile ARN are blocked. This is useful when you want to guarantee every Bedrock call is tagged for cost attribution.

Profile creation should be in a separate admin role, not the LiteLLM runtime role. Create AIPs via CI/CD or Terraform, not at application runtime.

Terraform resource (if managing profiles as IaC):

resource "aws_bedrock_inference_profile" "fraud_detection" {
  name        = "fraud-detection-prod"
  description = "Fraud detection team Claude 3.5 Sonnet"
  model_source {
    copy_from = "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0"
  }
  tags = {
    Project     = "fraud-detection"
    Team        = "data-science"
    CostCenter  = "CC-8821"
    Environment = "production"
  }
}

9.6 Cost Attribution Flow to Cost Explorer

  1. Create AIP with tags → tags are on the AIP resource
  2. Activate those tag keys as cost allocation tags in AWS Billing console (one-time, per tag key)
  3. 24-hour propagation delay before tags appear in Cost Explorer
  4. In Cost Explorer: Group by tag (e.g., Project) → see aggregated daily USD spend per project
  5. In CUR 2.0: query resource_tags/user_Project column for line-item detail

Granularity ceiling: AIPs deliver aggregated billed dollars per usage type per day. They do not produce per-request cost. For per-prompt token counts, use LiteLLM’s own LiteLLM_SpendLogs table or enable Bedrock model invocation logging to S3 + Athena.

10. Day-2 Operations

10.1 Secret Rotation Without Downtime

Rotating LiteLLM virtual keys (automated):

# Enable in env
LITELLM_KEY_ROTATION_ENABLED=true
LITELLM_KEY_ROTATION_CHECK_INTERVAL_SECONDS=86400  # default: 24h

# Per-key grace period — old and new key both valid during overlap
curl http://localhost:4000/key/regenerate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -d '{"key": "sk-old-key", "grace_period": "24h"}'

Rotating the master key (requires brief coordination):

  1. Back up PostgreSQL (pg_dump).
  2. POST to /key/regenerate with current master key and the new master key value — this re-encrypts all model credentials in Proxy_ModelTable.
  3. Update LITELLM_MASTER_KEY environment variable (Kubernetes Secret or AWS Secrets Manager).
  4. Rolling restart pods to pick up the new env var.
  5. Validate with a test request using the new key.

CRITICAL: LITELLM_SALT_KEY must never be rotated unless you are prepared to re-encrypt every stored provider credential. It is the encryption key for provider API keys in Proxy_ModelTable. If you must rotate it, decrypt all stored keys with the old salt first, update the salt, then re-encrypt.

Rotating the PostgreSQL password (DB credential rotation):

  1. Add the new password to AWS Secrets Manager (or Vault) alongside the old password.
  2. Update the External Secrets Operator ExternalSecret to point to the new secret version.
  3. Wait for ESO to sync the Kubernetes secret (typically <1 minute with refreshInterval: 1m).
  4. Rolling restart LiteLLM pods — each new pod picks up the refreshed DATABASE_URL.
  5. After all pods cycle and are healthy, revoke the old DB password.

The old pods continue running with the old password during the rolling update; new pods come up with the new password. This gives zero-downtime rotation if maxUnavailable: 0 is set.

Rotating Redis password: Same pattern as PostgreSQL — update the Kubernetes Secret, rolling restart pods. Redis Sentinel handles the password refresh transparently if you use Bitnami’s REDIS_MASTER_PASSWORD_FILE mechanism.

10.2 Upgrade Strategy

Version cadence: LiteLLM ships new releases several times per week. The main-stable tag (format: main-v<VERSION>-stable) is released weekly with a curated set of fixes. For production, pin to stable.

# In Helm values — pin the image tag explicitly
image:
  tag: "main-v1.89.0-stable"

Never use latest or main in production — LiteLLM’s fast release cadence means main can include breaking changes within hours.

Upgrade sequence:

  1. Review the LiteLLM Release Notes for DB migration notes, breaking config changes, or known K8s issues.
  2. Back up PostgreSQL: pg_dump -h <host> -U <user> -d litellm -F c -f litellm_backup_$(date +%Y%m%d_%H%M%S).dump
  3. Upgrade Helm chart: helm upgrade litellm ./litellm-helm --version <new-version> -f values-prod.yaml
  4. The initContainer migration (prisma migrate deploy) runs automatically before the main container starts.
  5. Verify /health returns 200; check LiteLLM_SpendLogs is still populating.

Known upgrade hazard — September 6, 2025 release: Caused Out of Memory errors on Kubernetes. Pattern: fast-moving releases occasionally introduce memory regressions. Always stage upgrades in a non-production environment first.

Rollback procedure:

  1. helm rollback litellm [revision-number] — reverts Helm chart to previous revision.
  2. If DB migration applied new tables/columns that the old version doesn’t understand: connect to PostgreSQL and DELETE rows from _prisma_migrations that belong to the version being rolled back. LiteLLM will re-apply them when you re-upgrade.
  3. LITELLM_SALT_KEY must remain unchanged — do not swap it during rollback.
  4. After rollback: validate with GET /health, GET /health/readiness, and a test inference call.

10.3 Alerting Thresholds

Prometheus / AlertManager rules (recommended production baselines):

groups:
  - name: litellm
    rules:
      # P95 latency — proxy overhead should be <50ms; >500ms indicates upstream provider issue
      - alert: LiteLLMHighP95Latency
        expr: histogram_quantile(0.95, rate(litellm_request_duration_seconds_bucket[5m])) > 0.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "LiteLLM p95 latency exceeds 500ms"

      # Error rate — normal is <1%; >5% indicates provider degradation or config issue
      - alert: LiteLLMHighErrorRate
        expr: rate(litellm_requests_total{status="error"}[5m]) / rate(litellm_requests_total[5m]) > 0.05
        for: 5m
        labels:
          severity: critical

      # Budget exhaustion — fire before teams hit their cap, not after
      - alert: LiteLLMBudgetNearExhaustion
        expr: litellm_team_budget_used_dollars / litellm_team_budget_limit_dollars > 0.85
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Team {{ $labels.team_id }} at 85% budget"

      # Hanging requests (LiteLLM Enterprise PagerDuty integration)
      # Configure in litellm_config.yaml:
      # alerting_args:
      #   failure_threshold: 10
      #   failure_threshold_window_seconds: 60
      #   hanging_threshold_seconds: 30
      #   hanging_threshold_window_seconds: 60

      # DB connection pool exhaustion
      - alert: LiteLLMDBPoolExhaustion
        expr: litellm_db_connection_pool_size - litellm_db_connection_pool_available < 2
        for: 2m
        labels:
          severity: critical

      # Redis circuit breaker tripped
      - alert: LiteLLMRedisCircuitBreakerOpen
        expr: litellm_redis_circuit_breaker_state == 1
        for: 1m
        labels:
          severity: warning

Datadog / Grafana: LiteLLM’s official Grafana dashboard (ID 24055) covers request count, latency percentiles, error rates, and cache efficiency. Datadog has a native LiteLLM integration with APM tracing.

PagerDuty alerting (Enterprise only):

general_settings:
  alerting: ["pagerduty"]
  alerting_args:
    failure_threshold: 10
    failure_threshold_window_seconds: 60
    hanging_threshold_seconds: 30
    hanging_threshold_window_seconds: 60

Requires PAGERDUTY_API_KEY env var. Not available in OSS; 7-day Enterprise trial available.

10.4 Log Aggregation

What LiteLLM writes where:

Data Destination Notes
Request/response HTTP logs stdout (JSON) Set json_logs: true; pipe to CloudWatch / Datadog / Splunk
Spend logs (token usage, cost, timing) PostgreSQL LiteLLM_SpendLogs + stdout Disable with disable_spend_logs: True if DB write pressure is high
Error logs stdout + PostgreSQL Disable DB write with disable_error_logs: True in production
Audit log (config changes) PostgreSQL LiteLLM_AuditLog Disabled by default; enable if compliance requires it
Callback data (Langfuse, Helicone, etc.) External callback endpoint Configured via success_callback/failure_callback

Recommended production logging config:

litellm_settings:
  set_verbose: false          # suppress debug logs
  json_logs: true             # structured JSON to stdout
  disable_error_logs: true    # keep DB writes lean; errors go to stdout only

general_settings:
  disable_spend_logs: false   # keep spend logs in DB for UI; disable if >10K RPS

Log shipping: Use Fluent Bit / Fluentd as a DaemonSet sidecar to ship stdout JSON to CloudWatch Logs, Datadog, or Splunk. LiteLLM’s stdout format with json_logs: true is compatible with CloudWatch Insights queries without further parsing.

10.5 PostgreSQL Backup and Restore

What to back up:

Table Criticality Notes
LiteLLM_VerificationToken CRITICAL Virtual keys, budgets, model access
LiteLLM_TeamTable CRITICAL Team budgets and rate limits
LiteLLM_UserTable CRITICAL User accounts
LiteLLM_BudgetTable CRITICAL Budget configurations
Proxy_ModelTable CRITICAL Encrypted provider API keys
LiteLLM_SpendLogs HIGH Historical spend; large; can be rebuilt from OTEL if needed
LiteLLM_AuditLog MEDIUM Compliance; disabled by default

Backup commands:

# Full backup
pg_dump -h $DB_HOST -U $DB_USER -d litellm \
  -F c -f litellm_backup_$(date +%Y%m%d_%H%M%S).dump

# Critical tables only (smaller, faster)
pg_dump -h $DB_HOST -U $DB_USER -d litellm \
  -F c \
  -t '"LiteLLM_VerificationToken"' \
  -t '"LiteLLM_TeamTable"' \
  -t '"LiteLLM_UserTable"' \
  -t '"LiteLLM_BudgetTable"' \
  -t '"Proxy_ModelTable"' \
  -f litellm_critical_$(date +%Y%m%d).dump

# Restore
pg_restore -h $DB_HOST -U $DB_USER -d litellm --clean litellm_backup_20260616_120000.dump

For RDS: Use automated snapshots (RPO of 5 minutes with PITR). Supplement with a daily pg_dump to S3 for cross-region durability.

Spend logs size management: LiteLLM_SpendLogs grows at roughly 1–2 KB per request. At 1,000 RPS this is ~86 GB/day. Implement a partition or archival job:

-- Archive logs older than 90 days to separate table or drop
DELETE FROM "LiteLLM_SpendLogs"
WHERE "startTime" < NOW() - INTERVAL '90 days';

10.6 Multi-Region: Active-Active vs. Active-Passive

Active-passive (recommended starting point):

  • Primary region: full LiteLLM stack (pods + PostgreSQL + Redis)
  • Standby region: LiteLLM pods ready but routing 0% traffic; PostgreSQL read replica
  • Failover: promote standby PostgreSQL to primary, update DATABASE_URL in standby LiteLLM pods, shift DNS/ALB to standby region
  • RTO: 5–15 minutes (automated with Route 53 health checks)
  • RPO: seconds (Aurora Global Database with 1-second replication lag)

Active-active (via LiteLLM HA Control Plane — Enterprise): LiteLLM’s HA Control Plane deploys independent worker instances per region, each with its own PostgreSQL and Redis, managed via a single admin UI:

# control-plane config
general_settings:
  worker_registry:
    - worker_id: "worker-us-east-1"
      name: "US East"
      url: "http://litellm-us-east.internal:4000"
    - worker_id: "worker-eu-west-1"
      name: "EU West"
      url: "http://litellm-eu-west.internal:4000"

Each regional worker has full autonomy — no shared state between regions. This means:

  • Rate limits are per-region (a team in US and EU have separate counters unless you enforce at the ALB level)
  • Spend is per-region (aggregate in your BI/OTEL layer)
  • Virtual keys must be issued per-region or replicated via your IaC tooling

AWS Bedrock cross-region note: When using Bedrock system inference profiles (e.g., us.anthropic.claude-3-5-sonnet-20241022-v2:0) from LiteLLM, Bedrock itself handles regional routing within the zone. This is separate from LiteLLM multi-region — LiteLLM can be in us-east-1 while Bedrock distributes across us-east-1, us-west-2, and us-west-1 transparently. For this reason, many enterprises run a single-region LiteLLM deployment and rely on Bedrock’s built-in cross-region inference for throughput, reserving LiteLLM multi-region only for data residency requirements.

Aurora Global Database is the recommended PostgreSQL backend for active-active, with one writer endpoint per region and sub-1-second replication. Wire each regional LiteLLM stack to its local Aurora endpoint to avoid cross-region write latency.

11. Bedrock Cost Attribution — All Mechanisms

Application Inference Profiles (§9) are the AWS-native attribution path but are broken in LiteLLM (issue #18258, open as of Jun 2026). Three viable alternatives cover every granularity requirement.

Mechanism 1 — IAM Principal Attribution (GA April 2026, Recommended)

AWS natively records the caller IAM identity on every bedrock-runtime call and maps principal tags to Cost Explorer. No API changes to LiteLLM calls required.

Option A — Static role tag per project (simplest):

aws iam create-role --role-name BedrockRole-Mantle \
  --assume-role-policy-document file://trust.json

aws iam tag-role --role-name BedrockRole-Mantle \
  --tags Key=Project,Value=Mantle Key=CostCenter,Value=CC-1042

LiteLLM config.yaml:

model_list:
  - model_name: bedrock-claude-mantle
    litellm_params:
      model: bedrock/anthropic.claude-opus-4-5
      aws_role_name: arn:aws:iam::123456789012:role/BedrockRole-Mantle
      aws_session_name: litellm-mantle
      aws_region_name: us-east-1

Trust policy on BedrockRole-Mantle must allow sts:AssumeRole and sts:TagSession from the LiteLLM execution role.

Option B — STS session tags on a shared role (dynamic, multi-tenant):

import boto3
from functools import lru_cache

@lru_cache(maxsize=64)
def get_project_credentials(project_name: str):
    sts = boto3.client("sts")
    return sts.assume_role(
        RoleArn="arn:aws:iam::123456789012:role/BedrockSharedRole",
        RoleSessionName=f"litellm-{project_name}",
        Tags=[
            {"Key": "Project", "Value": project_name},
            {"Key": "CostCenter", "Value": PROJECT_COST_CENTERS[project_name]},
        ],
    )["Credentials"]

Do not call AssumeRole per request — STS default quota is 500 calls/sec/account. Cache credentials for session lifetime (up to 12h).

Activation (one-time): AWS Billing console → Cost Allocation Tags → filter type “IAM principal” → activate Project and CostCenter. Allow 24h propagation. Tags appear in CUR 2.0 with prefix iamPrincipal/Project.

Granularity: per-usage-type per-day per-identity. Not per-request.

Limitation: Only bedrock-runtime APIs. Does not apply to Bedrock Agents or Knowledge Bases managed APIs.

Mechanism 2 — requestMetadata + S3/Athena (Per-Request Detail)

LiteLLM passes requestMetadata natively to the Bedrock Converse API since PR #14570 (Sep 22, 2025; use LiteLLM ≥ 1.53). This does not appear in Cost Explorer — it appears in Bedrock invocation logs only.

Enable Bedrock invocation logging:

aws bedrock put-model-invocation-logging-configuration \
  --logging-config '{
    "s3Config": {"bucketName": "my-bedrock-logs", "keyPrefix": "bedrock-logs/"},
    "textDataDeliveryEnabled": true,
    "imageDataDeliveryEnabled": false,
    "embeddingDataDeliveryEnabled": false
  }'

Pass metadata on every call:

response = litellm.completion(
    model="bedrock/anthropic.claude-opus-4-5",
    messages=[{"role": "user", "content": "..."}],
    requestMetadata={
        "project": "Mantle",
        "virtual_key": "sk-mantle-prod",
        "environment": "prod",
        "feature": "contract-review",
    }
)

Limits: max 16 key-value pairs, keys/values max 256 chars.

Athena cost query:

SELECT
  requestMetadata.project         AS project,
  modelId,
  SUM(input.inputTokenCount)      AS input_tokens,
  SUM(output.outputTokenCount)    AS output_tokens,
  ROUND(SUM(input.inputTokenCount)  * 0.000015, 4) AS est_input_cost_usd,
  ROUND(SUM(output.outputTokenCount) * 0.000075, 4) AS est_output_cost_usd
FROM bedrock_invocation_logs
WHERE requestMetadata.project = 'Mantle'
  AND dt BETWEEN '2026-06-01' AND '2026-06-30'
GROUP BY 1, 2
ORDER BY est_input_cost_usd DESC;

Token-count-based cost is an estimate — multiply against current Bedrock pricing. For invoice-accurate totals, join to CUR at the model+usage-type grain (no per-request ID in CUR line items).

Mechanism 3 — LiteLLM SpendLogs as Internal Chargeback

LiteLLM’s LiteLLM_SpendLogs table is a fully viable substitute for Cost Explorer chargeback at per-request granularity. Assign one virtual key per project.

Create a virtual key for a project:

curl -X POST https://your-litellm-proxy/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -d '{
    "team_id": "team-mantle",
    "metadata": {"project": "Mantle"},
    "tags": ["project:mantle", "env:prod"],
    "key_alias": "mantle-prod"
  }'

Per-project spend query:

SELECT
  v.team_id,
  DATE_TRUNC('day', s."startTime") AS day,
  SUM(s.spend)                      AS cost_usd,
  SUM(s.prompt_tokens)              AS input_tokens,
  SUM(s.completion_tokens)          AS output_tokens
FROM "LiteLLM_SpendLogs" s
JOIN "LiteLLM_VerificationToken" v ON s.api_key = v.token
WHERE v.team_id = 'team-mantle'
  AND s."startTime" >= '2026-06-01'
GROUP BY 1, 2
ORDER BY 2 DESC;

Or via API: GET /global/spend/report?group_by=team

Critical caveat: LiteLLM calculates spend using its own rate card, not the AWS invoice. Rates drift if you have reserved capacity, batch pricing, or negotiated rates. Use SpendLogs for quota enforcement and allocation; reconcile to CUR for invoice accuracy.

Scaling: At >1M rows, enable GCS/BigQuery callback in config.yaml to stream SpendLogs out of Postgres.

Decision Matrix

Mechanism In Cost Explorer Per-request LiteLLM config Works today
IAM role tag (static) Yes (iamPrincipal/) No (daily) aws_role_name per model Yes (GA Apr 2026)
STS session tags Yes (iamPrincipal/) No (daily) AssumeRole hook + aws_role_name Yes (GA Apr 2026)
requestMetadata + Athena No (logs only) Yes requestMetadata param Yes (LiteLLM ≥1.53)
Separate AWS account Yes (native, zero config) No (daily) Separate credentials per model Yes
LiteLLM SpendLogs No (internal) Yes Virtual keys + team_id Yes
Application Inference Profiles Yes No (daily) Model ARN Broken (#18258)

Run all three in parallel — they are independent and non-conflicting:

  1. Billing-accurate daily totals in Cost Explorer: IAM session tags (Mechanism 1B). One AssumeRole call per project session, cached. Activate Project as cost allocation tag in Billing console.

  2. Per-prompt attribution for audit/debugging: Bedrock invocation logging + requestMetadata (Mechanism 2). Ships to S3, queryable via Athena or CloudWatch Logs Insights.

  3. Near-real-time quota enforcement and internal chargeback: LiteLLM SpendLogs (Mechanism 3). One virtual key per project, query Postgres or export to warehouse.

Sources: AWS Bedrock IAM principal attribution (GA Apr 9, 2026), AWS Bedrock request-level metadata (GA May 2026), LiteLLM PR #14570, LiteLLM issue #18258.