← Agent Frameworks 🕐 15 min read
Agent Frameworks

LiteLLM Production Deployment at Scale — Multi-Worker, Redis, HA (2026)

Overview · March 2026

LiteLLM Proxy is a FastAPI-based LLM gateway that fronts 100+ providers with a unified OpenAI-compatible API surface.

LiteLLM Proxy is a FastAPI-based LLM gateway that fronts 100+ providers with a unified OpenAI-compatible API surface. At enterprise scale (50+ teams, 1,000+ users, sustained throughput above 500 RPS), naive single-instance deployments develop predictable failure modes: PostgreSQL connection exhaustion, race conditions on shared budget state, Prometheus metrics that flicker under load, and SpendLogs tables that balloon to hundreds of millions of rows within months.

This note covers the full production stack — worker process topology, Redis distributed state, Postgres schema and growth management, ALB load balancing, HA with Redis Sentinel, Kubernetes manifests, enterprise config patterns, known bugs with issue numbers, and observed latency overhead benchmarks.

1. Multi-Worker Deployment: Uvicorn, Gunicorn, and PROMETHEUS_MULTIPROC_DIR

Server topology options

LiteLLM ships with three supported process topologies:

Mode Command Use case
Single uvicorn litellm --config config.yaml Development, single pod
Multi-worker uvicorn litellm --config config.yaml --num_workers 4 Single large VM
Gunicorn + uvicorn workers litellm --config config.yaml --run_gunicorn --num_workers 4 Multi-worker with process supervision

The official guidance as of 2026 is one uvicorn worker per Kubernetes pod, scaled horizontally. Running multiple workers per container is supported but introduces two complications: Prometheus metrics fragmentation (see below) and unpredictable per-pod memory growth that makes HPA thresholds unreliable.

When multiple workers per pod are required (e.g., bare-metal deployments without a container orchestrator), set --max_requests_before_restart to bound memory growth:

litellm \
  --config /app/config.yaml \
  --run_gunicorn \
  --num_workers 4 \
  --max_requests_before_restart 50000

Workers recycle after 50,000 requests each, preventing unbounded RSS growth from Python object accumulation.

PROMETHEUS_MULTIPROC_DIR: why it exists and how to configure it

Each uvicorn worker maintains its own in-memory Prometheus counter state after the fork. Without multiprocess mode, the /metrics endpoint is served by whichever worker receives the scrape request. Because Prometheus uses round-robin or random worker selection, counters appear to decrease between scrapes — a spike in worker 2 disappears when the next scrape lands on worker 1.

The prometheus_client library solves this with shared memory-mapped files:

# Create the directory and export before starting the proxy
mkdir -p /tmp/prometheus_multiproc
export PROMETHEUS_MULTIPROC_DIR=/tmp/prometheus_multiproc

Critical: the directory must be empty at proxy startup and persist for the lifetime of the proxy process. Stale files from a previous run cause counter resets. In Kubernetes, use an emptyDir volume mounted at the path — it is automatically wiped on pod restart.

# Kubernetes volume for multiprocess Prometheus
volumes:
  - name: prometheus-multiproc
    emptyDir: {}

volumeMounts:
  - name: prometheus-multiproc
    mountPath: /tmp/prometheus_multiproc

env:
  - name: PROMETHEUS_MULTIPROC_DIR
    value: /tmp/prometheus_multiproc

The /metrics handler must use MultiProcessCollector to aggregate across files:

# LiteLLM handles this internally when PROMETHEUS_MULTIPROC_DIR is set,
# but if you build a custom metrics middleware:
from prometheus_client import multiprocess, CollectorRegistry, generate_latest
from fastapi import Response

@app.get("/metrics")
async def metrics():
    registry = CollectorRegistry()
    multiprocess.MultiProcessCollector(registry)
    return Response(generate_latest(registry), media_type="text/plain")

When the Gunicorn worker_exit hook fires, register multiprocess.mark_process_dead(os.getpid()) to remove stale files from exiting workers.

Metrics available in production

Metric Type Description
litellm_proxy_total_requests_metric Counter Requests by model, key, team
litellm_proxy_failed_requests_metric Counter Failures by error type
litellm_request_total_latency_metric Histogram End-to-end latency
litellm_llm_api_latency_metric Histogram Provider round-trip only
litellm_spend_metric Gauge Cumulative spend by user/key/team
litellm_remaining_team_budget_metric Gauge Remaining budget headroom
litellm_in_flight_requests Gauge Current queue depth
litellm_deployment_failure_responses Counter Backend failures by model_id
litellm_redis_spend_update_queue_size Gauge Pending DB flush queue depth
litellm_pod_lock_manager_size Gauge Active flush lock holders

Enable authentication on the metrics endpoint in production:

litellm_settings:
  require_auth_for_metrics_endpoint: true

2. Redis: Distributed Rate Limiting, Request Deduplication, and Auth Caching

Redis is not optional at scale. Without it, each proxy instance maintains independent rate-limit counters — two pods will both allow a user to 2x their TPM limit before the batch write reconciles. Redis is the shared state layer that makes horizontal scaling safe.

Core Redis configuration

litellm_settings:
  cache: true
  cache_params:
    type: redis
    host: os.environ/REDIS_HOST
    port: os.environ/REDIS_PORT
    password: os.environ/REDIS_PASSWORD
    max_connections: 100        # per-pod connection pool
    namespace: litellm_prod     # isolate from other Redis users
    ttl: 3600                   # default cache TTL in seconds
    default_in_redis_ttl: 86400 # spend tracking window

Always use os.environ/ references, never hardcoded credentials. Redis 7.0+ is required.

Redis use cases in a production LiteLLM deployment

Rate limit counters — litellm_settings.cache: true with Redis causes LiteLLM to write per-key, per-user, and per-team TPM/RPM counters to Redis on every request. All proxy pods read and write the same counters, giving consistent enforcement across the fleet.

Redis transaction buffer — the recommended HA configuration for preventing PostgreSQL deadlocks (see section 5). Proxy pods write spend increments to Redis instead of Postgres; a single elected pod flushes the queue to the DB.

Virtual key auth caching — reduces database reads on every request. Each incoming API key triggers a DB lookup to validate the key and load its budget state. At 1,000 RPS with 100ms DB round-trips, that is 100 DB connections/second just for auth. Caching amortizes this:

general_settings:
  enable_redis_auth_cache: true

Response caching — semantic and exact-match caching to avoid redundant provider calls:

litellm_settings:
  cache: true
  cache_params:
    type: redis
    supported_call_types:
      - acompletion
      - completion

Redis Cluster vs. Redis Sentinel

For HA Redis, choose based on data volume:

Mode When to use LiteLLM support
Redis Sentinel <100GB dataset, simple ops, existing replication setup Via startup_nodes pointing to sentinel
Redis Cluster >100GB, horizontal write scaling needed RedisClusterCache with startup_nodes list
Managed (ElastiCache, Upstash, Redis Cloud) Preferred for teams without Redis ops expertise Standard host/port/password, add SSL

Redis Sentinel minimum architecture: 1 primary, 2 replicas, 3 sentinel processes. Sentinels require a quorum of 2 to elect a new primary. Configure the primary to require at least one replica acknowledgment before accepting writes:

# redis.conf (primary)
min-replicas-to-write 1
min-replicas-max-lag 10

LiteLLM does not natively expose Sentinel URLs in config.yaml — use a Sentinel-aware proxy (redis-sentinel-proxy or HAProxy with health-check-based backend selection) in front of Redis and point LiteLLM at the virtual IP. Alternatively, configure via environment:

REDIS_HOST=redis-sentinel-ha.internal
REDIS_PORT=6379
REDIS_PASSWORD=<password>
REDIS_SSL=true

Connection pool sizing

At 500 RPS with 4 proxy pods and 4 workers each, peak Redis connections = 4 pods × 100 max_connections = 400 connections. Add headroom for burst. Redis 7.0 default maxclients is 10,000; most managed tiers cap at 1,000-10,000. Monitor litellm_redis_spend_update_queue_size — a growing queue means the flush rate is falling behind.

Known issue #9024: under sustained load, connections can be closed by the Redis server with “Connection closed by server” errors. The workaround is to set max_connections explicitly (it defaults low) and to enable TCP keepalive at the OS level.

3. Database Connection Pooling and PostgreSQL at Scale

Why Postgres connection exhaustion happens

LiteLLM uses Prisma as its ORM. Without a connection pooler, each uvicorn worker maintains its own connection pool of database_connection_pool_limit connections to Postgres. The default is 10. With 10 pods × 1 worker × 10 connections = 100 connections — fine for max_connections=100 on a default Postgres instance.

The problem scales quadratically: 20 pods × 4 workers × 10 connections = 800 connections. A db.t3.medium RDS instance tops out at 90 connections. Even db.r6g.2xlarge caps at ~800. Connection exhaustion produces:

FATAL: sorry, too many clients already
PrismaClientInitializationError: Can't reach database server

PgBouncer as connection pooler

Deploy PgBouncer in transaction-mode pooling between LiteLLM pods and Postgres. LiteLLM’s Prisma-generated queries are transaction-safe and work correctly in transaction mode.

# pgbouncer.ini
[databases]
litellm = host=postgres-primary port=5432 dbname=litellm

[pgbouncer]
pool_mode = transaction
max_client_conn = 1000      # matches expected peak proxy connections
default_pool_size = 20      # actual connections to Postgres
reserve_pool_size = 5
reserve_pool_timeout = 3
server_idle_timeout = 600
log_connections = 0         # disable at scale — log volume overhead

With PgBouncer, LiteLLM’s DATABASE_URL points to PgBouncer’s port (5432 by default). Prisma session-level features (prepared statements, advisory locks) do not work in transaction mode — this is fine because LiteLLM does not rely on session-level Postgres features.

LiteLLM connection pool settings

general_settings:
  database_connection_pool_limit: 10     # per worker, to PgBouncer
  database_connection_timeout: 10        # seconds before failing a checkout
  database_socket_timeout: 30            # idle socket close

Set database_connection_pool_limit to 10 even when PgBouncer is in front — let PgBouncer manage the multiplexing. Setting it higher multiplies client-side memory overhead from Prisma’s connection objects without benefit.

Appending SSL and connection parameters

general_settings:
  database_extra_connection_params:
    sslmode: require
    connect_timeout: 10
    application_name: litellm_prod

4. LiteLLM_SpendLogs: Schema, Growth Management, and Archival

Full table schema (from schema.prisma, main branch, June 2026)

-- Equivalent SQL DDL for LiteLLM_SpendLogs
CREATE TABLE "LiteLLM_SpendLogs" (
    "request_id"               TEXT PRIMARY KEY,
    "call_type"                TEXT NOT NULL,
    "api_key"                  TEXT NOT NULL DEFAULT '',
    "spend"                    FLOAT8 NOT NULL DEFAULT 0.0,
    "total_tokens"             INT4 NOT NULL DEFAULT 0,
    "prompt_tokens"            INT4 NOT NULL DEFAULT 0,
    "completion_tokens"        INT4 NOT NULL DEFAULT 0,
    "startTime"                TIMESTAMPTZ NOT NULL,
    "endTime"                  TIMESTAMPTZ NOT NULL,
    "request_duration_ms"      INT4,
    "completionStartTime"      TIMESTAMPTZ,
    "model"                    TEXT NOT NULL DEFAULT '',
    "model_id"                 TEXT DEFAULT '',
    "model_group"              TEXT DEFAULT '',
    "custom_llm_provider"      TEXT DEFAULT '',
    "api_base"                 TEXT DEFAULT '',
    "user"                     TEXT DEFAULT '',
    "metadata"                 JSONB DEFAULT '{}',
    "cache_hit"                TEXT DEFAULT '',
    "cache_key"                TEXT DEFAULT '',
    "request_tags"             JSONB DEFAULT '[]',
    "team_id"                  TEXT,
    "organization_id"          TEXT,
    "end_user"                 TEXT,
    "requester_ip_address"     TEXT,
    "messages"                 JSONB DEFAULT '{}',
    "response"                 JSONB DEFAULT '{}',
    "session_id"               TEXT,
    "status"                   TEXT,
    "mcp_namespaced_tool_name" TEXT,
    "agent_id"                 TEXT,
    "proxy_server_request"     JSONB DEFAULT '{}'
);

-- Indexes (from Prisma schema)
CREATE INDEX "LiteLLM_SpendLogs_startTime_idx" ON "LiteLLM_SpendLogs" ("startTime");
CREATE INDEX "LiteLLM_SpendLogs_startTime_request_id_idx" ON "LiteLLM_SpendLogs" ("startTime", "request_id");
CREATE INDEX "LiteLLM_SpendLogs_end_user_idx" ON "LiteLLM_SpendLogs" ("end_user");
CREATE INDEX "LiteLLM_SpendLogs_session_id_idx" ON "LiteLLM_SpendLogs" ("session_id");

The messages and response JSONB columns are the primary storage consumers. A typical ChatCompletion request with a 500-token exchange stores ~2-8KB per row. At 500 RPS, that is 43M rows/day and ~200GB/day before any JSONB compression. Disable prompt/response storage in production unless you have a specific compliance requirement:

general_settings:
  store_prompts_in_spend_logs: false  # default false; double-check this is not set

Growth estimate by traffic tier

RPS Rows/day Rows/30d Row size (no JSONB) Row size (with JSONB)
50 4.3M 129M ~160GB ~260GB
200 17.3M 519M ~640GB ~1TB
500 43.2M 1.3B ~1.6TB ~2.5TB

These numbers assume store_prompts_in_spend_logs: false. Enable it and multiply by 3-5x.

Native TTL: maximum_spend_logs_retention_period

LiteLLM 1.x introduced a built-in cleanup job:

general_settings:
  maximum_spend_logs_retention_period: "90d"     # delete rows older than 90 days
  maximum_spend_logs_retention_interval: "24h"    # run cleanup every 24 hours

The cleanup job runs DELETE FROM "LiteLLM_SpendLogs" WHERE "startTime" < NOW() - INTERVAL '90 days'. On a large table, this issues a full-table scan. On a table with 500M+ rows this causes table bloat, lock pressure, and autovacuum storms. The native TTL is suitable for tables under ~100M rows. For larger deployments, use partitioning.

Declarative partitioning by month (Postgres 13+)

Prisma does not generate partitioned tables. Apply this DDL once, outside Prisma migrations, and configure LiteLLM with DISABLE_SCHEMA_UPDATE=true on all pods that are not the migration leader.

-- Step 1: Rename existing table
ALTER TABLE "LiteLLM_SpendLogs" RENAME TO "LiteLLM_SpendLogs_unpartitioned";

-- Step 2: Create partitioned parent (same column list)
CREATE TABLE "LiteLLM_SpendLogs" (
    "request_id"               TEXT NOT NULL,
    "call_type"                TEXT NOT NULL,
    "api_key"                  TEXT NOT NULL DEFAULT '',
    "spend"                    FLOAT8 NOT NULL DEFAULT 0.0,
    "total_tokens"             INT4 NOT NULL DEFAULT 0,
    "prompt_tokens"            INT4 NOT NULL DEFAULT 0,
    "completion_tokens"        INT4 NOT NULL DEFAULT 0,
    "startTime"                TIMESTAMPTZ NOT NULL,
    "endTime"                  TIMESTAMPTZ NOT NULL,
    "request_duration_ms"      INT4,
    "completionStartTime"      TIMESTAMPTZ,
    "model"                    TEXT NOT NULL DEFAULT '',
    "model_id"                 TEXT DEFAULT '',
    "model_group"              TEXT DEFAULT '',
    "custom_llm_provider"      TEXT DEFAULT '',
    "api_base"                 TEXT DEFAULT '',
    "user"                     TEXT DEFAULT '',
    "metadata"                 JSONB DEFAULT '{}',
    "cache_hit"                TEXT DEFAULT '',
    "cache_key"                TEXT DEFAULT '',
    "request_tags"             JSONB DEFAULT '[]',
    "team_id"                  TEXT,
    "organization_id"          TEXT,
    "end_user"                 TEXT,
    "requester_ip_address"     TEXT,
    "messages"                 JSONB DEFAULT '{}',
    "response"                 JSONB DEFAULT '{}',
    "session_id"               TEXT,
    "status"                   TEXT,
    "mcp_namespaced_tool_name" TEXT,
    "agent_id"                 TEXT,
    "proxy_server_request"     JSONB DEFAULT '{}'
) PARTITION BY RANGE ("startTime");

-- Step 3: Create partitions (automate with pg_partman for ongoing creation)
CREATE TABLE "LiteLLM_SpendLogs_2026_06"
    PARTITION OF "LiteLLM_SpendLogs"
    FOR VALUES FROM ('2026-06-01') TO ('2026-07-01');

CREATE TABLE "LiteLLM_SpendLogs_2026_07"
    PARTITION OF "LiteLLM_SpendLogs"
    FOR VALUES FROM ('2026-07-01') TO ('2026-08-01');

-- Step 4: Migrate existing data (run during maintenance window or use logical replication)
INSERT INTO "LiteLLM_SpendLogs" SELECT * FROM "LiteLLM_SpendLogs_unpartitioned";

-- Step 5: Create indexes on each partition
CREATE INDEX ON "LiteLLM_SpendLogs_2026_06" ("startTime");
CREATE INDEX ON "LiteLLM_SpendLogs_2026_06" ("team_id", "startTime");
CREATE INDEX ON "LiteLLM_SpendLogs_2026_06" ("api_key", "startTime");

Archival: drop old partitions in a cron job. Dropping a partition is an O(1) metadata operation — no row-level DELETE, no autovacuum, no table bloat:

-- Drop partition older than 90 days (run via pg_cron or Lambda)
DROP TABLE IF EXISTS "LiteLLM_SpendLogs_2026_03";

Use pg_partman to automate monthly partition creation and retention:

SELECT partman.create_parent(
    p_parent_table := 'public.LiteLLM_SpendLogs',
    p_control := 'startTime',
    p_type := 'range',
    p_interval := '1 month',
    p_retention := '3 months',
    p_retention_schema := 'archive'  -- move to archive schema instead of drop
);

5. Load Balancing: ALB, Sticky Sessions, and Health Checks

Stateless architecture (preferred)

LiteLLM is designed to be stateless when Redis is present. All rate-limit counters, key validation state, and spend updates live in Redis and Postgres. Any pod can serve any request. ALB configuration is standard round-robin without sticky sessions:

                   ┌─────────────┐
                   │     ALB     │
                   │  round-robin│
                   └──────┬──────┘
          ┌───────────────┼───────────────┐
          ▼               ▼               ▼
   ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
   │  LiteLLM    │ │  LiteLLM    │ │  LiteLLM    │
   │  Pod 1      │ │  Pod 2      │ │  Pod 3      │
   └──────┬──────┘ └──────┬──────┘ └──────┬──────┘
          │               │               │
          └───────────────┼───────────────┘
                          ▼
              ┌───────────────────────┐
              │  Redis (rate limits,  │
              │  auth cache, spend    │
              │  update queue)        │
              └───────────┬───────────┘
                          │
              ┌───────────▼───────────┐
              │  PostgreSQL +         │
              │  PgBouncer            │
              └───────────────────────┘

Sticky sessions are actively harmful in this architecture. If a pod becomes unhealthy, sticky sessions hold clients to that pod after ALB removes it from the rotation, producing a sustained stream of 5xx errors until the sticky cookie expires (default: 1 day). Keep ALB stickiness off.

ALB health check configuration

Health check path:   /health/readiness
Protocol:            HTTP (or HTTPS if TLS terminates at pod)
Port:                4000 (LiteLLM default)
Healthy threshold:   2 consecutive successes
Unhealthy threshold: 3 consecutive failures
Timeout:             5 seconds
Interval:            30 seconds

/health/readiness performs a DB round-trip and returns {"status": "healthy", "db": "connected"}. This is the correct probe for ALB — it removes pods whose DB connection is broken from rotation. /health/liveliness (which returns "I'm alive!" without a DB check) is appropriate only for Kubernetes liveness probes, not ALB health checks.

Graceful drain at deregistration

ALB waits up to deregistration_delay.timeout_seconds (default: 300s) after marking a target unhealthy before terminating in-flight connections. For LiteLLM, this should exceed the longest expected LLM call. Long-running streaming calls to GPT-o3 or Claude Opus can exceed 120 seconds. Set:

deregistration_delay.timeout_seconds = 620

This matches the terminationGracePeriodSeconds: 620 recommendation in the Kubernetes section.

6. HA Architecture: Primary + Replica, Redis Transaction Buffer

Full HA topology

                       ┌──────────────────────────────────────┐
                       │              Route 53 / ALB           │
                       └───────────────────┬──────────────────┘
                                           │
              ┌────────────────────────────┼───────────────────────────┐
              │                            │                           │
   ┌──────────▼───────────┐   ┌────────────▼──────────┐  ┌───────────▼────────────┐
   │   LiteLLM Pod (AZ-a) │   │  LiteLLM Pod (AZ-b)   │  │  LiteLLM Pod (AZ-c)   │
   │   1 worker / pod     │   │  1 worker / pod        │  │  1 worker / pod        │
   └──────────┬───────────┘   └────────────┬──────────┘  └───────────┬────────────┘
              │                            │                          │
              └──────────────┬─────────────┘──────────────┘──────────┘
                             │
              ┌──────────────▼──────────────────────────────────────┐
              │             Redis (primary + 2 replicas              │
              │             Sentinel quorum = 3 sentinels)           │
              │             ElastiCache Multi-AZ (managed option)    │
              └──────────────┬──────────────────────────────────────┘
                             │  (Redis transaction buffer flushes here)
              ┌──────────────▼──────────────────────────────────────┐
              │   PostgreSQL Primary (AZ-a)                         │
              │   + Read Replica (AZ-b) — dashboard queries only    │
              │   + PgBouncer (transaction mode) on each pod        │
              └─────────────────────────────────────────────────────┘

Redis transaction buffer for deadlock prevention

Under concurrent load, multiple LiteLLM pods execute simultaneous UPDATE statements on the same LiteLLM_UserTable, LiteLLM_TeamTable, and LiteLLM_VerificationToken rows (incrementing spend). PostgreSQL row-level locks cause deadlocks above roughly 50 concurrent writers.

The Redis transaction buffer routes all spend increments through Redis first:

general_settings:
  use_redis_transaction_buffer: true

litellm_settings:
  cache: true
  cache_params:
    type: redis
    max_connections: 100

One pod acquires a distributed lock (litellm_pod_lock_manager_size metric tracks this) and flushes the Redis queue to Postgres in a single aggregated transaction. All other pods skip the DB write and return immediately. The elected flusher changes on each cycle, providing leader rotation without a separate election process.

Monitor queue depth — if litellm_redis_spend_update_queue_size grows monotonically, the flusher is falling behind and spend updates will be delayed in the UI:

general_settings:
  alerting:
    - slack
  alerting_threshold: 300   # alert if spend queue exceeds 300 pending items

Postgres read replica for dashboard queries

The LiteLLM admin UI’s /global/spend/report endpoint issues expensive GROUP BY queries against SpendLogs. Route these to the read replica to prevent analytics scans from competing with write traffic:

DATABASE_URL="postgresql://litellm:pass@pgbouncer-primary:5432/litellm"
DATABASE_URL_REPLICA="postgresql://litellm:pass@postgres-replica:5432/litellm"

LiteLLM does not yet natively support read/write split at the ORM level. The practical pattern is to run a second LiteLLM instance in read-only mode (pointing its DATABASE_URL to the replica) and expose it only to the internal admin dashboard, not to the request path.

7. Virtual Keys and Teams at Scale: Database vs. In-Memory

Database-backed (production requirement)

All virtual key state — budgets, spend, rate limits, model allowlists — persists in Postgres. Without a database, keys exist only in the running process and are lost on pod restart. For multi-pod deployments, in-memory keys are outright broken: pod 1 generates the key, pod 2 has no record of it, request fails with 401.

The relevant tables:

LiteLLM_VerificationToken  — one row per virtual key
LiteLLM_TeamTable          — one row per team (has spend, budget fields)
LiteLLM_UserTable          — one row per user
LiteLLM_BudgetTable        — reusable budget objects linked to keys/teams/users
LiteLLM_OrganizationTable  — top-level namespace

Budget hierarchy (spend enforcement order): key budget → team budget → organization budget. The strictest non-null limit wins.

Key generation patterns for 50+ teams

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

  # Prevent users from self-issuing unlimited keys
  default_key_generate_params:
    max_budget: 10.0        # $10 default per key
    budget_duration: "30d"  # reset monthly
    tpm_limit: 100000       # 100k tokens/min default

  # Hard cap — API rejects key/generate requests exceeding these
  upperbound_key_generate_params:
    max_budget: 1000.0      # $1000 max any key can be issued
    tpm_limit: 1000000      # 1M tokens/min max
    max_parallel_requests: 100

  # Require team assignment on key generation
  default_team_disabled: true  # prevent personal keys outside of teams
  enforce_rbac: true

For programmatic key provisioning (CI/CD pipelines issuing keys per deployment):

import httpx

def provision_team_key(team_id: str, service_name: str, budget_usd: float = 50.0):
    response = httpx.post(
        "https://litellm-proxy.internal/key/generate",
        headers={"Authorization": f"Bearer {MASTER_KEY}"},
        json={
            "team_id": team_id,
            "key_alias": f"{service_name}-prod",
            "max_budget": budget_usd,
            "budget_duration": "30d",
            "models": ["gpt-4o", "claude-sonnet-4-6", "gemini-2.5-pro"],
            "metadata": {
                "service": service_name,
                "provisioned_by": "terraform",
                "environment": "production"
            }
        }
    )
    return response.json()["key"]

Auto-rotation

Enterprise feature for rotating keys without downtime:

general_settings:
  key_rotation_settings:
    enabled: true
    rotation_interval: "90d"
    grace_period: "24h"   # old key stays valid for 24h after new key issued

In-memory caching of key auth (Redis auth cache)

At 1,000+ RPS, every request hitting Postgres for key validation saturates the DB. Enable auth caching:

general_settings:
  enable_redis_auth_cache: true

This stores the full key auth payload in Redis with a TTL equal to default_in_redis_ttl. Budget changes (e.g., an admin bumps a team’s limit) take up to TTL seconds to propagate. For real-time budget enforcement on security-sensitive keys, set fail_closed_budget_enforcement: true — this forces a DB check on every budgeted request, bypassing the cache for budget validation only.

8. Cost Attribution Pipeline: SpendLogs → S3 → Athena → Dashboard

S3 export via s3_v2 callback

litellm_settings:
  callbacks: ["s3_v2", "prometheus"]
  s3_callback_params:
    s3_bucket_name: litellm-spend-logs-prod
    s3_region_name: us-east-1
    s3_aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
    s3_aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
    s3_path: "year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}"
    s3_use_team_prefix: true   # prefix object key with team_alias
    s3_use_key_prefix: true    # prefix with key_alias after team
    s3_strip_base64_files: true  # strip base64 image data before export

Objects land at: s3://litellm-spend-logs-prod/year=2026/month=06/day=18/my-team/my-key/<uuid>.json

The Hive-style partition prefix (year=.../month=.../day=...) enables Athena partition projection without a Glue crawler.

Athena table definition

CREATE EXTERNAL TABLE litellm_spend_logs (
    request_id        STRING,
    call_type         STRING,
    api_key           STRING,
    spend             DOUBLE,
    total_tokens      INT,
    prompt_tokens     INT,
    completion_tokens INT,
    startTime         STRING,
    endTime           STRING,
    request_duration_ms INT,
    model             STRING,
    model_group       STRING,
    custom_llm_provider STRING,
    team_id           STRING,
    organization_id   STRING,
    end_user          STRING,
    cache_hit         STRING,
    session_id        STRING,
    status            STRING,
    agent_id          STRING
)
PARTITIONED BY (year STRING, month STRING, day STRING)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
LOCATION 's3://litellm-spend-logs-prod/'
TBLPROPERTIES (
    'projection.enabled' = 'true',
    'projection.year.type' = 'integer',
    'projection.year.range' = '2025,2030',
    'projection.month.type' = 'integer',
    'projection.month.range' = '1,12',
    'projection.month.digits' = '2',
    'projection.day.type' = 'integer',
    'projection.day.range' = '1,31',
    'projection.day.digits' = '2',
    'storage.location.template' = 's3://litellm-spend-logs-prod/year=${year}/month=${month}/day=${day}'
);

Sample cost attribution queries

-- Monthly spend by team
SELECT
    team_id,
    SUM(spend) AS total_spend_usd,
    SUM(total_tokens) AS total_tokens,
    COUNT(*) AS request_count,
    AVG(request_duration_ms) AS avg_latency_ms
FROM litellm_spend_logs
WHERE year = '2026' AND month = '06'
GROUP BY team_id
ORDER BY total_spend_usd DESC;

-- Model cost breakdown by team for budget forecasting
SELECT
    team_id,
    model_group,
    SUM(spend) AS spend_usd,
    SUM(prompt_tokens) AS input_tokens,
    SUM(completion_tokens) AS output_tokens,
    ROUND(SUM(spend) / NULLIF(SUM(total_tokens), 0) * 1e6, 4) AS cost_per_million_tokens
FROM litellm_spend_logs
WHERE year = '2026' AND month = '06'
GROUP BY team_id, model_group;

-- Cache hit rate by team (savings calculation)
SELECT
    team_id,
    COUNT(*) AS total_requests,
    SUM(CASE WHEN cache_hit = 'True' THEN 1 ELSE 0 END) AS cache_hits,
    ROUND(100.0 * SUM(CASE WHEN cache_hit = 'True' THEN 1 ELSE 0 END) / COUNT(*), 2) AS cache_hit_pct,
    SUM(CASE WHEN cache_hit = 'True' THEN spend ELSE 0 END) AS estimated_savings_usd
FROM litellm_spend_logs
WHERE year = '2026' AND month = '06'
GROUP BY team_id;

GCS PubSub for BigQuery (alternative at 1M+ logs/day)

For deployments generating over 1M spend log rows per day, S3 JSON export creates Athena scan costs and cold start latency. The GCS PubSub callback routes events through a topic into BigQuery streaming inserts:

litellm_settings:
  callbacks: ["gcs_pubsub"]
  gcs_pubsub_topic_id: litellm-spend-events
  gcs_pubsub_project_id: os.environ/GCP_PROJECT_ID

BigQuery handles partitioned tables natively and is better suited for BI tools (Looker, Metabase, Superset) than Athena.

9. Production config.yaml for Enterprise (50+ Teams, 1000+ Users)

# litellm-config-enterprise.yaml
# For 50+ teams, 1000+ users, 500+ RPS sustained

model_list:
  # Tier 1: High capacity — load balanced across multiple deployments
  - model_name: gpt-4o
    litellm_params:
      model: azure/gpt-4o-prod-eastus
      api_base: os.environ/AZURE_EASTUS_API_BASE
      api_key: os.environ/AZURE_EASTUS_API_KEY
      api_version: "2025-01-01-preview"
      tpm: 2000000
      rpm: 10000
  - model_name: gpt-4o
    litellm_params:
      model: azure/gpt-4o-prod-westus
      api_base: os.environ/AZURE_WESTUS_API_BASE
      api_key: os.environ/AZURE_WESTUS_API_KEY
      api_version: "2025-01-01-preview"
      tpm: 2000000
      rpm: 10000

  # Tier 2: Anthropic via direct API
  - model_name: claude-sonnet-4-6
    litellm_params:
      model: anthropic/claude-sonnet-4-6
      api_key: os.environ/ANTHROPIC_API_KEY_1
      tpm: 400000
  - model_name: claude-sonnet-4-6
    litellm_params:
      model: anthropic/claude-sonnet-4-6
      api_key: os.environ/ANTHROPIC_API_KEY_2   # second key for load distribution
      tpm: 400000

  # Tier 3: Internal/open models via vLLM
  - model_name: llama-3-70b
    litellm_params:
      model: openai/llama-3-70b
      api_base: http://vllm-service.inference:8000/v1
      api_key: "none"
      tpm: 5000000

router_settings:
  routing_strategy: usage-based-routing-v2
  enable_pre_call_checks: true        # validate model availability before routing
  allowed_fails: 3                    # cooldown after 3 failures
  cooldown_time: 60                   # seconds
  retry_policy:
    AuthenticationErrorRetries: 0
    RateLimitErrorRetries: 3
    TimeoutErrorRetries: 2
    ContentPolicyViolationErrorRetries: 0
  fallbacks:
    - gpt-4o: ["claude-sonnet-4-6"]
    - claude-sonnet-4-6: ["gpt-4o"]
  context_window_fallbacks:
    - gpt-4o: ["gpt-4o-128k"]
  background_health_checks: true
  health_check_interval: 60

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

  # Database
  database_connection_pool_limit: 10
  database_connection_timeout: 10
  database_socket_timeout: 30

  # Redis transaction buffer (prevents DB deadlocks at scale)
  use_redis_transaction_buffer: true

  # Auth caching
  enable_redis_auth_cache: true

  # Key generation guardrails
  default_key_generate_params:
    max_budget: 50.0
    budget_duration: "30d"
    tpm_limit: 500000
  upperbound_key_generate_params:
    max_budget: 2000.0
    tpm_limit: 2000000
    max_parallel_requests: 500

  # Organizational defaults
  default_team_disabled: false    # allow personal keys but with budget caps
  enforce_rbac: true

  # Budget enforcement
  fail_closed_budget_enforcement: true   # DB check on every budgeted request

  # Spend log retention
  maximum_spend_logs_retention_period: "90d"
  maximum_spend_logs_retention_interval: "24h"
  store_prompts_in_spend_logs: false

  # Allow service continuity if DB is momentarily unreachable
  allow_requests_on_db_unavailable: true

  # Observability
  alerting:
    - slack
  alerting_threshold: 300
  slack_alerting_params:
    webhook_url: os.environ/SLACK_WEBHOOK_URL
  service_callbacks:
    - prometheus
    - datadog
  background_health_checks: true

litellm_settings:
  # Logging callbacks
  callbacks:
    - prometheus
    - s3_v2

  s3_callback_params:
    s3_bucket_name: os.environ/SPEND_LOGS_BUCKET
    s3_region_name: us-east-1
    s3_aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
    s3_aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
    s3_path: "year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}"
    s3_use_team_prefix: true
    s3_use_key_prefix: true
    s3_strip_base64_files: true

  # Redis cache
  cache: true
  cache_params:
    type: redis
    host: os.environ/REDIS_HOST
    port: os.environ/REDIS_PORT
    password: os.environ/REDIS_PASSWORD
    max_connections: 100
    namespace: litellm_prod
    ttl: 3600
    default_in_redis_ttl: 86400
    supported_call_types:
      - acompletion
      - completion

  # Prometheus
  prometheus_initialize_budget_metrics: true
  require_auth_for_metrics_endpoint: true

  # Request behavior
  request_timeout: 600
  drop_params: true              # ignore unsupported params silently
  set_verbose: false
  json_logs: true

  # Batch write frequency (seconds)
  background_job_expiry: 60

10. Known Production Bugs and Workarounds

Issue #15360 — SpendLogs bulk insert execution plan explosion

Status: Open, unresolved. Prisma Python client is archived; fix unlikely.

Root cause: create_many() in Prisma Python generates SQL with randomly ordered column lists. Each call produces a unique query string → PostgreSQL creates a new execution plan for each → pg_stat_statements accumulates thousands of near-identical plans → query planner cache (shared_buffers query plan cache) bloats and plan reuse drops to zero.

Symptoms: pg_stat_statements shows thousands of entries for INSERT INTO "LiteLLM_SpendLogs". DB CPU spikes during batch write windows (every 60 seconds). Query insights (Datadog APM, pganalyze) show zero plan reuse for spend log inserts.

Workaround: Increase shared_buffers on Postgres and accept the overhead. Alternatively, disable SpendLogs DB writes entirely and route to S3 only:

general_settings:
  disable_spend_updates: true   # stops DB writes; Prometheus + S3 callbacks still fire

You lose the LiteLLM UI spend dashboard but retain full Prometheus metrics and S3-based Athena analysis.

Issue #15740 — /v1/responses duplicates spend logs for Anthropic/Gemini (v1.77.1+)

Status: Identified, fix in progress. Affects deployments using the Responses API endpoint.

Root cause: The /v1/responses handler triggers two spend log writes per request when the provider is Anthropic or Gemini — one from the main handler and one from the streaming callback.

Workaround: If using /v1/responses, pin to litellm-database:<1.77.1 or disable spend log DB writes and use Prometheus counters for cost attribution.

Issue #21894 — Tag spend views double-count multi-tag requests

Status: Open. The DailyTagSpend database view creates one row per tag. A request with 3 tags appears 3 times in the view, each carrying the full spend amount — producing 3x inflation in tag-based dashboards.

Workaround: Do not use DailyTagSpend view for billing. Query LiteLLM_SpendLogs directly with a LATERAL unnest of request_tags, or use the S3/Athena pipeline where you control the aggregation logic.

Issue #9024 — Redis connection closed by server under load

Status: Known, requires manual tuning.

Symptom: ResponseError: max number of clients reached or ConnectionError: Connection closed by server under sustained load.

Cause: max_connections in cache_params defaults too low (often 10). Under high concurrency, all connections are checked out simultaneously and new requests fail to acquire one.

Fix: Set max_connections: 100 (or higher based on concurrency profile). Also enable OS-level TCP keepalive to prevent Redis server-side idle connection cleanup from closing apparently-idle connections in the pool.

Issue #8498 — PostgreSQL exhaustion with pgCat pooler

Status: Closed as not planned (external to LiteLLM).

Symptom: Intermittent All connection attempts failed errors through pgCat, while direct PgBouncer connections succeed.

Cause: pgCat’s session-state tracking detects SET and PREPARE statements issued by Prisma’s connection setup and marks the server connection as “altered,” discarding it. Prisma issues SET statements as part of connection initialization in some configurations.

Fix: Use PgBouncer instead of pgCat. PgBouncer’s transaction mode is fully compatible with Prisma. pgCat in session mode also works but negates the connection-multiplexing benefit.

Issue #13133 — Broken Postgres connection after idle period

Status: Reported in 2024; partially addressed in later versions.

Symptom: After several hours of low traffic, Postgres connections become stale and requests fail until pods restart.

Fix: Set database_socket_timeout: 30 in general_settings. This closes idle connections that have exceeded the socket timeout, forcing re-establishment rather than using a dead connection. Also set keepalives_idle=60 in database_extra_connection_params.

Supply chain incident — March 2026 (v1.82.7, v1.82.8)

PyPI packages v1.82.7 and v1.82.8 were compromised for approximately 40 minutes. The Docker images on GHCR were not affected. Always pin to GHCR image tags (ghcr.io/berriai/litellm-database:<version>) verified with cosign, not PyPI packages, in production.

11. Performance Benchmarks: Proxy Overhead

Official LiteLLM benchmarks (litellm-performance-benchmarks, 2025-2026)

Test setup: 4 vCPU, 8GB RAM, mocked OpenAI endpoint (60ms response latency), locust load generator.

Configuration Throughput (RPS) Median latency P95 latency P99 latency Proxy overhead (median)
1 instance, 1 worker ~260 RPS 260ms — — ~200ms
2 instances, 1 worker each 1,036 RPS 200ms 630ms 1,200ms 12ms
4 instances, 1 worker each 1,170 RPS 100ms 150ms 240ms 2ms
4 instances, Realtime API 1,207 RPS 59ms 67ms 99ms ~2ms

The 4-instance configuration achieves 2ms median proxy overhead — the sub-millisecond target in the sidecar architecture applies to the forwarding path only; the full proxy path including auth, rate-limit checks, and logging runs at 2-12ms median depending on instance count.

Throughput vs. pod count scaling

Throughput scales approximately linearly with pod count up to the point where Redis or Postgres become the bottleneck. The doubling from 2 to 4 instances halved median latency (200ms → 100ms) and improved P95 from 630ms to 150ms — a near-linear improvement, confirming the stateless horizontal scaling model works correctly when Redis state is shared.

When the proxy overhead matters

For models with <200ms provider latency (small models, cached responses), 12ms proxy overhead at 2 instances represents 6% overhead. For GPT-4o at 800ms TTFB, the same 12ms is 1.5%. The overhead is most visible in:

  1. High-frequency, low-latency requests (embeddings, classification) — consider running embeddings directly, bypassing the proxy
  2. Streaming responses — first-token latency adds the full proxy overhead; subsequent tokens stream at wire speed
  3. Batch inference — proxy overhead is amortized across many tokens; negligible

Issue #21046 — Throughput degradation at 500 concurrent requests

A documented production case reported throughput dropping from 16 req/s on direct GPU to 9 req/s through LiteLLM (a 1.78x overhead ratio) at 500 concurrent requests on a 4-vCPU VM running both LiteLLM and PostgreSQL. The reported optimizations (ulimit tuning, PgBouncer, disabling spend logs) did not resolve the issue.

Analysis: 4 vCPUs shared between LiteLLM and PostgreSQL is a co-location anti-pattern. Separate the database to a dedicated host or RDS. At 500 concurrent requests, LiteLLM’s async event loop competes with Postgres for CPU on the same host.

12. Kubernetes Deployment Patterns

Deployment manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: litellm-proxy
  namespace: ai-gateway
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0        # zero-downtime rolling restarts
  selector:
    matchLabels:
      app: litellm-proxy
  template:
    metadata:
      labels:
        app: litellm-proxy
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "4000"
        prometheus.io/path: "/metrics"
    spec:
      terminationGracePeriodSeconds: 620
      containers:
        - name: litellm
          image: ghcr.io/berriai/litellm-database:v1.85.0   # pin exact tag
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 4000
          args:
            - "--config"
            - "/app/config.yaml"
            - "--port"
            - "4000"
            - "--num_workers"
            - "1"              # one worker per pod; scale via replicas
          env:
            - name: LITELLM_MODE
              value: "PRODUCTION"
            - name: LITELLM_LOG
              value: "ERROR"
            - name: LITELLM_SALT_KEY
              valueFrom:
                secretKeyRef:
                  name: litellm-secrets
                  key: salt-key
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: litellm-secrets
                  key: database-url
            - name: REDIS_HOST
              valueFrom:
                secretKeyRef:
                  name: litellm-secrets
                  key: redis-host
            - name: REDIS_PORT
              value: "6379"
            - name: REDIS_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: litellm-secrets
                  key: redis-password
            - name: PROMETHEUS_MULTIPROC_DIR
              value: /tmp/prometheus_multiproc
            - name: DISABLE_SCHEMA_UPDATE
              value: "true"    # only the migration init-container runs migrations
          resources:
            requests:
              cpu: "1"
              memory: "4Gi"
            limits:
              cpu: "2"
              memory: "8Gi"
          readinessProbe:
            httpGet:
              path: /health/readiness
              port: 4000
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
            timeoutSeconds: 5
          livenessProbe:
            httpGet:
              path: /health/liveliness
              port: 4000
            initialDelaySeconds: 60
            periodSeconds: 30
            failureThreshold: 3
            timeoutSeconds: 3
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 5"]   # drain from ALB before shutdown
          volumeMounts:
            - name: config
              mountPath: /app/config.yaml
              subPath: config.yaml
            - name: prometheus-multiproc
              mountPath: /tmp/prometheus_multiproc
            - name: tmp
              mountPath: /tmp
          securityContext:
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 1000
      initContainers:
        - name: db-migrate
          image: ghcr.io/berriai/litellm-database:v1.85.0
          command: ["sh", "-c", "prisma migrate deploy --schema /app/schema.prisma"]
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: litellm-secrets
                  key: database-url
      volumes:
        - name: config
          configMap:
            name: litellm-config
        - name: prometheus-multiproc
          emptyDir: {}
        - name: tmp
          emptyDir: {}

Note on init-container migration strategy: running prisma migrate deploy in every pod’s init-container risks concurrent migrations. For fleets larger than 2 pods, use a Helm PreSync hook or a dedicated migration Job that runs before the Deployment rollout. All non-migration pods set DISABLE_SCHEMA_UPDATE=true.

HorizontalPodAutoscaler

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: litellm-proxy-hpa
  namespace: ai-gateway
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: litellm-proxy
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
    # Scale on in-flight requests using Prometheus adapter
    - type: Pods
      pods:
        metric:
          name: litellm_in_flight_requests
        target:
          type: AverageValue
          averageValue: "50"     # scale when average queue depth exceeds 50/pod
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
        - type: Pods
          value: 2
          periodSeconds: 60      # add at most 2 pods per minute
    scaleDown:
      stabilizationWindowSeconds: 300   # wait 5 minutes before scaling down
      policies:
        - type: Pods
          value: 1
          periodSeconds: 120

Scaling on litellm_in_flight_requests requires the Prometheus Adapter deployed in the cluster and a MetricServer rule mapping the Prometheus metric to the HPA API. This provides request-rate-aware autoscaling that reacts faster than CPU metrics (which lag behind traffic spikes by 30-60 seconds).

PodDisruptionBudget

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: litellm-proxy-pdb
  namespace: ai-gateway
spec:
  minAvailable: 2               # always keep 2 pods running during node drains
  selector:
    matchLabels:
      app: litellm-proxy

With minAvailable: 2 and maxReplicas: 20, a cluster node drain that tries to evict all pods on a single node will be blocked until the replacement pods are scheduled and ready. This prevents the scenario where node maintenance during a traffic spike takes the gateway offline.

Architecture Quick Reference

ENTERPRISE LITELLM HA STACK — COMPONENT SUMMARY

┌─────────────────────────────────────────────────────────────────────┐
│  INBOUND                                                            │
│  ALB → /health/readiness probe → round-robin, no sticky sessions   │
│  terminationGracePeriodSeconds: 620, preStop: sleep 5              │
└─────────────────────────┬───────────────────────────────────────────┘
                          │
┌─────────────────────────▼───────────────────────────────────────────┐
│  PROXY LAYER (Kubernetes, ai-gateway namespace)                     │
│  Deployment: 4 replicas min, 20 max (HPA on CPU + in_flight reqs)  │
│  Image: ghcr.io/berriai/litellm-database:<pinned>                  │
│  1 uvicorn worker / pod, --max_requests_before_restart 50000       │
│  PROMETHEUS_MULTIPROC_DIR → emptyDir volume                        │
│  DISABLE_SCHEMA_UPDATE=true (migration via init-container)         │
│  PodDisruptionBudget: minAvailable: 2                              │
└──────┬────────────────────────────────────────────────┬────────────┘
       │                                                │
┌──────▼──────────┐                         ┌──────────▼──────────────┐
│  REDIS HA       │                         │  POSTGRES HA             │
│  Primary +      │                         │  Primary (AZ-a) +        │
│  2 replicas +   │◄────────────────────────│  Read replica (AZ-b)    │
│  3 sentinels    │  Redis tx buffer flush  │  PgBouncer               │
│                 │                         │  transaction mode        │
│  Rate limits    │                         │  pool_size: 20           │
│  Auth cache     │                         │  max_client_conn: 1000   │
│  Spend queue    │                         │                          │
│  Response cache │                         │  SpendLogs partitioned   │
└─────────────────┘                         │  by month via pg_partman │
                                            │  90d retention           │
                                            └──────────────────────────┘
                                                        │
                                            ┌──────────▼──────────────┐
                                            │  S3 SPEND LOG EXPORT    │
                                            │  s3_v2 callback         │
                                            │  Hive partitioned path  │
                                            │  team + key prefix      │
                                            └──────────┬──────────────┘
                                                       │
                                            ┌──────────▼──────────────┐
                                            │  ATHENA / BIGQUERY      │
                                            │  Cost attribution SQL   │
                                            │  Team / model drill-down│
                                            │  BI dashboard source    │
                                            └─────────────────────────┘

References and Source Material