← Agent Frameworks 🕐 6 min read
Agent Frameworks

LiteLLM Budget Governance for Enterprise Deployments — 2026

1. Virtual Key Budget Controls · March 2026

curl -X POST 'http://localhost:4000/key/generate' \

See also (wiki): litellm-budget-governance · ai-cost-chargeback-showback

Pillar: 19 — Agent Frameworks
Date: 2026-06-17
Audience: Fortune 500 platform engineers running LiteLLM as an AI gateway
Scope: Budget controls, rate limits, alerting, IaC, cost estimation, and cost-based routing

1.1 Core Parameters

Parameter Type Description
max_budget float (USD) Hard spending cap for the key
budget_duration string Window before spend resets
soft_budget float (USD) Alert threshold (does not block)
model_max_budget dict Per-model budget caps within a single key
temp_budget_increase float Temporary additive increase to max_budget
temp_budget_expiry string Expiry of the temporary increase

Valid budget_duration strings: "30s", "30m", "30h", "1d", "7d", "10d", "30d", "90d", "1mo", "2mo". The format is <integer><unit> where unit is s/m/h/d/mo.

1.2 Key Generation with Budget

curl -X POST 'http://localhost:4000/key/generate' \
  -H 'Authorization: Bearer sk-1234' \
  -H 'Content-Type: application/json' \
  -d '{
    "key_alias": "team-alpha-prod",
    "max_budget": 500.0,
    "budget_duration": "30d",
    "soft_budget": 400.0,
    "tpm_limit": 100000,
    "rpm_limit": 600,
    "model_max_budget": {
      "gpt-4o": {"max_budget": 200.0, "budget_duration": "30d"},
      "claude-3-5-sonnet": {"max_budget": 150.0, "budget_duration": "30d"}
    }
  }'

1.3 Multiple Concurrent Budget Windows

A single key can carry multiple budget windows that reset independently. This prevents daily runaway from exhausting the monthly cap:

curl -X POST 'http://0.0.0.0:4000/key/generate' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{
    "budget_limits": [
      {"budget_duration": "24h", "max_budget": 25},
      {"budget_duration": "30d", "max_budget": 500}
    ]
  }'

Note: Multiple concurrent windows were added in 2026. See Issue #28235 for tracking status.

1.4 Temporary Budget Override (Emergency)

curl -X POST 'http://localhost:4000/key/update' \
  -H 'Authorization: Bearer sk-1234' \
  -H 'Content-Type: application/json' \
  -d '{"key": "sk-abc123", "temp_budget_increase": 100, "temp_budget_expiry": "48h"}'

1.5 Soft vs Hard Limits

Behavior Mechanism
Hard limit (max_budget) Blocks requests once exceeded. Returns BudgetExceededError / HTTP 429. Spend tracked in LiteLLM_VerificationTokenTable.
Soft limit (soft_budget) Sends email alert when crossed; requests continue. Requires email integration (SendGrid, Resend, or SMTP).

There is no native “warn at 80%” threshold on virtual keys. Approximate workaround: set soft_budget = max_budget * 0.8 and max_budget separately. For team-level alerts, the soft budget is configured on the team object (see §2.3).

1.6 Budget Reset Behavior

  • Budget resets are handled by a background cron job that runs by default every 10 minutes (proxy_budget_rescheduler_min_time in general_settings).
  • If budget_duration is not set, the budget never resets.
  • When a key’s budget window expires, spend resets to zero and the window restarts from that moment.
  • Manual reset is not exposed as a dedicated endpoint; use /key/update with spend=0 via the admin API or direct DB manipulation.

Known bug: Issue #25495 — Organization-level budget_duration does not reset after expiry (spend field never zeroed). Affects orgs, not virtual keys directly. Verify behavior for any org-scoped key before production rollout.

Known bug: Issue #27735 — BudgetExceededError fires against stale Redis reservation values; /key/info shows spend below max while requests are being blocked. Root cause: async spend reservation not synchronized with DB truth. Status: open as of June 2026. Mitigation: set max_budget 10–15% above your true intended limit if false-block rate is unacceptable.

2. Team Budget Controls

2.1 Budget Hierarchy

Organization (optional)
  └─ Team (max_budget, budget_duration, soft_budget)
       ├─ Team Member (max_budget_in_team — ceiling within team pool)
       └─ Virtual Key (max_budget — independent per-key cap)

Precedence rule: “If a key belongs to a team, the team budget is applied, not the user’s personal budget.” Key-level max_budget acts as an additional ceiling; the binding limit is whichever is hit first.

2.2 Creating a Team with Budget

curl -X POST 'http://0.0.0.0:4000/team/new' \
  -H 'Authorization: Bearer sk-1234' \
  -H 'Content-Type: application/json' \
  -d '{
    "team_alias": "data-science",
    "max_budget": 2000.0,
    "budget_duration": "30d",
    "tpm_limit": 5000000,
    "rpm_limit": 3000
  }'

The response returns a team_id UUID. All keys generated with "team_id": "<uuid>" share the team’s budget pool.

2.3 Team Soft Budget Alerts

curl -X POST 'http://localhost:4000/team/update' \
  -H 'Authorization: Bearer sk-1234' \
  -H 'Content-Type: application/json' \
  -d '{
    "team_id": "de35b29e-6ca8-4f47-b804-2b79d07aa99a",
    "soft_budget": 1600.0,
    "metadata": {
      "soft_budget_alerting_emails": ["platform-lead@corp.com", "finops@corp.com"]
    }
  }'

Alert deduplication window: 24 hours (one email per team per day, regardless of how many requests cross the threshold). No proxy restart required when adding soft budgets.

2.4 Per-Member Spending Cap Within a Team

# 1. Create or update user
curl -X POST 'http://localhost:4000/user/new' \
  -d '{"user_id": "alice@corp.com"}'

# 2. Add to team with individual ceiling
curl -X POST 'http://localhost:4000/team/member_add' \
  -d '{
    "team_id": "de35b29e-...",
    "member": {"role": "user", "user_id": "alice@corp.com"},
    "max_budget_in_team": 200.0
  }'

2.5 Default Budget for JWT Auto-Provisioned Teams

# config.yaml
litellm_jwtauth:
  team_id_upsert: true
  team_id_jwt_field: "team_id"

litellm_settings:
  default_team_settings:
    - team_id: "default-settings"
      max_budget: 500.0
      budget_duration: "30d"

3. Per-User (End-User) Budget Controls

For pass-through end users (consumers of an app that calls LiteLLM), budgets can be enforced via the user parameter in completions without creating full user accounts:

# config.yaml
litellm_settings:
  max_end_user_budget: 0.05  # $0.05 per user per budget_duration

# Application code — pass end-user ID
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[...],
    user="end-user-tenant-id-xyz"   # tracked against max_end_user_budget
)

For internal users needing full spend visibility, use POST /user/new with max_budget + budget_duration.

4. Model-Level Rate Limits (TPM/RPM)

4.1 Per-Key Rate Limits

curl -X POST 'http://localhost:4000/key/generate' \
  -d '{
    "tpm_limit": 50000,
    "rpm_limit": 300,
    "max_parallel_requests": 20
  }'

By default, TPM counts total tokens (input + output). Override in config.yaml:

litellm_settings:
  token_rate_limit_type: "input"   # "input" | "output" | "total" (default)

4.2 Per-Model TPM/RPM Within a Key

curl -X POST 'http://localhost:4000/key/generate' \
  -d '{
    "metadata": {
      "model_rpm_limit": {"gpt-4o": 100, "claude-3-5-sonnet": 200},
      "model_tpm_limit": {"gpt-4o": 50000, "claude-3-5-sonnet": 80000}
    }
  }'

4.3 Rate Limit Tiers (Enterprise)

The /budget/new + budget_id system (Enterprise) lets you define reusable rate-limit templates and assign them to keys without re-specifying limits per key:

# Create tier
curl -X POST 'http://localhost:4000/budget/new' \
  -d '{"budget_id": "standard-tier", "rpm_limit": 300, "tpm_limit": 100000, "max_budget": 500}'

# Assign to key
curl -X POST 'http://localhost:4000/key/generate' \
  -d '{"budget_id": "standard-tier"}'

# Upgrade key to premium tier — no key rotation required
curl -X POST 'http://localhost:4000/key/update' \
  -d '{"key": "sk-abc123", "budget_id": "premium-tier"}'

4.4 LiteLLM vs. Provider-Side Enforcement (Double Enforcement)

LiteLLM enforces TPM/RPM limits before the request reaches the provider. Provider limits (e.g., AWS Bedrock service quotas, OpenAI account-level TPM) fire after. Both layers are independent. At scale:

  • LiteLLM’s enforcement is in-process (Redis-backed for multi-node) — fast but subject to the stale-reservation bug documented in Issue #28146.
  • Provider limits return RateLimitError; LiteLLM’s router fallback logic handles these if configured — but see §4.5.

Known bug: Issue #28146 — LiteLLM v1.85.0: virtual keys with TPM/RPM limits inject internal _litellm_* parameters into provider API calls. OpenAI and Anthropic reject these with validation errors, breaking fallback chains. Status: open. Workaround: do not configure per-key TPM/RPM limits on keys where fallback chains must function. Use team-level limits instead.

5. Budget Alerting

5.1 Slack Alerting Configuration

# config.yaml
general_settings:
  master_key: sk-1234
  alerting: ["slack"]
  alerting_threshold: 300        # seconds — alert on slow responses beyond this
  spend_report_frequency: "1d"   # daily spend digest

alert_types:
  - budget_alerts
  - spend_reports
  - daily_reports
  - failed_tracking_spend

alerting_args:
  budget_alert_ttl: 86400        # dedup window: 1 alert per key/team per day
  report_check_interval: 3600    # background check every hour

export SLACK_WEBHOOK_URL="https://hooks.slack.com/services/T.../B.../..."

5.2 Route Alert Types to Different Channels

general_settings:
  alerting: ["slack"]
  alert_to_webhook_url:
    budget_alerts: "https://hooks.slack.com/services/.../budget-alerts-channel"
    spend_reports: "https://hooks.slack.com/services/.../finops-channel"
    daily_reports:  "https://hooks.slack.com/services/.../finops-channel"

5.3 Generic Webhook / PagerDuty

LiteLLM does not have a native PagerDuty integration. Use the generic webhook callback and route to PagerDuty via a Webhook→PagerDuty bridge (e.g., AWS EventBridge, Zapier, or a thin Lambda):

general_settings:
  alerting: ["webhook"]

environment_variables:
  WEBHOOK_URL: "https://events.pagerduty.com/integration/<key>/enqueue"

Webhook payload events include: budget_crossed, threshold_crossed, projected_limit_exceeded.

5.4 Team Soft Budget Email

Configured directly on the team object (see §2.3). Requires SendGrid, Resend, or SMTP integration. No additional config.yaml entry needed once email is wired.

6. Prometheus Budget Metrics

6.1 Enable Prometheus

litellm_settings:
  callbacks: ["prometheus"]
  success_callback: ["prometheus"]
  failure_callback: ["prometheus"]
  prometheus_initialize_budget_metrics: true   # cron every 5 min for all keys/teams

Multi-worker deployments require:

export PROMETHEUS_MULTIPROC_DIR="/tmp/prometheus_multiproc"

Metrics endpoint: http://localhost:4000/metrics

6.2 Budget Metric Inventory

Metric Labels Description
litellm_remaining_team_budget_metric team, team_alias Remaining USD budget for team
litellm_team_max_budget_metric team, team_alias Configured max budget
litellm_team_budget_remaining_hours_metric team, team_alias Hours until budget window resets
litellm_remaining_api_key_budget_metric hashed_api_key, api_key_alias Remaining USD for virtual key
litellm_api_key_max_budget_metric hashed_api_key, api_key_alias Key max budget
litellm_api_key_budget_remaining_hours_metric hashed_api_key, api_key_alias Hours until key window resets
litellm_spend_metric end_user, hashed_api_key, api_key_alias, model, team, team_alias, user Running spend counter

6.3 Prometheus Alert Rules (PromQL)

# prometheus-alerts.yaml
groups:
  - name: litellm_budget
    rules:

      # Team at or above 80% of budget
      - alert: LiteLLMTeamBudgetWarning
        expr: |
          (litellm_team_max_budget_metric - litellm_remaining_team_budget_metric)
          / litellm_team_max_budget_metric > 0.80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Team {{ $labels.team_alias }} at {{ $value | humanizePercentage }} of budget"

      # Team budget exhausted
      - alert: LiteLLMTeamBudgetExhausted
        expr: litellm_remaining_team_budget_metric <= 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Team {{ $labels.team_alias }} budget exhausted"

      # Key at or above 90% of budget
      - alert: LiteLLMKeyBudgetWarning
        expr: |
          (litellm_api_key_max_budget_metric - litellm_remaining_api_key_budget_metric)
          / litellm_api_key_max_budget_metric > 0.90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "API key {{ $labels.api_key_alias }} near budget limit"

      # Budget window expiring within 6 hours with spend > 50%
      - alert: LiteLLMTeamBudgetWindowExpiringSoon
        expr: |
          litellm_team_budget_remaining_hours_metric < 6
          and
          (litellm_team_max_budget_metric - litellm_remaining_team_budget_metric)
          / litellm_team_max_budget_metric > 0.50
        labels:
          severity: warning
        annotations:
          summary: "Team {{ $labels.team_alias }} window expires in {{ $value }}h with >50% spent"

7. Budget Governance at Scale (50+ Keys / 10+ Teams)

7.1 Governance Architecture Pattern

For Fortune 500 deployments, the recommended hierarchy is:

Cost Center / BU          →  LiteLLM Team  (shared $budget, shared TPM/RPM)
  Product/Service          →  Tag           (cross-cutting cost tracking)
    Application Instance   →  Virtual Key   (fine-grained key per deployment)
      End User             →  user= param   (tenant-level cap via max_end_user_budget)

Tags are additive to teams: a key can belong to team “data-science” AND carry tag “project-ragbot”. If either budget is exhausted, requests block.

7.2 Tag-Based Cross-Cutting Budgets

# Create tag budget for a project spanning multiple teams
curl -X POST 'http://0.0.0.0:4000/tag/new' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{
    "name": "project-ragbot",
    "max_budget": 1000.0,
    "budget_duration": "30d"
  }'

# Assign tag to keys across teams
curl -X POST 'http://0.0.0.0:4000/key/generate' \
  -d '{"team_id": "...", "tags": ["project-ragbot", "cost-center-engineering"]}'

7.3 Terraform / IaC Provisioning

There is no official LiteLLM Terraform provider. Multiple community providers exist on the Terraform Registry:

Provider Maintainer Last updated
ncecere/litellm Independent, tracks LiteLLM versions Jan 2026 rewrite
vCra/litellm Independent Feb 2026 (v1.2.0)
scalepad/litellm Independent 2025
gzamboni/litellm Independent 2025

None are officially endorsed by BerriAI. For enterprise IaC, treat these providers as thin API wrappers and validate against LiteLLM’s /key/info and /team/info after apply. Consider writing your own provider using the REST API directly if long-term stability is required.

Example (ncecere/litellm pattern):

terraform {
  required_providers {
    litellm = {
      source  = "ncecere/litellm"
      version = "~> 1.86"
    }
  }
}

resource "litellm_team" "data_science" {
  team_alias      = "data-science"
  max_budget      = 2000.0
  budget_duration = "30d"
  tpm_limit       = 5000000
  rpm_limit       = 3000
}

resource "litellm_key" "ds_prod" {
  key_alias       = "ds-prod-app"
  team_id         = litellm_team.data_science.id
  max_budget      = 500.0
  budget_duration = "30d"
  tpm_limit       = 100000
  rpm_limit       = 300
}

7.4 API-Driven Bulk Key Provisioning

There is no native bulk /key/generate endpoint (single key per call). For 50+ keys, script a loop against the REST API:

import httpx, asyncio

PROXY_URL = "http://localhost:4000"
MASTER_KEY = "sk-1234"

async def provision_keys(team_id: str, count: int, config: dict):
    async with httpx.AsyncClient() as client:
        tasks = [
            client.post(
                f"{PROXY_URL}/key/generate",
                headers={"Authorization": f"Bearer {MASTER_KEY}"},
                json={"team_id": team_id, **config}
            )
            for i in range(count)
        ]
        results = await asyncio.gather(*tasks)
    return [r.json()["key"] for r in results]

# Provision 20 keys for a team
asyncio.run(provision_keys(
    team_id="de35b29e-...",
    count=20,
    config={
        "max_budget": 100.0,
        "budget_duration": "30d",
        "tpm_limit": 50000,
        "tags": ["batch-provisioned"]
    }
))

7.5 Key Rotation Without Interruption (Enterprise)

LiteLLM Enterprise supports zero-downtime rotation via a grace period:

curl -X POST 'http://localhost:4000/key/sk-abc123/regenerate' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{"duration": "30d", "grace_period": "24h"}'

During the grace period, both old and new keys are valid. Clients can rotate without a hard cutover window. Grace period values use the same duration string format as budget_duration.

For open-source deployments, key rotation requires deleting and re-creating the key, with a brief period where both keys are valid if you provision the new key before deleting the old. Issue #14877 (PR, merged) adds scheduled rotation support.

Note: Issue #19445 documents a mismatch between LiteLLM auto-rotation and AWS Secrets Manager when both are active simultaneously — pick one rotation mechanism.

8. Pre-Flight Cost Estimation

8.1 Token Counting Before API Call

from litellm import token_counter

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": prompt_text}
]
input_tokens = token_counter(model="gpt-4o", messages=messages)
print(f"Estimated input tokens: {input_tokens}")

Uses model-specific tokenizers; falls back to tiktoken for unknown models.

8.2 Cost Per Token (Lookup)

from litellm import cost_per_token

prompt_cost, completion_cost = cost_per_token(
    model="anthropic/claude-3-5-sonnet-20241022",
    prompt_tokens=1000,
    completion_tokens=500
)
print(f"Estimated cost: ${prompt_cost + completion_cost:.6f}")

Pricing data is fetched live from api.litellm.ai and can be overridden with custom pricing (see §8.4).

8.3 Pre-Flight Cost Gate Pattern

from litellm import token_counter, cost_per_token

def preflight_cost_check(model: str, messages: list, budget_remaining: float) -> bool:
    """Returns False if estimated cost exceeds remaining budget."""
    input_tokens = token_counter(model=model, messages=messages)
    # Estimate output tokens (e.g., 25% of context window or known typical output)
    estimated_output = min(input_tokens // 4, 2000)
    input_cost, output_cost = cost_per_token(
        model=model,
        prompt_tokens=input_tokens,
        completion_tokens=estimated_output
    )
    estimated_total = input_cost + output_cost
    return estimated_total <= budget_remaining

8.4 Post-Call Cost Extraction

from litellm import completion, completion_cost

response = completion(model="gpt-4o", messages=messages)
cost = completion_cost(completion_response=response)
print(f"Actual call cost: ${float(cost):.8f}")

8.5 Model Cost Map

from litellm import model_cost

# Returns dict: model_name → {max_tokens, input_cost_per_token, output_cost_per_token, ...}
sonnet_pricing = model_cost.get("claude-3-5-sonnet-20241022", {})

The LiteLLM proxy also exposes GET /model/info which returns cost data alongside model capabilities.

9. Multi-Model Cost Routing

9.1 Provider Budget Routing (Route Away from Exhausted Providers)

When a provider’s daily/monthly budget is consumed, LiteLLM skips it and routes to the next available provider. This is the primary mechanism for “degrade to cheaper model when budget exhausted”:

# config.yaml
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

  - model_name: gpt-4o
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: gpt-4o
    litellm_params:
      model: anthropic/claude-3-haiku-20240307
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  provider_budget_config:
    openai:
      budget_limit: 200
      time_period: 1d
    anthropic:
      budget_limit: 500
      time_period: 1d
  redis_host: os.environ/REDIS_HOST
  redis_port: os.environ/REDIS_PORT
  redis_password: os.environ/REDIS_PASSWORD

general_settings:
  master_key: sk-1234

When OpenAI’s $200/day is consumed, requests to gpt-4o automatically route to Anthropic equivalents. Check /provider/budgets to inspect remaining budget and reset times.

9.2 Cost-Based Routing Strategy

The Router selects the cheapest healthy deployment among a model group:

router_settings:
  routing_strategy: "cost-based-routing"

from litellm import Router

router = Router(
    model_list=[
        {
            "model_name": "smart-model",
            "litellm_params": {
                "model": "openai/gpt-4o",
                "input_cost_per_token": 0.0000025,
                "output_cost_per_token": 0.00001
            }
        },
        {
            "model_name": "smart-model",
            "litellm_params": {
                "model": "anthropic/claude-3-5-sonnet-20241022",
                "input_cost_per_token": 0.000003,
                "output_cost_per_token": 0.000015
            }
        }
    ],
    routing_strategy="cost-based-routing"
)

response = await router.acompletion(
    model="smart-model",
    messages=[{"role": "user", "content": "..."}]
)

Limitation: Cost-based routing picks the cheapest deployment within a model group. It does not dynamically degrade based on remaining key/team budget. For budget-triggered degradation, combine with provider budget routing or application-layer logic.

9.3 Budget-Triggered Model Degradation (Application Pattern)

LiteLLM does not natively route “use Haiku when team is at 80% of budget.” This requires application logic:

import httpx

def get_model_for_team(team_id: str, base_model: str) -> str:
    """Returns cheaper fallback if team budget is >80% consumed."""
    r = httpx.get(
        f"http://localhost:4000/team/info?team_id={team_id}",
        headers={"Authorization": "Bearer sk-1234"}
    ).json()
    
    spend = r["team_info"]["spend"]
    max_budget = r["team_info"]["max_budget"]
    
    if max_budget and (spend / max_budget) > 0.80:
        return FALLBACK_MODELS.get(base_model, base_model)
    return base_model

FALLBACK_MODELS = {
    "claude-3-5-sonnet-20241022": "claude-3-haiku-20240307",
    "gpt-4o": "gpt-4o-mini"
}

Alternatively, use the dynamic_rate_limiter_v3 callback with priority_reservation to guarantee capacity for prod traffic and starve dev traffic when the model is saturated (§4, Dynamic Rate Limits).

9.4 model_group_alias for Cost-Tiered Pools

model_group_alias maps a single alias to a different internal model group, allowing clients to use a stable name while routing to different cost tiers based on configuration:

model_list:
  - model_name: premium
    litellm_params:
      model: openai/gpt-4o

  - model_name: standard
    litellm_params:
      model: openai/gpt-4o-mini

router_settings:
  model_group_alias:
    "default": "standard"      # Default pool for most keys
    "premium-tier": "premium"  # Alias for high-priority keys

Keys are then generated with {"model": "default"} or {"model": "premium-tier"}, enabling tier assignment without exposing model names to consumers. The hidden: true flag on an alias entry excludes it from the public /model/list endpoint.

10. Known Bugs Summary (as of June 2026)

Issue Severity Description Status
#27735 High Stale Redis reservations trigger false BudgetExceededError on virtual keys Open
#28146 High TPM/RPM-limited keys inject _litellm_* params into provider calls, breaking fallbacks (OpenAI, Anthropic) Open — affects v1.85.0
#25495 Medium Organization budget_duration does not reset spend after window expires Closed (verify fix)
#10052 Medium Key-level model_max_budget not enforced; TPM limits fire ~50% late in DB mode Closed (PRs #10965, #10497)
#28235 Low Multiple concurrent budget windows (budget_limits array) — feature request / partial impl Open

11. Sources