AI Cost Incident Response and Token Waste Auditing Playbook (2026)

Enterprise AI deployments in 2026 operate at a cost surface that has no equivalent in prior software generations. A misrouted agent loop, an uncompressed chat history, or a single compromised API key can generate $50K–$200K in unexpected spend before a human reviews a dashboard. The operational discipline required is closer to incident response for a runaway database query than to standard software cost monitoring — because the cost is a function of computational behavior that can compound within a single request chain.

This playbook covers the full lifecycle: classification, triage, containment, systematic audit, root cause taxonomy, post-incident review, and proactive prevention. It assumes a deployment using AWS Bedrock as the primary inference layer, LiteLLM as the proxy/gateway, and Athena over S3-based cost and usage data. Patterns translate directly to Azure OpenAI + API Management + Log Analytics and Google Cloud Vertex AI + Cloud Billing Export + BigQuery — the queries and runbook steps map across providers with surface-level syntax changes.

---

Section 1 — AI Cost Incident Classification

Severity Tiers

P3 — Anomaly (Investigate within 48 hours)

• Trigger: >20% week-over-week spend increase in any AI service account or >15% deviation from 30-day rolling baseline
• Characteristics: gradual, may be legitimate (new feature traffic, usage growth), no immediate financial exposure
• Response: engineer reviews on next business day; no escalation required; update cost forecast
• Decision: if confirmed legitimate growth, adjust budget alert thresholds; if unexplained, escalate to P2

P2 — Spike (Investigate same day, within 4 hours)

• Trigger: >50% week-over-week spike, OR AWS Budget alert fires, OR cost anomaly detection flags a root cause
• Characteristics: sharp inflection point visible on spend graph; often correlated with a specific deployment, feature flag, or time window
• Response: on-call engineer triages within 4 hours; management notified; optional soft containment (rate limit specific key or team)
• Decision: identify root cause before EOD; determine if ongoing or one-time; consider containment even if cause is known

P1 — Runaway (Immediate response, page on-call now)

• Trigger: exponential growth in spend curve, OR >$10,000 unexpected spend in any 24-hour window, OR LiteLLM spend log shows single key consuming >5x its 7-day average in under 2 hours
• Characteristics: non-linear cost growth; often an agent loop, compromised key, or recursive tool call; may compound every few minutes
• Response: contain first, investigate second; hard stop on offending resource within 15 minutes; engineering manager and finance notified
• Decision: surgical kill or full rollback — see Section 3

The Three Confounders

Model price changes masquerading as spikes. AWS, Anthropic, and OpenAI periodically reprice models. If a model tier dropped 30% but you migrated some traffic to a higher tier in the same billing period, the net spend may spike even if volume dropped. Check: divide spend by total tokens consumed. If cost-per-million-tokens changed, this is a price event, not a volume event. Not an incident — update your per-token cost baseline.

Genuine usage growth vs runaway loops. Legitimate growth produces a monotonic spend curve that tracks known traffic metrics (API requests, active users, jobs completed). Runaway loops produce a curve that grows faster than any business metric can explain — often exponential with a visible knee in the curve. Check: plot spend/request ratio over time. If cost-per-request is growing (not just request count), the problem is within requests, not across them.

Regional mis-routing causing apparent duplication. A misconfigured load balancer or Bedrock endpoint can route requests to an unexpected region, causing what looks like doubled spend. Both copies of the traffic complete — end users see no errors. Check: split spend by region in Cost Explorer. If two regions show mirrored traffic patterns starting at the same timestamp, this is a routing incident, not an AI cost incident.

Quick Triage Decision Tree

Is cost curve exponential (doubles every N hours)?
  YES → P1 immediately. Go to containment (Section 3) before investigating.
  NO  → Is spike >50% WoW or did a budget alert fire?
          YES → P2. Start 30-minute runbook (Section 2).
          NO  → Is spike >20% WoW?
                  YES → P3. Schedule investigation within 48h.
                  NO  → Not an incident. Update baseline if needed.

---

Section 2 — First 30 Minutes Runbook for a Cost Spike

This runbook assumes P2. For P1, go to containment (Section 3) first, then return here for root cause.

Step 1 — Read the Cost Anomaly Detection Alert (Minutes 0–5)

Navigate to AWS Cost Management → Cost Anomaly Detection → Alert details.

Extract:

• Service: Bedrock vs SageMaker vs Lambda (proxy layer) vs S3 (log storage spike can be a proxy for inference)
• Region: us-east-1 vs eu-west-1 vs ap-northeast-1
• Date range of anomaly: when did it start? when did it peak?
• Estimated impact: dollar amount flagged
• Linked usage type: AmazonBedrock:InputTokens, AmazonBedrock:OutputTokens, AmazonBedrock:OnDemandInferenceUnits

If the alert is not specific (e.g., flagged at account level), drill into Cost Explorer: Group by → Usage Type, filter to the anomaly date range, sort by cost descending.

Step 2 — Athena Query: Top-10 Usage Types by Cost in the Anomaly Window (Minutes 5–10)

This query runs against your AWS Cost and Usage Report (CUR) partitioned by month in S3.

-- Replace 'your_cur_database' and 'your_cur_table' with actual CUR table name
-- Replace anomaly window dates
SELECT
    line_item_usage_type,
    line_item_product_code,
    line_item_resource_id,
    product_region,
    SUM(line_item_unblended_cost)      AS total_cost,
    SUM(line_item_usage_amount)        AS total_usage_amount,
    COUNT(*)                           AS record_count,
    MIN(line_item_usage_start_date)    AS first_seen,
    MAX(line_item_usage_start_date)    AS last_seen
FROM your_cur_database.your_cur_table
WHERE line_item_usage_start_date >= TIMESTAMP '2026-06-16 00:00:00'
  AND line_item_usage_start_date <  TIMESTAMP '2026-06-17 23:59:59'
  AND line_item_product_code = 'AmazonBedrock'
  AND line_item_line_item_type = 'Usage'
GROUP BY 1, 2, 3, 4
ORDER BY total_cost DESC
LIMIT 10;

What you are looking for:

• A single line_item_resource_id (ARN) consuming disproportionate cost → targeted kill candidate
• InputTokens vs OutputTokens ratio: if input tokens dominate, context bloat is likely; if output tokens dominate, agent loops generating verbose output are likely
• Unexpected region appearing that was not in yesterday’s baseline

Step 3 — LiteLLM SpendLogs: Identify the Team/Key Driving the Spike (Minutes 10–15)

LiteLLM exposes a Postgres-backed spend log. Query it directly or via the /spend/logs API endpoint.

-- LiteLLM spend_logs table — adjust schema for your version
SELECT
    api_key,
    team_id,
    user_id,
    model,
    DATE_TRUNC('hour', startTime)          AS hour_bucket,
    SUM(spend)                             AS total_spend,
    SUM(total_tokens)                      AS total_tokens,
    SUM(prompt_tokens)                     AS prompt_tokens,
    SUM(completion_tokens)                 AS completion_tokens,
    COUNT(*)                               AS request_count,
    ROUND(SUM(spend) / NULLIF(COUNT(*),0), 6) AS avg_spend_per_request,
    ROUND(SUM(prompt_tokens) / NULLIF(COUNT(*),0), 0) AS avg_prompt_tokens
FROM spend_logs
WHERE startTime >= NOW() - INTERVAL '48 hours'
GROUP BY 1, 2, 3, 4, 5
ORDER BY total_spend DESC
LIMIT 20;

Flag any row where:

• avg_spend_per_request is >5x the baseline for that model (indicates context bloat or long output chains)
• avg_prompt_tokens is >10x normal (context window accumulation)
• A single api_key accounts for >40% of total spend in the window

Cross-reference the flagged api_key against your key registry to determine which application, team, or external integration owns it.

Step 4 — CloudWatch: InvocationLatency and InvocationThrottles (Minutes 15–20)

# Check Bedrock invocation metrics in the anomaly window
aws cloudwatch get-metric-statistics \
  --namespace AWS/Bedrock \
  --metric-name InvocationLatency \
  --dimensions Name=ModelId,Value=anthropic.claude-3-5-sonnet-20241022-v2:0 \
  --start-time 2026-06-16T00:00:00Z \
  --end-time 2026-06-17T00:00:00Z \
  --period 3600 \
  --statistics Average Maximum \
  --region us-east-1

# Check throttles — throttle spikes + high latency = retry storms amplifying cost
aws cloudwatch get-metric-statistics \
  --namespace AWS/Bedrock \
  --metric-name InvocationThrottles \
  --dimensions Name=ModelId,Value=anthropic.claude-3-5-sonnet-20241022-v2:0 \
  --start-time 2026-06-16T00:00:00Z \
  --end-time 2026-06-17T00:00:00Z \
  --period 3600 \
  --statistics Sum \
  --region us-east-1

If throttles spike coincident with the cost spike: the application is retrying throttled requests with exponential backoff that never backs off far enough, creating a retry storm. Each retry is a billable request. Throttle the client, not just the Bedrock service.

If latency spikes without throttles: long-running requests (large context, large output). Look at p99 latency — if p99 is 10x p50, a small number of requests are consuming most of the cost.

Step 5 — CloudTrail: New Workload Deployments in the Window (Minutes 20–25)

# Find Lambda function creates/updates and ECS task deployments in the anomaly window
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=CreateFunction20150331 \
  --start-time 2026-06-15T00:00:00Z \
  --end-time 2026-06-17T00:00:00Z \
  --region us-east-1 \
  --query 'Events[*].{Time:EventTime,User:Username,Resource:Resources[0].ResourceName}' \
  --output table

# Also check UpdateFunctionCode, CreateService (ECS), RunTask
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=UpdateFunctionCode20150331v2 \
  --start-time 2026-06-15T00:00:00Z \
  --end-time 2026-06-17T00:00:00Z \
  --region us-east-1 \
  --query 'Events[*].{Time:EventTime,User:Username,Resource:Resources[0].ResourceName}' \
  --output table

If a new Lambda or ECS service was deployed within 2 hours before the cost curve inflected, that deployment is the primary suspect. Pull its code and look for: missing token limits, hardcoded claude-3-opus where claude-3-haiku was intended, unbounded chat history appends, missing max_tokens parameter.

Step 6 — Ongoing vs One-Time Assessment (Minutes 25–30)

Pull the last 7 days of hourly spend:

SELECT
    DATE_TRUNC('hour', line_item_usage_start_date) AS hour_bucket,
    SUM(line_item_unblended_cost) AS hourly_cost
FROM your_cur_database.your_cur_table
WHERE line_item_usage_start_date >= NOW() - INTERVAL '7 days'
  AND line_item_product_code = 'AmazonBedrock'
GROUP BY 1
ORDER BY 1;

Plot mentally or export to a spreadsheet. Answer: is this a one-time spike that has already subsided, or is spend still elevated now?

• Subsided: root cause was likely a one-time batch job, a test run, or a completed deployment. Document the cause, add an alert, no containment needed.
• Still elevated: containment is required regardless of root cause. Proceed to Section 3.

---

Section 3 — Containment Actions

Containment Principles

Contain first when: spend is still growing, root cause is unknown, or the pattern is non-linear. Do not wait for full root cause identification before containing a P1.

Prefer surgical over broad: kill the offending key/role/service, not the entire platform. A surgical kill takes 5–10 minutes longer to execute but avoids downstream production outages that generate their own incident.

Action A — LiteLLM Virtual Key Budget Suspension (Immediate, Soft Block)

For a key identified in Step 3:

# Suspend a virtual key immediately via LiteLLM admin API
curl -X POST "https://your-litellm-proxy.internal/key/update" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "key": "sk-identified-offending-key",
    "max_budget": 0.01,
    "budget_reset_at": null,
    "metadata": {"suspended_by": "incident-response", "incident": "P2-2026-06-17", "suspended_at": "2026-06-17T10:00:00Z"}
  }'

# Alternatively, block the key entirely
curl -X POST "https://your-litellm-proxy.internal/key/block" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"key": "sk-identified-offending-key"}'

Effect: immediate. The key will receive BudgetExceededError on next request. Applications using this key fail loudly (not silently) — you want them to fail loudly so operators know containment is active. Does not affect other keys. Does not touch IAM or AWS-level policies.

Action B — IAM Deny Policy on Bedrock for Specific Role (Hard Stop, ~5-Minute Propagation)

Use when: the LiteLLM layer is bypassed, the spend is coming directly from an AWS role (e.g., an ECS task role or Lambda execution role), or you need a guarantee the traffic stops even if LiteLLM is unavailable.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "EmergencyDenyBedrock",
      "Effect": "Deny",
      "Action": [
        "bedrock:InvokeModel",
        "bedrock:InvokeModelWithResponseStream"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:PrincipalArn": "arn:aws:iam::123456789012:role/offending-service-role"
        }
      }
    }
  ]
}

# Attach the deny policy to the offending role
aws iam put-role-policy \
  --role-name offending-service-role \
  --policy-name EmergencyBedrockDeny \
  --policy-document file://emergency-bedrock-deny.json

# Verify attachment
aws iam get-role-policy \
  --role-name offending-service-role \
  --policy-name EmergencyBedrockDeny

IAM policy evaluation: Deny always wins over Allow. Even if the role has an allow policy for Bedrock, this deny will block it. Propagation is typically under 60 seconds but can take up to 5 minutes in large deployments. Keep checking CloudWatch InvocationThrottles — when it drops to zero, the deny is active.

Action C — AWS Budget Action (Automated Hard Stop for Future Incidents)

This is a prevention measure, but configure it during the incident while it is top of mind.

# Create a budget action that applies an IAM policy when spend threshold is hit
aws budgets create-budget-action \
  --account-id 123456789012 \
  --budget-name "Bedrock-Monthly-Emergency-Cap" \
  --notification-type ACTUAL \
  --action-type APPLY_IAM_POLICY \
  --action-threshold '{"ActionThresholdValue": 5000, "ActionThresholdType": "ABSOLUTE_VALUE"}' \
  --definition '{
    "IamActionDefinition": {
      "PolicyArn": "arn:aws:iam::123456789012:policy/BedrockEmergencyDenyAll",
      "Roles": ["offending-service-role", "batch-inference-role"]
    }
  }' \
  --execution-role-arn "arn:aws:iam::123456789012:role/BudgetActionExecutionRole" \
  --approval-model AUTOMATIC \
  --subscribers '[{"SubscriptionType": "EMAIL", "Address": "platform-oncall@yourcompany.com"}]'

Action D — Cost-Based Circuit Breaker in Application Code

This pattern belongs in every AI-calling service. When spend for the current hour exceeds a threshold, the circuit opens and requests fail fast with a clear error (not a Bedrock 429).

import time
import threading
from dataclasses import dataclass, field
from typing import Optional
import litellm

@dataclass
class AICostCircuitBreaker:
    hourly_budget_usd: float = 50.0
    cooldown_minutes: int = 30
    _spend_this_hour: float = field(default=0.0, init=False)
    _hour_start: float = field(default_factory=time.time, init=False)
    _tripped: bool = field(default=False, init=False)
    _trip_time: Optional[float] = field(default=None, init=False)
    _lock: threading.Lock = field(default_factory=threading.Lock, init=False)

    def record_spend(self, cost_usd: float) -> None:
        with self._lock:
            now = time.time()
            # Reset hourly counter
            if now - self._hour_start > 3600:
                self._spend_this_hour = 0.0
                self._hour_start = now
            self._spend_this_hour += cost_usd
            if self._spend_this_hour >= self.hourly_budget_usd and not self._tripped:
                self._tripped = True
                self._trip_time = now
                # Alert platform team here — emit metric, send Slack message, etc.

    def is_open(self) -> bool:
        with self._lock:
            if not self._tripped:
                return False
            # Auto-reset after cooldown (allows gradual recovery)
            if time.time() - self._trip_time > self.cooldown_minutes * 60:
                self._tripped = False
                self._spend_this_hour = 0.0
                return False
            return True

    def call_with_guard(self, **litellm_kwargs) -> dict:
        if self.is_open():
            raise RuntimeError(
                f"AI cost circuit breaker open — hourly budget of "
                f"${self.hourly_budget_usd} exceeded. Retry after "
                f"{self.cooldown_minutes} minutes."
            )
        response = litellm.completion(**litellm_kwargs)
        cost = litellm.completion_cost(completion_response=response)
        self.record_spend(cost)
        return response

# Usage
breaker = AICostCircuitBreaker(hourly_budget_usd=50.0)
try:
    result = breaker.call_with_guard(
        model="anthropic/claude-3-5-haiku-20241022",
        messages=[{"role": "user", "content": user_input}],
        max_tokens=1024
    )
except RuntimeError as e:
    return {"error": str(e), "fallback": "cost_limit_exceeded"}

Rollback vs Surgical Kill Decision

Condition	Action
Single key or role identified, production unaffected	Surgical: LiteLLM key suspension or IAM deny on specific role
Root cause unknown, spend still growing	Broad: suspend all non-critical LiteLLM keys (keep prod-critical tag active)
Deployment-correlated spike, code change suspected	Rollback: revert the deployment, then investigate
Compromised key (external actor suspected)	Hard stop: revoke key in LiteLLM + AWS Secrets Manager rotation, then investigate
Agent loop runaway (recursive pattern)	Surgical: kill the agent service specifically; agent loops don’t affect non-agent traffic

---

Section 4 — Token Waste Audit

Run this audit on any deployment that has been live for more than 30 days without a dedicated cost review. Each audit dimension produces a score (waste rate) and a priority recommendation.

Audit A — Model Tier Audit

What percentage of requests use expensive models when cheaper models would suffice?

-- LiteLLM spend_logs — model tier distribution
SELECT
    model,
    COUNT(*)                                       AS request_count,
    ROUND(100.0 * COUNT(*) / SUM(COUNT(*)) OVER(), 2) AS pct_of_requests,
    SUM(spend)                                     AS total_spend,
    ROUND(100.0 * SUM(spend) / SUM(SUM(spend)) OVER(), 2) AS pct_of_spend,
    ROUND(AVG(total_tokens), 0)                    AS avg_tokens_per_request,
    ROUND(AVG(completion_tokens), 0)               AS avg_output_tokens,
    SUM(spend) / NULLIF(SUM(total_tokens), 0) * 1e6  AS cost_per_million_tokens
FROM spend_logs
WHERE startTime >= NOW() - INTERVAL '30 days'
GROUP BY 1
ORDER BY total_spend DESC;

Interpretation:

• If >30% of requests use claude-3-5-sonnet or claude-3-opus for tasks with avg_output_tokens < 200, these are likely classification, routing, or simple extraction tasks — move them to Haiku
• Haiku is approximately 20–25x cheaper than Sonnet on a per-token basis (2026 pricing); moving 30% of Sonnet requests to Haiku can reduce total spend by 25–30% with no quality impact for simple tasks
• Segment by user_id or team_id to identify which teams are misconfigured

Waste threshold: If >40% of spend is on a model that handles tasks with avg output < 300 tokens, flag for immediate routing review.

Audit B — Context Window Audit

Is context accumulating across turns?

-- p50, p95, p99 input token counts
SELECT
    model,
    APPROX_PERCENTILE(prompt_tokens, 0.50)   AS p50_input_tokens,
    APPROX_PERCENTILE(prompt_tokens, 0.95)   AS p95_input_tokens,
    APPROX_PERCENTILE(prompt_tokens, 0.99)   AS p99_input_tokens,
    MAX(prompt_tokens)                        AS max_input_tokens,
    AVG(prompt_tokens)                        AS avg_input_tokens,
    -- Context bloat ratio: how much larger is p99 vs p50?
    ROUND(
        APPROX_PERCENTILE(prompt_tokens, 0.99) * 1.0 /
        NULLIF(APPROX_PERCENTILE(prompt_tokens, 0.50), 0),
    1) AS p99_to_p50_ratio
FROM spend_logs
WHERE startTime >= NOW() - INTERVAL '7 days'
GROUP BY 1
ORDER BY p95_input_tokens DESC;

Interpretation:

• p99_to_p50_ratio > 10 indicates a long-tail of very large requests that are likely accumulating chat history
• Track p95_input_tokens week over week; a rising trend (not a spike) is chronic context bloat
• A chat application without sliding window context management will eventually have p50 > 50K tokens as users maintain long sessions

Detection for accumulation pattern:

-- Find sessions (grouped by user) with growing input token counts over time
SELECT
    user_id,
    DATE(startTime)                            AS day,
    AVG(prompt_tokens)                         AS avg_input_tokens,
    MAX(prompt_tokens)                         AS max_input_tokens,
    COUNT(*)                                   AS request_count
FROM spend_logs
WHERE startTime >= NOW() - INTERVAL '14 days'
  AND user_id IS NOT NULL
GROUP BY 1, 2
ORDER BY user_id, day;
-- If avg_input_tokens for a user grows monotonically across days, they have no context pruning

Fix: Implement sliding window (keep last N turns), summarization (compress old context), or explicit context reset. Each strategy has latency and quality tradeoffs; summarization preserves quality but adds one LLM call per reset.

Audit C — Cache Hit Rate Audit

What fraction of prompts are cacheable but not cached?

-- Requires cache_hit field in spend_logs (LiteLLM populates this when caching is enabled)
SELECT
    model,
    DATE_TRUNC('day', startTime)               AS day,
    COUNT(*)                                   AS total_requests,
    SUM(CASE WHEN cache_hit = true THEN 1 ELSE 0 END) AS cache_hits,
    ROUND(100.0 * SUM(CASE WHEN cache_hit = true THEN 1 ELSE 0 END) / COUNT(*), 2) AS cache_hit_rate_pct,
    SUM(CASE WHEN cache_hit = true THEN spend ELSE 0 END) AS spend_on_cache_hits,
    -- Anthropic prompt caching: cache hits cost ~10% of full input token price
    -- Uncached equivalent cost of those hits:
    SUM(CASE WHEN cache_hit = true THEN prompt_tokens ELSE 0 END) * 0.000003 AS uncached_equivalent_cost,
    -- Savings from cache (90% discount on cache hit input tokens)
    SUM(CASE WHEN cache_hit = true THEN prompt_tokens ELSE 0 END) * 0.000003 * 0.90 AS estimated_cache_savings
FROM spend_logs
WHERE startTime >= NOW() - INTERVAL '30 days'
GROUP BY 1, 2
ORDER BY 1, 2;

Interpretation:

• Cache hit rate below 30% for an application with repetitive system prompts suggests prompt caching is not configured
• Anthropic prompt caching requires cache_control breakpoints in the system prompt and is not automatic; it must be explicitly enabled per request
• Cache savings compound: a 10K-token system prompt used in 1M requests/month saves ~$27K/month at Sonnet pricing when cached vs uncached
• A falling cache hit rate (trending down week over week) may indicate system prompt changes that invalidate the cache — log prompt versions

Audit D — Retry Rate Audit

What fraction of requests are retries indicating upstream failures?

-- Assumes a retry_count or is_retry field; if not present, infer from request patterns
-- Option 1: explicit retry field
SELECT
    model,
    SUM(CASE WHEN retry_count > 0 THEN 1 ELSE 0 END)     AS retried_requests,
    COUNT(*)                                               AS total_requests,
    ROUND(100.0 * SUM(CASE WHEN retry_count > 0 THEN 1 ELSE 0 END) / COUNT(*), 2) AS retry_rate_pct,
    SUM(CASE WHEN retry_count > 0 THEN spend ELSE 0 END)  AS spend_on_retries
FROM spend_logs
WHERE startTime >= NOW() - INTERVAL '7 days'
GROUP BY 1;

-- Option 2: infer from duplicate (user_id, conversation_id) + similar prompt_tokens within 60s
SELECT
    model,
    COUNT(*) AS suspected_retry_pairs,
    SUM(spend) AS spend_on_suspected_retries
FROM (
    SELECT
        s1.id,
        s1.model,
        s1.spend,
        s1.user_id
    FROM spend_logs s1
    INNER JOIN spend_logs s2
        ON s1.user_id = s2.user_id
        AND s1.model = s2.model
        AND ABS(s1.prompt_tokens - s2.prompt_tokens) < 50  -- near-identical input
        AND s1.startTime > s2.startTime
        AND s1.startTime - s2.startTime < INTERVAL '60 seconds'
        AND s1.id != s2.id
    WHERE s1.startTime >= NOW() - INTERVAL '7 days'
) retry_candidates
GROUP BY 1;

Interpretation:

• Retry rate >5% indicates instability in the LLM call path — throttles, timeouts, or upstream errors
• Retries that complete successfully are still full-cost billable events; they double (or triple) your cost for those requests
• Retries that fail also burn tokens if the model started generating before the failure
• Fix: add jitter to exponential backoff, set max_retries=2 not max_retries=10, implement fallback model routing in LiteLLM for specific error codes

Audit E — Batch-Eligible Workload Audit

What fraction of inference is synchronous but delay-tolerant?

-- Identify high-volume, low-latency-requirement request patterns
-- Proxy: requests that succeed with p95 latency > 5s are likely not user-facing
SELECT
    model,
    user_id,
    team_id,
    COUNT(*)                                              AS request_count,
    APPROX_PERCENTILE(
        EXTRACT(EPOCH FROM (endTime - startTime)), 0.95
    )                                                     AS p95_latency_seconds,
    AVG(prompt_tokens)                                    AS avg_input_tokens,
    AVG(completion_tokens)                                AS avg_output_tokens,
    SUM(spend)                                            AS total_spend
FROM spend_logs
WHERE startTime >= NOW() - INTERVAL '30 days'
  AND total_tokens > 500  -- filter out trivial requests
GROUP BY 1, 2, 3
HAVING COUNT(*) > 100  -- only workloads with meaningful volume
ORDER BY total_spend DESC;

Interpretation:

• Workloads with p95_latency_seconds > 10 and high avg_input_tokens are strong batch candidates
• AWS Bedrock Batch Inference and Anthropic Batch API offer 50% cost reduction for non-real-time workloads
• Common batch-eligible patterns: document summarization, nightly report generation, offline classification, embedding generation
• Move batch-eligible workloads from on-demand to batch: $X/month in Sonnet on-demand becomes $X/2 in Sonnet batch with 24-hour SLA

---

Section 5 — Cost Anomaly Root Cause Taxonomy

Pattern A — New Feature Launch Without Cost Estimate

Signature: Cost inflects sharply on a specific date, correlated with a CloudTrail deployment event. New user_id or team_id appears in top spenders. Spend curve is linear (not exponential) from launch.

Diagnostic: Cross-reference CloudTrail deployment events with LiteLLM spend by team/key. Calculate implied monthly run rate from first 48 hours of traffic.

Resolution: Retroactive cost estimate; if run rate exceeds budget, add rate limiting or model downgrade for the new feature. Update pre-launch checklist to require cost estimate sign-off.

Pattern B — Agent Loop Runaway

Signature: Exponential spend curve. A single user_id or session_id consuming thousands of requests in minutes. High completion_tokens relative to prompt_tokens (agent generating verbose tool calls or reasoning). Often correlated with a new agent deployment or a code change to tool execution logic.

Diagnostic:

SELECT user_id, session_id, COUNT(*) AS requests, SUM(spend) AS spend,
       MIN(startTime) AS first_request, MAX(startTime) AS last_request,
       EXTRACT(EPOCH FROM (MAX(startTime) - MIN(startTime))) AS duration_seconds
FROM spend_logs
WHERE startTime >= NOW() - INTERVAL '2 hours'
GROUP BY 1, 2
HAVING COUNT(*) > 50
ORDER BY requests DESC;

Resolution: Kill the agent session (LiteLLM key suspension or terminate the ECS task). Add loop detection in agent code: track tool call count per session, enforce a hard limit (e.g., 50 tool calls per session), fail gracefully.

Pattern C — Model Version Upgrade Changing Per-Token Pricing

Signature: No spike in request count or token count. Spend increases proportionally to a price ratio (e.g., 1.5x if migrated to a 1.5x more expensive model). Correlated with a deployment or config change that updated model IDs.

Diagnostic: Compare cost_per_million_tokens before and after the inflection date (from Audit A query). If it changed without corresponding quality improvement or token efficiency, this is a pricing artifact.

Resolution: Not a cost incident per se. Update cost forecast. If the model upgrade was unintended, revert model config. If intentional, communicate the price change to budget owners.

Pattern D — Context Window Bloat from Chat History Accumulation

Signature: Gradual spend increase over days/weeks without request count growth. p50_input_tokens rising week over week (from Audit B). Concentrated in a chat or conversation-type workload.

Diagnostic: Run Audit B across a 30-day window and plot avg_input_tokens over time. If it grows monotonically by 5–10% per week, context accumulation is the cause.

Resolution: Implement sliding window context (keep last N turns), add summarization after N turns, or enforce session TTL that clears context.

Pattern E — Test/Dev Traffic in Production Account

Signature: High prompt_tokens with unusual patterns (repetitive test phrases, “Hello world” variations, lorem ipsum, developer usernames). Often from a single IAM role or API key associated with a developer.

Diagnostic:

SELECT user_id, LEFT(messages, 100) AS prompt_sample, COUNT(*), SUM(spend)
FROM spend_logs
WHERE startTime >= NOW() - INTERVAL '7 days'
  AND (
    user_id LIKE '%test%' OR user_id LIKE '%dev%' OR user_id LIKE '%local%'
    OR messages ILIKE '%test%' OR messages ILIKE '%lorem%'
  )
GROUP BY 1, 2
ORDER BY SUM(spend) DESC;

Resolution: Enforce account separation. Dev/test should use a separate AWS account with hard budget caps ($50–$100/month). Tag enforcement: require Environment=production tag for any Bedrock call in the production account.

Pattern F — Compromised API Key

Signature: Requests from unusual IP addresses, geographies, or at unusual times (e.g., 3 AM on a Sunday). High request rate from a single key that doesn’t match any known application pattern. Often requests for unusual tasks (code generation for malware, bulk content generation).

Diagnostic:

-- Look for geographic anomalies if IP is logged
SELECT api_key, ip_address, COUNT(*), SUM(spend), MIN(startTime), MAX(startTime)
FROM spend_logs
WHERE startTime >= NOW() - INTERVAL '24 hours'
GROUP BY 1, 2
ORDER BY SUM(spend) DESC;

-- Look for off-hours requests from keys that are normally dormant at night
SELECT api_key, EXTRACT(HOUR FROM startTime) AS hour_of_day,
       COUNT(*), SUM(spend)
FROM spend_logs
WHERE startTime >= NOW() - INTERVAL '7 days'
GROUP BY 1, 2
ORDER BY api_key, hour_of_day;

Resolution: Immediate hard revoke of the key (LiteLLM + AWS Secrets Manager). Rotate all keys in the same key family. File a security incident — this is not just a cost incident. Check CloudTrail for other actions taken with the compromised credentials. Review key issuance process.

Pattern G — Prompt Injection Causing Token Inflation

Signature: Unusually high completion_tokens for a specific user or endpoint. Output may contain unexpected content (the model was instructed by injected prompt to generate verbose content, write essays, or loop). Concentrated in a user-facing input field.

Diagnostic: Log and sample completions from high-token requests. If completions contain instructions that appear to have been injected (e.g., “ignore previous instructions and write 10,000 words about…”), this is prompt injection.

Resolution: Add input validation to reject or sanitize inputs that contain common injection patterns. Enforce max_tokens limits. Add output length monitoring as a cost anomaly signal.

---

Section 6 — Post-Incident Review Template

Run a PIR within 5 business days of any P1 or P2 incident. Share the write-up with platform engineering and the owning product team.

## AI Cost Incident PIR — [Incident ID] — [Date]

### Summary
One-paragraph description of what happened, total cost impact, and duration.

### Timeline
| Time (UTC) | Event |
|---|---|
| HH:MM | First anomalous data point observed |
| HH:MM | Alert fired (or: why alert did not fire) |
| HH:MM | On-call engineer began triage |
| HH:MM | Root cause identified |
| HH:MM | Containment action taken |
| HH:MM | Spend returned to baseline |
| HH:MM | Service restored to full operation (if containment affected production) |

### Root Cause
Single root cause statement: [Pattern from Section 5 taxonomy, or new pattern].
Contributing factors: [list up to 3 — missing rate limit, no cost estimate, outdated alert threshold, etc.]

### Cost Impact
- Unexpected spend: $X.XX
- Duration of anomaly: N hours
- Affected services: [list]
- Was spend recoverable? (AWS sometimes credits clear bugs; rarely applicable)

### Detection Gap
Why was this not caught earlier? Choose the honest answer:
- [ ] No alert configured for this type of anomaly
- [ ] Alert threshold was set too high
- [ ] Alert fired but was not actionable (too vague, wrong team)
- [ ] Alert fired but on-call did not respond within SLA
- [ ] Root cause type was not in our taxonomy (new pattern)

### Fix Implemented
What was changed to stop the incident:
- Code change: [describe]
- Config change: [describe]
- Policy change: [describe]

### Prevention Measures Added
| Action | Owner | Due Date | Type |
|---|---|---|---|
| Add cost estimate gate to deployment checklist | Platform eng | [date] | Process |
| Set LiteLLM key budget for [team] | [team] | [date] | Config |
| Add loop detection to [agent] | [team] | [date] | Code |
| Lower anomaly detection threshold to $X | FinOps | [date] | Alerting |
| Add batch-eligible tag to [workload] | [team] | [date] | Architecture |

### Architectural Change vs Alerting Change Assessment
For each prevention measure, classify:
- **Alerting change** — catches the same incident faster next time. Does not prevent the incident.
- **Architectural change** — makes the incident structurally impossible (or orders of magnitude less likely).

Prioritize architectural changes. Alerting changes are a supplement, not a substitute.

### What We Would Do Differently
Free-form: what would have made this incident cheaper (in time and money) to handle?

---

Section 7 — Proactive Waste Prevention

Weekly Token Waste Scan (Cron Job)

Schedule this to run every Monday at 06:00 UTC. Output is posted to a #ai-cost-weekly Slack channel.

#!/usr/bin/env python3
"""Weekly AI cost waste scan — runs via cron, posts Slack digest."""

import os
import json
import boto3
import psycopg2
import requests
from datetime import datetime, timedelta, timezone

SLACK_WEBHOOK = os.environ["SLACK_WEEKLY_WEBHOOK"]
LITELLM_DB_URL = os.environ["LITELLM_DATABASE_URL"]
ATHENA_DATABASE = os.environ["ATHENA_CUR_DATABASE"]
ATHENA_TABLE = os.environ["ATHENA_CUR_TABLE"]
ATHENA_OUTPUT = os.environ["ATHENA_QUERY_OUTPUT_S3"]

def run_litellm_query(sql: str) -> list[dict]:
    conn = psycopg2.connect(LITELLM_DB_URL)
    cur = conn.cursor()
    cur.execute(sql)
    cols = [d[0] for d in cur.description]
    rows = [dict(zip(cols, row)) for row in cur.fetchall()]
    cur.close()
    conn.close()
    return rows

def model_tier_waste() -> dict:
    rows = run_litellm_query("""
        SELECT model,
               COUNT(*) AS requests,
               SUM(spend) AS spend,
               AVG(completion_tokens) AS avg_output_tokens
        FROM spend_logs
        WHERE startTime >= NOW() - INTERVAL '7 days'
        GROUP BY model ORDER BY spend DESC
    """)
    total_spend = sum(r["spend"] for r in rows)
    expensive_models = {"claude-3-5-sonnet", "claude-3-opus", "gpt-4o", "gpt-4-turbo"}
    expensive_spend = sum(r["spend"] for r in rows if any(m in r["model"] for m in expensive_models))
    expensive_low_output = sum(
        r["spend"] for r in rows
        if any(m in r["model"] for m in expensive_models) and (r["avg_output_tokens"] or 0) < 200
    )
    return {
        "expensive_pct": round(100 * expensive_spend / max(total_spend, 0.01), 1),
        "potentially_downgradeable_pct": round(100 * expensive_low_output / max(total_spend, 0.01), 1),
        "potential_savings_usd": round(expensive_low_output * 0.93, 2),  # ~93% savings moving to Haiku
    }

def context_bloat_check() -> dict:
    rows = run_litellm_query("""
        SELECT
            APPROX_PERCENTILE(prompt_tokens, 0.95) AS p95_input,
            APPROX_PERCENTILE(prompt_tokens, 0.50) AS p50_input,
            AVG(prompt_tokens) AS avg_input
        FROM spend_logs
        WHERE startTime >= NOW() - INTERVAL '7 days'
    """)
    r = rows[0] if rows else {}
    rows_prev = run_litellm_query("""
        SELECT APPROX_PERCENTILE(prompt_tokens, 0.95) AS p95_input
        FROM spend_logs
        WHERE startTime >= NOW() - INTERVAL '14 days'
          AND startTime < NOW() - INTERVAL '7 days'
    """)
    prev_p95 = rows_prev[0].get("p95_input", r.get("p95_input", 1)) if rows_prev else 1
    current_p95 = r.get("p95_input", 0)
    return {
        "p95_input_tokens": int(current_p95 or 0),
        "p50_input_tokens": int(r.get("p50_input") or 0),
        "wow_p95_change_pct": round(100 * (current_p95 - prev_p95) / max(prev_p95, 1), 1),
    }

def cache_hit_rate() -> dict:
    rows = run_litellm_query("""
        SELECT
            ROUND(100.0 * SUM(CASE WHEN cache_hit THEN 1 ELSE 0 END) / COUNT(*), 2) AS hit_rate,
            SUM(CASE WHEN cache_hit THEN spend ELSE 0 END) AS cached_spend,
            SUM(spend) AS total_spend
        FROM spend_logs
        WHERE startTime >= NOW() - INTERVAL '7 days'
    """)
    r = rows[0] if rows else {}
    return {
        "cache_hit_rate_pct": float(r.get("hit_rate") or 0),
        "cached_spend_usd": float(r.get("cached_spend") or 0),
        "total_spend_usd": float(r.get("total_spend") or 0),
    }

def format_slack_message(tier: dict, bloat: dict, cache: dict) -> dict:
    week = datetime.now(timezone.utc).strftime("%Y-%m-%d")
    warnings = []
    if tier["potentially_downgradeable_pct"] > 20:
        warnings.append(f":warning: {tier['potentially_downgradeable_pct']}% of spend is on expensive models with low output — potential ${tier['potential_savings_usd']:.0f}/wk savings by routing to Haiku")
    if bloat["wow_p95_change_pct"] > 10:
        warnings.append(f":warning: p95 input tokens grew {bloat['wow_p95_change_pct']}% WoW — context accumulation likely")
    if cache["cache_hit_rate_pct"] < 25:
        warnings.append(f":warning: Cache hit rate {cache['cache_hit_rate_pct']}% — prompt caching may not be configured")
    status = ":white_check_mark: No anomalies detected" if not warnings else "\n".join(warnings)
    return {
        "text": f"*AI Cost Weekly Scan — {week}*\n\n"
                f"• Model tier waste: {tier['potentially_downgradeable_pct']}% of spend potentially downgradeable\n"
                f"• p95 input tokens: {bloat['p95_input_tokens']:,} (WoW: {bloat['wow_p95_change_pct']:+.1f}%)\n"
                f"• Cache hit rate: {cache['cache_hit_rate_pct']}%\n"
                f"• Total spend (7d): ${cache['total_spend_usd']:.2f}\n\n"
                f"{status}"
    }

def main():
    tier = model_tier_waste()
    bloat = context_bloat_check()
    cache = cache_hit_rate()
    msg = format_slack_message(tier, bloat, cache)
    requests.post(SLACK_WEBHOOK, json=msg, timeout=10)

if __name__ == "__main__":
    main()

Add to crontab:

# Weekly AI cost scan — Mondays at 06:00 UTC
0 6 * * 1 /opt/venv/bin/python3 /opt/scripts/weekly_ai_cost_scan.py >> /var/log/ai_cost_scan.log 2>&1

Athena Weekly Anomalous Pattern Query

Run via AWS EventBridge + Lambda for teams without a dedicated data warehouse.

-- Weekly anomalous request patterns: find usage_types growing faster than account baseline
WITH weekly_by_type AS (
    SELECT
        line_item_usage_type,
        DATE_TRUNC('week', line_item_usage_start_date) AS week,
        SUM(line_item_unblended_cost) AS weekly_cost
    FROM your_cur_database.your_cur_table
    WHERE line_item_usage_start_date >= CURRENT_DATE - INTERVAL '8' WEEK
      AND line_item_product_code = 'AmazonBedrock'
    GROUP BY 1, 2
),
with_lag AS (
    SELECT *,
        LAG(weekly_cost, 1) OVER (PARTITION BY line_item_usage_type ORDER BY week) AS prior_week_cost,
        LAG(weekly_cost, 4) OVER (PARTITION BY line_item_usage_type ORDER BY week) AS four_week_ago_cost
    FROM weekly_by_type
),
growth_rates AS (
    SELECT *,
        ROUND(100.0 * (weekly_cost - prior_week_cost) / NULLIF(prior_week_cost, 0), 1) AS wow_growth_pct,
        ROUND(100.0 * (weekly_cost - four_week_ago_cost) / NULLIF(four_week_ago_cost, 0), 1) AS mom_growth_pct
    FROM with_lag
    WHERE week = DATE_TRUNC('week', CURRENT_DATE - INTERVAL '1' WEEK)
)
SELECT line_item_usage_type, weekly_cost, prior_week_cost,
       wow_growth_pct, mom_growth_pct
FROM growth_rates
WHERE ABS(wow_growth_pct) > 20
   OR ABS(mom_growth_pct) > 50
ORDER BY ABS(wow_growth_pct) DESC;

Model Tier Distribution Shift Detection

Week-over-week model mix change that might indicate misconfigured routing:

WITH this_week AS (
    SELECT model, SUM(spend) AS spend
    FROM spend_logs
    WHERE startTime >= NOW() - INTERVAL '7 days'
    GROUP BY 1
),
last_week AS (
    SELECT model, SUM(spend) AS spend
    FROM spend_logs
    WHERE startTime >= NOW() - INTERVAL '14 days'
      AND startTime < NOW() - INTERVAL '7 days'
    GROUP BY 1
)
SELECT
    COALESCE(t.model, l.model) AS model,
    COALESCE(t.spend, 0) AS this_week_spend,
    COALESCE(l.spend, 0) AS last_week_spend,
    ROUND(100.0 * COALESCE(t.spend, 0) /
          NULLIF(SUM(COALESCE(t.spend, 0)) OVER(), 0), 2) AS this_week_pct,
    ROUND(100.0 * COALESCE(l.spend, 0) /
          NULLIF(SUM(COALESCE(l.spend, 0)) OVER(), 0), 2) AS last_week_pct,
    ROUND(
        100.0 * COALESCE(t.spend, 0) / NULLIF(SUM(COALESCE(t.spend, 0)) OVER(), 0)
        - 100.0 * COALESCE(l.spend, 0) / NULLIF(SUM(COALESCE(l.spend, 0)) OVER(), 0)
    , 2) AS share_point_change
FROM this_week t
FULL OUTER JOIN last_week l USING (model)
ORDER BY ABS(share_point_change) DESC;

Flag if any model gains or loses >10 share points week over week without a known reason.

---

Appendix — Alert Thresholds Reference

Alert	Threshold	Severity	Action
Bedrock daily spend	>$500 (or >120% of 7-day avg)	P3	Review next business day
Bedrock hourly spend	>$200 in any single hour	P2	Same-day triage
Single LiteLLM key	>40% of total hourly spend	P2	Check key ownership
p95 input tokens	>50K for any model	P2	Context audit
p95 input tokens WoW	>20% growth	P3	Context audit scheduled
Cache hit rate	<20% for workload with >1000 req/day	P3	Caching configuration review
Retry rate	>10%	P2	Stability investigation
Agent session requests	>100 requests in 5 minutes	P1	Immediate loop investigation
Off-hours spend (10PM–6AM)	>200% of prior-week same-window	P2	Compromised key check

---

Synthesis current as of June 2026. Pricing figures reflect Anthropic and AWS public pricing as of Q2 2026. Query syntax tested against PostgreSQL 15 (LiteLLM) and Athena 3.x (AWS CUR). Adjust table names and column aliases for your deployment.