← Agent Frameworks 🕐 11 min read

Agent Frameworks

Bedrock Cost Anomaly Detection and Automated Remediation (2026)

1. AWS Cost Anomaly Detection for Bedrock · March 2026

AWS Cost Anomaly Detection (CAD) uses a multi-layered ML model trained on each account's historical spend patterns. It is not a static threshold monitor. The model learns seasonality (weekday vs.

Audience: Platform engineers and AI infrastructure teams managing AWS Bedrock at enterprise scale.

Problem statement: A Bedrock agent loop left running overnight can consume $50K+ in a single incident. AWS Cost Anomaly Detection fires 24–48 hours after the anomaly starts. This note covers the full detection stack — from sub-minute LiteLLM burn rate to day-old CUR signals — and the automated remediation layer that turns an alert into a stopped workload.

What it actually does

AWS Cost Anomaly Detection (CAD) uses a multi-layered ML model trained on each account’s historical spend patterns. It is not a static threshold monitor. The model learns seasonality (weekday vs. weekend, month-end spikes) and applies anomaly scoring against that learned baseline. An alert fires when actual spend deviates from the expected band by a configured absolute or percentage threshold.

Key distinction: CAD works on Cost and Usage Report (CUR) data, which has a 24–48 hour ingestion lag. For most AWS services, this is acceptable. For Bedrock, where a runaway agent can exhaust a monthly budget in 4 hours, it is not the right first line of defense — it is a backstop for anomalies that slipped past near-real-time detection.

Monitor types

Linked account monitor: Covers all services in a payer/linked account. Useful for detecting unexpected Bedrock charges when Bedrock is not yet a primary workload. Fires on total account spend deviation.

Service monitor: Scoped to a single AWS service. Create a dedicated AWS Bedrock service monitor to isolate Bedrock anomalies from unrelated EC2/RDS noise. This is the correct configuration for any team where Bedrock is an active cost center.

Cost category / tag monitor: Scoped to a custom dimension (e.g., CostCenter=AI-Platform). Requires that cost allocation tags are enforced upstream (see Section 8). Enables anomaly detection at the team or product level rather than the account level.

Recommended configuration

Monitor type:        AWS Service — Amazon Bedrock
Threshold type:      ABSOLUTE_VALUE + PERCENTAGE (use ABSOLUTE to avoid alert fatigue on small bases)
Alert threshold:     $200 absolute OR 50% above baseline (whichever fires first)
Notification:        SNS topic → Lambda → Slack + PagerDuty
Frequency:          DAILY (IMMEDIATE is not available for service monitors)

The IMMEDIATE notification frequency is only available on linked-account monitors. Service monitors send at most one alert per day. This further reinforces that CAD is a lagging signal.

SNS → Slack/PagerDuty routing

{
  "AnomalyMonitorName": "BedrockServiceMonitor",
  "AnomalySubscriptionName": "BedrockAnomalyP2Alert",
  "MonitorArnList": ["arn:aws:ce::123456789012:anomalymonitor/MONITOR_ID"],
  "Subscribers": [
    {
      "Address": "arn:aws:sns:us-east-1:123456789012:bedrock-cost-alerts",
      "Type": "SNS",
      "Status": "CONFIRMED"
    }
  ],
  "Threshold": 200.0,
  "Frequency": "DAILY"
}

The SNS message payload from CAD includes anomalyScore, impact.maxImpact, impact.totalImpact, rootCauses (service, region, usage type, linked account), and anomalyStartDate/anomalyEndDate. A Lambda subscriber can parse rootCauses to identify which Bedrock model or usage type drove the anomaly.

Limitations for Bedrock specifically

Limitation	Impact
24–48h CUR lag	Alert fires the morning after the incident, not during it
Daily notification frequency for service monitors	A 4-hour spike on Monday shows up Tuesday
No sub-service granularity	CAD cannot distinguish `claude-3-5-sonnet` from `nova-micro` in the anomaly root cause
No automated remediation built in	CAD stops at notification; no native action
Baseline requires 10+ days of history	New Bedrock accounts have no reliable anomaly model for weeks

Verdict: Configure CAD as a P2 backstop. Never rely on it as the primary alerting layer for Bedrock.

2. Near-Real-Time Anomaly Detection via LiteLLM

LiteLLM proxies all Bedrock calls and emits a litellm_spend_metric Prometheus counter. Because it is instrumented at the request level, it reflects spend in near-real-time — latency is the Prometheus scrape interval (typically 15–30 seconds), not 24–48 hours.

Prometheus metric structure

# HELP litellm_spend_metric Total spend in USD
# TYPE litellm_spend_metric counter
litellm_spend_metric{
  model="bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0",
  team_id="platform-eng",
  user_api_key_alias="agent-prod-01",
  call_type="acompletion"
} 142.37

Additional useful metrics:

litellm_input_tokens_metric — input token counter by model/team/key
litellm_output_tokens_metric — output token counter
litellm_requests_metric — request count, useful for loop detection
litellm_failed_requests_metric — error rate

Grafana burn rate alert

Burn rate is the rate at which spend is accelerating relative to a rolling baseline. The pattern below fires when the 5-minute spend rate is more than 5× the 1-hour rolling average — catching sudden spikes without false-positiving on gradual ramp-ups.

# Grafana alert rule (YAML provisioning format)
apiVersion: 1
groups:
  - name: BedrockSpendAnomalies
    interval: 1m
    rules:
      - uid: bedrock-burn-rate-spike
        title: "Bedrock Spend Burn Rate Anomaly"
        condition: C
        data:
          - refId: A
            queryType: ""
            relativeTimeRange:
              from: 300
              to: 0
            datasourceUid: prometheus-ds
            model:
              expr: rate(litellm_spend_metric[5m]) * 3600
              intervalMs: 60000
              maxDataPoints: 43200
          - refId: B
            queryType: ""
            relativeTimeRange:
              from: 3600
              to: 0
            datasourceUid: prometheus-ds
            model:
              expr: rate(litellm_spend_metric[1h]) * 3600
              intervalMs: 60000
              maxDataPoints: 43200
          - refId: C
            type: classic_conditions
            datasourceUid: "-100"
            model:
              conditions:
                - evaluator:
                    type: gt
                    params: [5]
                  operator:
                    type: and
                  query:
                    params: [A]
                  reducer:
                    type: last
                  type: query
        noDataState: NoData
        execErrState: Alerting
        for: 5m
        annotations:
          summary: "Bedrock spend rate is {{ $values.A.Value | printf \"%.2f\" }} USD/hr (baseline {{ $values.B.Value | printf \"%.2f\" }} USD/hr)"
        labels:
          severity: critical
          team: platform-eng

For the burn rate ratio alert (fires when current rate > N × baseline):

# Burn rate ratio — fires when 5m rate > 5x the 1h average
(
  rate(litellm_spend_metric[5m]) 
  / 
  rate(litellm_spend_metric[1h])
) > 5

Python burn rate calculation against LiteLLM SpendLogs

For teams running scheduled anomaly scans against the LiteLLM database rather than Prometheus:

"""
litellm_burn_rate.py

Queries LiteLLM SpendLogs table and computes rolling burn rate.
Fires alert if current period spend > BURN_RATE_MULTIPLIER × baseline period.

Requires: sqlalchemy, psycopg2, boto3 (for SNS alerting)
"""

import os
import json
from datetime import datetime, timedelta, timezone
from typing import Optional

import boto3
from sqlalchemy import create_engine, text

# Configuration
DB_URL = os.environ["LITELLM_DB_URL"]  # postgresql://user:pass@host/litellm
SNS_TOPIC_ARN = os.environ["ALERT_SNS_TOPIC_ARN"]
BURN_RATE_MULTIPLIER = float(os.environ.get("BURN_RATE_MULTIPLIER", "5.0"))
CURRENT_WINDOW_MINUTES = int(os.environ.get("CURRENT_WINDOW_MINUTES", "15"))
BASELINE_WINDOW_MINUTES = int(os.environ.get("BASELINE_WINDOW_MINUTES", "60"))
MIN_BASELINE_SPEND = float(os.environ.get("MIN_BASELINE_SPEND", "1.0"))  # avoid div/0 on idle


def get_spend_in_window(engine, start: datetime, end: datetime) -> float:
    """Sum spend from LiteLLM SpendLogs in the given time window."""
    query = text("""
        SELECT COALESCE(SUM(spend), 0.0) as total_spend
        FROM "LiteLLM_SpendLogs"
        WHERE startTime >= :start
          AND startTime < :end
    """)
    with engine.connect() as conn:
        result = conn.execute(query, {"start": start, "end": end})
        return float(result.scalar())


def get_spend_by_model(engine, start: datetime, end: datetime) -> list[dict]:
    """Breakdown by model for anomaly context."""
    query = text("""
        SELECT model, COALESCE(SUM(spend), 0.0) as spend, COUNT(*) as requests
        FROM "LiteLLM_SpendLogs"
        WHERE startTime >= :start
          AND startTime < :end
        GROUP BY model
        ORDER BY spend DESC
        LIMIT 10
    """)
    with engine.connect() as conn:
        result = conn.execute(query, {"start": start, "end": end})
        return [{"model": row.model, "spend": float(row.spend), "requests": int(row.requests)}
                for row in result]


def check_burn_rate_anomaly() -> Optional[dict]:
    """
    Returns anomaly dict if burn rate exceeds threshold, else None.
    """
    engine = create_engine(DB_URL)
    now = datetime.now(timezone.utc)
    
    current_start = now - timedelta(minutes=CURRENT_WINDOW_MINUTES)
    baseline_start = now - timedelta(minutes=BASELINE_WINDOW_MINUTES)
    
    current_spend = get_spend_in_window(engine, current_start, now)
    baseline_spend = get_spend_in_window(engine, baseline_start, current_start)
    
    # Normalize to per-minute rate
    current_rate = current_spend / CURRENT_WINDOW_MINUTES
    baseline_rate = baseline_spend / BASELINE_WINDOW_MINUTES
    
    if baseline_rate < (MIN_BASELINE_SPEND / BASELINE_WINDOW_MINUTES):
        # Not enough baseline activity — skip to avoid spurious alerts on cold start
        return None
    
    ratio = current_rate / baseline_rate
    
    if ratio >= BURN_RATE_MULTIPLIER:
        model_breakdown = get_spend_by_model(engine, current_start, now)
        return {
            "anomaly_type": "BURN_RATE_SPIKE",
            "current_rate_per_min": round(current_rate, 4),
            "baseline_rate_per_min": round(baseline_rate, 4),
            "ratio": round(ratio, 2),
            "current_window_spend": round(current_spend, 4),
            "current_window_minutes": CURRENT_WINDOW_MINUTES,
            "baseline_window_minutes": BASELINE_WINDOW_MINUTES,
            "model_breakdown": model_breakdown,
            "detected_at": now.isoformat(),
        }
    return None


def send_sns_alert(anomaly: dict) -> None:
    sns = boto3.client("sns")
    subject = f"[BEDROCK COST ANOMALY] Burn rate {anomaly['ratio']}x baseline"
    message = json.dumps(anomaly, indent=2)
    sns.publish(TopicArn=SNS_TOPIC_ARN, Subject=subject, Message=message)
    print(f"Alert sent: {subject}")


def lambda_handler(event, context):
    """Entry point for EventBridge-scheduled Lambda (run every 5 minutes)."""
    anomaly = check_burn_rate_anomaly()
    if anomaly:
        send_sns_alert(anomaly)
        return {"status": "ALERT_SENT", "anomaly": anomaly}
    return {"status": "OK"}


if __name__ == "__main__":
    anomaly = check_burn_rate_anomaly()
    if anomaly:
        print(json.dumps(anomaly, indent=2))
        send_sns_alert(anomaly)
    else:
        print("No anomaly detected.")

Why LiteLLM is the right first line of defense:

Dimension	LiteLLM Prometheus	AWS Cost Anomaly Detection
Detection lag	15–30 seconds (scrape interval)	24–48 hours (CUR lag)
Granularity	Per-model, per-key, per-team	Service-level, limited root cause
Remediation integration	Direct DB/API access to kill keys	Notification only
Cost	Included in LiteLLM proxy overhead	Free (no per-alert charge)
False positive tuning	Configurable multiplier	ML-controlled, limited tuning

3. CloudWatch Metric-Based Anomaly Detection

CloudWatch provides two complementary mechanisms: static threshold alarms and ML-based anomaly detection bands. For Bedrock, both are valuable at different layers.

Bedrock CloudWatch metrics (2026)

Metric	Namespace	Dimensions	Cost signal
`InputTokenCount`	`AWS/Bedrock`	ModelId, Region	Direct — input cost driver
`OutputTokenCount`	`AWS/Bedrock`	ModelId, Region	Direct — output cost driver (higher)
`InvocationThrottles`	`AWS/Bedrock`	ModelId, Region	Indirect — retry storms inflate cost
`InvocationLatency`	`AWS/Bedrock`	ModelId, Region	Indirect — high latency → long agent loops
`InvocationsPerMinute`	`AWS/Bedrock`	ModelId, Region	Volume proxy

CloudWatch Anomaly Detection alarm (JSON)

The following CloudFormation-compatible alarm JSON uses ANOMALY_DETECTION_BAND to create ML-based upper and lower bounds. It fires when OutputTokenCount exceeds the predicted band by a configurable standard deviation multiplier.

{
  "Type": "AWS::CloudWatch::Alarm",
  "Properties": {
    "AlarmName": "Bedrock-OutputTokenCount-Anomaly-claude-sonnet",
    "AlarmDescription": "Output token count exceeds ML anomaly band for claude-3-5-sonnet. Possible runaway agent or prompt injection.",
    "Metrics": [
      {
        "Id": "m1",
        "MetricStat": {
          "Metric": {
            "Namespace": "AWS/Bedrock",
            "MetricName": "OutputTokenCount",
            "Dimensions": [
              {
                "Name": "ModelId",
                "Value": "anthropic.claude-3-5-sonnet-20241022-v2:0"
              }
            ]
          },
          "Period": 300,
          "Stat": "Sum"
        },
        "ReturnData": true
      },
      {
        "Id": "ad1",
        "Expression": "ANOMALY_DETECTION_BAND(m1, 3)",
        "Label": "OutputTokenCount (expected)",
        "ReturnData": true
      }
    ],
    "ComparisonOperator": "GreaterThanUpperThreshold",
    "ThresholdMetricId": "ad1",
    "EvaluationPeriods": 2,
    "TreatMissingData": "notBreaching",
    "AlarmActions": [
      "arn:aws:sns:us-east-1:123456789012:bedrock-cost-alerts"
    ],
    "OKActions": [
      "arn:aws:sns:us-east-1:123456789012:bedrock-cost-alerts"
    ]
  }
}

The ANOMALY_DETECTION_BAND(m1, 3) expression sets the band at ±3 standard deviations from the ML-predicted baseline. A value of 2 is more sensitive (more alerts); 3 is appropriate for production cost alerting (fewer false positives).

InvocationThrottles spike detection (static threshold)

Throttle spikes indicate retry storms — the agent retries at full cost after each throttle. Use a static threshold here since anomaly detection on throttles is less meaningful.

{
  "Type": "AWS::CloudWatch::Alarm",
  "Properties": {
    "AlarmName": "Bedrock-InvocationThrottles-Spike",
    "AlarmDescription": "Throttle count exceeds 50 in 5 minutes. Possible retry storm — check agent loop configuration.",
    "Namespace": "AWS/Bedrock",
    "MetricName": "InvocationThrottles",
    "Dimensions": [
      { "Name": "ModelId", "Value": "anthropic.claude-3-5-sonnet-20241022-v2:0" }
    ],
    "Statistic": "Sum",
    "Period": 300,
    "EvaluationPeriods": 1,
    "Threshold": 50,
    "ComparisonOperator": "GreaterThanOrEqualToThreshold",
    "TreatMissingData": "notBreaching",
    "AlarmActions": [
      "arn:aws:sns:us-east-1:123456789012:bedrock-cost-alerts"
    ]
  }
}

InvocationLatency degradation as indirect cost signal

High latency on long-context models correlates with expensive completions. An alarm on P99 latency above 30 seconds is a useful early warning for unexpectedly large prompts or runaway tool-call loops.

{
  "Type": "AWS::CloudWatch::Alarm",
  "Properties": {
    "AlarmName": "Bedrock-InvocationLatency-P99-High",
    "AlarmDescription": "P99 invocation latency above 30s. Possible large context window or runaway agent turn.",
    "Namespace": "AWS/Bedrock",
    "MetricName": "InvocationLatency",
    "Dimensions": [
      { "Name": "ModelId", "Value": "anthropic.claude-3-5-sonnet-20241022-v2:0" }
    ],
    "ExtendedStatistic": "p99",
    "Period": 300,
    "EvaluationPeriods": 3,
    "Threshold": 30000,
    "ComparisonOperator": "GreaterThanThreshold",
    "TreatMissingData": "notBreaching",
    "AlarmActions": [
      "arn:aws:sns:us-east-1:123456789012:bedrock-cost-alerts"
    ]
  }
}

CloudWatch Anomaly Detection ML bands vs. static thresholds

Approach	Best for	Limitation
`ANOMALY_DETECTION_BAND`	Metrics with clear seasonality (weekday/weekend, business hours)	Requires 2 weeks of history to build model; poor on new deployments
Static threshold	Metrics with no seasonal pattern (throttle counts, hard error rates)	Requires manual recalibration as usage grows
Combination	Anomaly detection for normal-operation bounds + static for absolute floors	Higher alarm count and cost

Cost of CloudWatch alarms for Bedrock

Each CloudWatch alarm costs approximately $0.10/alarm/month in us-east-1 (2026 pricing). An anomaly detection alarm costs $0.10 + $0.30 for the anomaly detection model = $0.40/alarm/month. A complete Bedrock alarm set covering 5 models × 4 metrics = 20 alarms ≈ $6–8/month. This is negligible relative to the cost of a single missed anomaly.

4. EventBridge + Lambda Automated Remediation

Detection without remediation is a notification system. The patterns below close the loop — an anomaly alert becomes an automated action within seconds, not minutes.

Architecture overview

LiteLLM Prometheus Alert  ──┐
CloudWatch Alarm            ├──► SNS Topic (bedrock-cost-alerts)
AWS Cost Anomaly Detection  ──┘         │
                                         ▼
                                 EventBridge Rule
                                 (or SNS → Lambda direct)
                                         │
                              ┌──────────┼──────────┐
                              ▼          ▼           ▼
                         Lambda A    Lambda B    Lambda C
                       (hard stop) (TPM reduce) (circuit break)

Pattern 1: Budget breach → Lambda → LiteLLM team budget to $0 (hard stop)

"""
lambda_hard_stop.py

Triggered by AWS Budgets SNS notification when Bedrock budget is exceeded.
Sets the offending LiteLLM team's budget to $0 to halt all spending immediately.
"""

import json
import os
import urllib.request
import urllib.error

LITELLM_BASE_URL = os.environ["LITELLM_BASE_URL"]  # e.g. http://litellm.internal:4000
LITELLM_MASTER_KEY = os.environ["LITELLM_MASTER_KEY"]
DEFAULT_TEAM_ID = os.environ.get("DEFAULT_TEAM_ID", "")  # fallback team


def set_team_budget_zero(team_id: str) -> dict:
    """Set LiteLLM team max_budget to 0 (hard stop all API calls for this team)."""
    url = f"{LITELLM_BASE_URL}/team/update"
    payload = json.dumps({
        "team_id": team_id,
        "max_budget": 0.0,
        "budget_duration": None,  # no reset — stay stopped until manual intervention
        "metadata": {
            "stopped_by": "lambda_hard_stop",
            "reason": "budget_breach_auto_remediation",
        }
    }).encode()
    
    req = urllib.request.Request(
        url,
        data=payload,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {LITELLM_MASTER_KEY}",
        },
        method="PATCH",
    )
    with urllib.request.urlopen(req, timeout=10) as resp:
        return json.loads(resp.read())


def extract_team_id_from_sns(event: dict) -> str:
    """
    Parse team_id from SNS message. 
    Expects either a LiteLLM burn-rate alert (has team_id in model_breakdown)
    or a Budgets notification (use DEFAULT_TEAM_ID or tag-derived value).
    """
    try:
        message = json.loads(event["Records"][0]["Sns"]["Message"])
        # LiteLLM burn rate alert format
        if "model_breakdown" in message and message.get("model_breakdown"):
            return message.get("team_id", DEFAULT_TEAM_ID)
        # AWS Budgets format — use configured default
        return DEFAULT_TEAM_ID
    except (KeyError, json.JSONDecodeError):
        return DEFAULT_TEAM_ID


def lambda_handler(event, context):
    print(f"Received event: {json.dumps(event)}")
    team_id = extract_team_id_from_sns(event)
    
    if not team_id:
        print("ERROR: No team_id found and DEFAULT_TEAM_ID not set. Cannot remediate.")
        return {"status": "ERROR", "reason": "no_team_id"}
    
    print(f"Hard-stopping team: {team_id}")
    result = set_team_budget_zero(team_id)
    print(f"LiteLLM response: {result}")
    
    return {
        "status": "STOPPED",
        "team_id": team_id,
        "litellm_response": result,
    }

Pattern 2: Anomaly alert → Lambda → reduce TPM quota on virtual key

"""
lambda_tpm_throttle.py

Triggered by CloudWatch OutputTokenCount anomaly alarm.
Reduces TPM (tokens per minute) limit on active virtual keys for the affected model.
More surgical than a hard stop — production traffic degrades gracefully rather than halting.
"""

import json
import os
import urllib.request
from typing import Optional

LITELLM_BASE_URL = os.environ["LITELLM_BASE_URL"]
LITELLM_MASTER_KEY = os.environ["LITELLM_MASTER_KEY"]
TPM_REDUCTION_FACTOR = float(os.environ.get("TPM_REDUCTION_FACTOR", "0.25"))  # reduce to 25%


def list_active_keys(model: Optional[str] = None) -> list[dict]:
    """List virtual keys, optionally filtered by model."""
    url = f"{LITELLM_BASE_URL}/key/list?include_team_keys=true"
    req = urllib.request.Request(
        url,
        headers={"Authorization": f"Bearer {LITELLM_MASTER_KEY}"},
    )
    with urllib.request.urlopen(req, timeout=10) as resp:
        data = json.loads(resp.read())
    keys = data.get("keys", [])
    if model:
        keys = [k for k in keys if model in (k.get("models") or [])]
    return keys


def reduce_key_tpm(key_hash: str, current_tpm: int, factor: float) -> dict:
    """Reduce a key's TPM limit to factor × current_tpm."""
    new_tpm = max(int(current_tpm * factor), 1000)  # floor at 1K TPM to avoid full stop
    url = f"{LITELLM_BASE_URL}/key/update"
    payload = json.dumps({
        "key": key_hash,
        "tpm_limit": new_tpm,
        "metadata": {
            "auto_throttled": True,
            "original_tpm": current_tpm,
            "throttle_reason": "output_token_anomaly",
        }
    }).encode()
    req = urllib.request.Request(
        url,
        data=payload,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {LITELLM_MASTER_KEY}",
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=10) as resp:
        return json.loads(resp.read())


def parse_model_from_alarm(event: dict) -> Optional[str]:
    """Extract ModelId from CloudWatch alarm SNS message."""
    try:
        message = json.loads(event["Records"][0]["Sns"]["Message"])
        trigger = message.get("Trigger", {})
        dims = trigger.get("Dimensions", [])
        for dim in dims:
            if dim.get("name") == "ModelId":
                return dim.get("value")
    except (KeyError, json.JSONDecodeError):
        pass
    return None


def lambda_handler(event, context):
    print(f"Received event: {json.dumps(event)}")
    model = parse_model_from_alarm(event)
    print(f"Affected model: {model or 'all models'}")
    
    keys = list_active_keys(model=model)
    throttled = []
    
    for key in keys:
        current_tpm = key.get("tpm_limit") or 0
        if current_tpm > 0:
            key_hash = key.get("key_name") or key.get("token")
            result = reduce_key_tpm(key_hash, current_tpm, TPM_REDUCTION_FACTOR)
            throttled.append({"key": key_hash, "original_tpm": current_tpm, "result": result})
    
    print(f"Throttled {len(throttled)} keys for model {model}")
    return {"status": "THROTTLED", "model": model, "keys_affected": len(throttled)}

Pattern 3: Runaway agent loop detection → Lambda → disable API key

"""
lambda_kill_key.py

Triggered when agent-loop runaway signature is detected (see Section 5).
Disables the specific API key associated with the runaway agent.
"""

import json
import os
import urllib.request

LITELLM_BASE_URL = os.environ["LITELLM_BASE_URL"]
LITELLM_MASTER_KEY = os.environ["LITELLM_MASTER_KEY"]


def disable_key(key_hash: str, reason: str) -> dict:
    """Set key status to 'inactive' — all subsequent calls with this key return 403."""
    url = f"{LITELLM_BASE_URL}/key/update"
    payload = json.dumps({
        "key": key_hash,
        "key_alias": None,
        "blocked": True,  # LiteLLM 'blocked' flag prevents all usage
        "metadata": {
            "disabled_by": "lambda_kill_key",
            "reason": reason,
        }
    }).encode()
    req = urllib.request.Request(
        url,
        data=payload,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {LITELLM_MASTER_KEY}",
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=10) as resp:
        return json.loads(resp.read())


def lambda_handler(event, context):
    """
    Expected event format (from EventBridge custom event or SNS):
    {
        "key_hash": "sk-...",
        "reason": "agent_loop_runaway",
        "detected_at": "2026-06-18T10:23:00Z",
        "evidence": { ... }
    }
    """
    print(f"Received kill-key request: {json.dumps(event)}")
    
    # Handle both direct invocation and SNS wrapping
    if "Records" in event:
        payload = json.loads(event["Records"][0]["Sns"]["Message"])
    else:
        payload = event
    
    key_hash = payload.get("key_hash")
    reason = payload.get("reason", "automated_anomaly_remediation")
    
    if not key_hash:
        return {"status": "ERROR", "reason": "no_key_hash_in_event"}
    
    print(f"Disabling key: {key_hash}, reason: {reason}")
    result = disable_key(key_hash, reason)
    
    return {
        "status": "KEY_DISABLED",
        "key_hash": key_hash,
        "reason": reason,
        "litellm_response": result,
    }

Pattern 4: Circuit breaker — SNS → SQS → Lambda with exponential backoff before re-enabling

The circuit breaker pattern prevents the “alert → remediate → re-enable → alert” thrash cycle. After a hard stop, the system waits with exponential backoff before testing whether the condition has cleared.

"""
lambda_circuit_breaker.py

Implements a circuit breaker with states: CLOSED (normal) → OPEN (stopped) → HALF_OPEN (testing).
Uses SQS message visibility timeout as the backoff mechanism.
SSM Parameter Store holds circuit state.

Flow:
1. Anomaly detected → send message to circuit-breaker SQS queue
2. Lambda triggered → opens circuit (disables team/key), sets state in SSM
3. SQS message re-queued with visibility timeout = backoff_seconds
4. After backoff, message reappears → Lambda checks current spend rate
5. If spend rate is normal → closes circuit (re-enables), resets backoff
6. If spend rate is still anomalous → opens circuit again, doubles backoff
"""

import json
import os
import time
import math
import urllib.request
from datetime import datetime, timezone

import boto3

LITELLM_BASE_URL = os.environ["LITELLM_BASE_URL"]
LITELLM_MASTER_KEY = os.environ["LITELLM_MASTER_KEY"]
SSM_PREFIX = os.environ.get("SSM_CIRCUIT_PREFIX", "/bedrock/circuit-breaker")
SQS_QUEUE_URL = os.environ["CIRCUIT_BREAKER_QUEUE_URL"]
MAX_BACKOFF_SECONDS = int(os.environ.get("MAX_BACKOFF_SECONDS", "3600"))  # 1 hour max
BASE_BACKOFF_SECONDS = int(os.environ.get("BASE_BACKOFF_SECONDS", "60"))

ssm = boto3.client("ssm")
sqs = boto3.client("sqs")


def get_circuit_state(team_id: str) -> dict:
    param_name = f"{SSM_PREFIX}/{team_id}"
    try:
        resp = ssm.get_parameter(Name=param_name)
        return json.loads(resp["Parameter"]["Value"])
    except ssm.exceptions.ParameterNotFound:
        return {"state": "CLOSED", "attempt": 0, "opened_at": None}


def set_circuit_state(team_id: str, state: dict) -> None:
    param_name = f"{SSM_PREFIX}/{team_id}"
    ssm.put_parameter(
        Name=param_name,
        Value=json.dumps(state),
        Type="String",
        Overwrite=True,
    )


def get_current_spend_rate(team_id: str) -> float:
    """Query LiteLLM for team's current spend in last 5 minutes."""
    url = f"{LITELLM_BASE_URL}/team/info?team_id={team_id}"
    req = urllib.request.Request(
        url,
        headers={"Authorization": f"Bearer {LITELLM_MASTER_KEY}"},
    )
    with urllib.request.urlopen(req, timeout=10) as resp:
        data = json.loads(resp.read())
    # spend_5m is a custom metric — requires LiteLLM 1.40+ with spend window tracking
    return float(data.get("team_info", {}).get("spend_5m", 0.0))


def set_team_budget(team_id: str, budget: float) -> None:
    url = f"{LITELLM_BASE_URL}/team/update"
    payload = json.dumps({"team_id": team_id, "max_budget": budget}).encode()
    req = urllib.request.Request(
        url, data=payload,
        headers={"Content-Type": "application/json", "Authorization": f"Bearer {LITELLM_MASTER_KEY}"},
        method="PATCH",
    )
    urllib.request.urlopen(req, timeout=10)


def compute_backoff(attempt: int) -> int:
    """Exponential backoff: base * 2^attempt, capped at max."""
    return min(BASE_BACKOFF_SECONDS * (2 ** attempt), MAX_BACKOFF_SECONDS)


def lambda_handler(event, context):
    for record in event.get("Records", []):
        receipt_handle = record["receiptHandle"]
        message = json.loads(record["body"])
        team_id = message["team_id"]
        normal_budget = float(message.get("normal_budget", 100.0))
        
        circuit = get_circuit_state(team_id)
        state = circuit["state"]
        
        if state == "CLOSED":
            # Anomaly just detected — open the circuit
            print(f"[{team_id}] CLOSED → OPEN: stopping spend")
            set_team_budget(team_id, 0.0)
            circuit = {"state": "OPEN", "attempt": 0, "opened_at": datetime.now(timezone.utc).isoformat()}
            set_circuit_state(team_id, circuit)
            # Re-queue with initial backoff
            backoff = compute_backoff(0)
            sqs.change_message_visibility(
                QueueUrl=SQS_QUEUE_URL,
                ReceiptHandle=receipt_handle,
                VisibilityTimeout=backoff,
            )
            print(f"[{team_id}] Re-queued with {backoff}s backoff (attempt 0)")
        
        elif state == "OPEN":
            # Backoff expired — test if condition cleared (HALF_OPEN)
            attempt = circuit["attempt"] + 1
            spend_rate = get_current_spend_rate(team_id)
            threshold = message.get("spend_threshold", 0.5)  # USD per 5 minutes
            
            print(f"[{team_id}] HALF_OPEN test: spend_rate={spend_rate:.4f}, threshold={threshold}")
            
            if spend_rate <= threshold:
                # Condition cleared — close circuit and restore budget
                print(f"[{team_id}] HALF_OPEN → CLOSED: restoring budget to {normal_budget}")
                set_team_budget(team_id, normal_budget)
                circuit = {"state": "CLOSED", "attempt": 0, "opened_at": None}
                set_circuit_state(team_id, circuit)
                # Delete message — circuit is closed
                sqs.delete_message(QueueUrl=SQS_QUEUE_URL, ReceiptHandle=receipt_handle)
            else:
                # Still anomalous — keep OPEN, double the backoff
                backoff = compute_backoff(attempt)
                print(f"[{team_id}] Still anomalous. Re-opening with {backoff}s backoff (attempt {attempt})")
                circuit = {**circuit, "state": "OPEN", "attempt": attempt}
                set_circuit_state(team_id, circuit)
                sqs.change_message_visibility(
                    QueueUrl=SQS_QUEUE_URL,
                    ReceiptHandle=receipt_handle,
                    VisibilityTimeout=backoff,
                )
    
    return {"status": "processed"}

5. Bedrock-Specific Anomaly Patterns

These signatures are the result of real-world Bedrock incidents. Each pattern includes the token-level fingerprint used to distinguish it from legitimate traffic changes.

Pattern A: Agent loop runaway

What happens: An agent calls tools repeatedly (e.g., a search tool that returns results that trigger more searches) without reaching a stop condition. Each iteration adds input tokens (the growing context) and output tokens (the reasoning + tool call).

Token signature:

OutputTokenCount: rising monotonically over rolling 15-minute window
InputTokenCount: also rising (context accumulates each turn)
InvocationsPerMinute: stable or rising — regular cadence from the loop
InvocationLatency: may decrease per call as context grows and generations get shorter (the model trying to resolve the loop)

Detection query (Prometheus):

# Rising output token rate over 15m window — flag if slope is positive and steep
deriv(litellm_output_tokens_metric[15m]) > 1000

Remediation: Kill the specific API key associated with the agent (lambda_kill_key.py).

Pattern B: Prompt injection attack

What happens: External data ingested by an agent contains injected instructions causing the model to produce unexpectedly long completions or call unintended tools.

Token signature:

OutputTokenCount: sudden spike on a specific key_hash and model
InputTokenCount: relatively stable (the injected payload may be small)
Spike is isolated to one key, not across the team
Timing correlates with ingestion of external data source

Detection query:

# Per-key output token spike relative to that key's own baseline
(
  rate(litellm_output_tokens_metric{user_api_key_alias="agent-prod-01"}[5m])
  / 
  rate(litellm_output_tokens_metric{user_api_key_alias="agent-prod-01"}[1h] offset 5m)
) > 10

Remediation: Disable key + alert security team. This is a security incident, not just a cost incident.

Pattern C: Credential leak / unauthorized use

What happens: A Bedrock API key or IAM role credential is leaked (e.g., committed to a public repo). External parties begin calling Bedrock using the compromised credential.

Token signature in CUR:

Usage from unexpected lineItem/ResourceId or IAM principal (visible in CUR identity/PrincipalId)
user:agent tag absent (legitimate internal callers set it; external callers do not)
Geographic anomaly in product/region (calls from regions not in approved list)

Detection: Athena CUR query (see Section 6, query C).

Remediation: Immediate IAM key rotation + SCPs (see Section 8).

Pattern D: Cache miss storm

What happens: Prompt caching is disabled, TTL is set too short, or a deploy changes cache key structure — causing all requests that were previously cache hits to become full model calls.

Token signature:

litellm_input_tokens_metric rises sharply (cache reads returned no tokens; now full input is processed)
LiteLLM cache_read_input_tokens metric drops to near zero
LiteLLM cache_creation_input_tokens spikes (every request writes a new cache entry)
Cost per request returns to full rate instead of cache-read rate (~10% of full)

Detection query:

# Cache hit ratio collapses — flag when cache_read drops below 20% of prior ratio
(
  rate(litellm_cache_read_input_tokens_metric[5m])
  /
  (rate(litellm_cache_read_input_tokens_metric[5m]) + rate(litellm_input_tokens_metric[5m]))
) < 0.2

Remediation: Roll back the deploy that changed cache configuration. Budget impact can be 5–10× until resolved.

Pattern E: Model routing misconfiguration

What happens: A configuration change routes requests intended for nova-micro ($0.035/$0.14 per million tokens) to claude-3-5-sonnet ($3/$15 per million tokens) — a 20–100× cost multiplier.

Token signature:

litellm_spend_metric for claude-3-5-sonnet spikes while nova-micro drops to zero
Request count stays flat (same traffic, wrong model)
Cost-per-request jumps 10–100× depending on which models are swapped

Detection query:

# Model usage distribution shifts — flag when a model that had near-zero share suddenly has >50%
(
  rate(litellm_requests_metric{model="bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0"}[5m])
  /
  rate(litellm_requests_metric[5m])
) > 0.5
# ...while that model had < 5% share previously

Remediation: Revert LiteLLM router configuration. This is a deploy-triggered incident — add model distribution alerting to the deploy pipeline.

6. Athena Anomaly Detection Queries

These queries run against the AWS Cost and Usage Report (CUR) Athena table. Replace your_cur_database.your_cur_table with your actual CUR table name. All queries assume CUR 2.0 column names.

Query A: Day-over-day spend spike (>3× rolling 7-day average)

-- Detect Bedrock service days where spend is >3x the preceding 7-day rolling average.
-- Run daily as a scheduled Athena query; results feed a CloudWatch custom metric or SNS alert.
WITH daily_spend AS (
  SELECT
    DATE(line_item_usage_start_date) AS usage_date,
    SUM(line_item_unblended_cost)    AS daily_cost
  FROM your_cur_database.your_cur_table
  WHERE product_service_code = 'AmazonBedrock'
    AND line_item_line_item_type IN ('Usage', 'SavingsPlanCoveredUsage')
    AND line_item_usage_start_date >= DATE_ADD('day', -30, CURRENT_DATE)
  GROUP BY DATE(line_item_usage_start_date)
),
rolling_avg AS (
  SELECT
    usage_date,
    daily_cost,
    AVG(daily_cost) OVER (
      ORDER BY usage_date
      ROWS BETWEEN 7 PRECEDING AND 1 PRECEDING
    ) AS rolling_7d_avg
  FROM daily_spend
)
SELECT
  usage_date,
  ROUND(daily_cost, 2)        AS daily_cost_usd,
  ROUND(rolling_7d_avg, 2)    AS rolling_7d_avg_usd,
  ROUND(daily_cost / NULLIF(rolling_7d_avg, 0), 2) AS spend_ratio
FROM rolling_avg
WHERE daily_cost > 3.0 * NULLIF(rolling_7d_avg, 0)
  AND rolling_7d_avg IS NOT NULL
ORDER BY usage_date DESC;

Query B: Unexpected model appearing in usage

-- Find model IDs that appeared in the last 7 days but were absent in the prior 30 days.
-- Catches model routing misconfigurations (Pattern E) that persist into CUR.
WITH recent_models AS (
  SELECT DISTINCT
    resource_tags_user_model_id AS model_id
  FROM your_cur_database.your_cur_table
  WHERE product_service_code = 'AmazonBedrock'
    AND line_item_usage_start_date >= DATE_ADD('day', -7, CURRENT_DATE)
    AND resource_tags_user_model_id IS NOT NULL
),
historical_models AS (
  SELECT DISTINCT
    resource_tags_user_model_id AS model_id
  FROM your_cur_database.your_cur_table
  WHERE product_service_code = 'AmazonBedrock'
    AND line_item_usage_start_date >= DATE_ADD('day', -37, CURRENT_DATE)
    AND line_item_usage_start_date < DATE_ADD('day', -7, CURRENT_DATE)
    AND resource_tags_user_model_id IS NOT NULL
)
SELECT
  r.model_id,
  'NEW_MODEL_FIRST_SEEN_IN_LAST_7_DAYS' AS anomaly_type
FROM recent_models r
LEFT JOIN historical_models h ON r.model_id = h.model_id
WHERE h.model_id IS NULL
ORDER BY r.model_id;

Query C: IAM principal with first-time Bedrock usage

-- Identify IAM principals that used Bedrock in the last 24 hours for the first time ever.
-- High-priority signal for credential leak (Pattern C) or unauthorized access.
WITH recent_principals AS (
  SELECT DISTINCT
    identity_principal_id AS principal_id,
    MIN(line_item_usage_start_date) AS first_seen_in_recent
  FROM your_cur_database.your_cur_table
  WHERE product_service_code = 'AmazonBedrock'
    AND line_item_usage_start_date >= DATE_ADD('day', -1, CURRENT_DATE)
  GROUP BY identity_principal_id
),
historical_principals AS (
  SELECT DISTINCT
    identity_principal_id AS principal_id
  FROM your_cur_database.your_cur_table
  WHERE product_service_code = 'AmazonBedrock'
    AND line_item_usage_start_date < DATE_ADD('day', -1, CURRENT_DATE)
)
SELECT
  r.principal_id,
  r.first_seen_in_recent,
  'FIRST_TIME_BEDROCK_PRINCIPAL' AS anomaly_type
FROM recent_principals r
LEFT JOIN historical_principals h ON r.principal_id = h.principal_id
WHERE h.principal_id IS NULL
ORDER BY r.first_seen_in_recent DESC;

Query D: Sudden cache hit rate collapse

-- Compare cache hit ratio (cache_read tokens / total input tokens) across days.
-- A drop >50% relative to the prior 7-day average signals a cache miss storm (Pattern D).
-- Requires CUR usage type fields to distinguish cache read vs. standard invocation.
WITH daily_cache AS (
  SELECT
    DATE(line_item_usage_start_date)                                    AS usage_date,
    SUM(CASE WHEN line_item_usage_type LIKE '%CacheRead%'
             THEN line_item_usage_amount ELSE 0 END)                    AS cache_read_tokens,
    SUM(CASE WHEN line_item_usage_type NOT LIKE '%Cache%'
             THEN line_item_usage_amount ELSE 0 END)                    AS standard_tokens,
    SUM(line_item_usage_amount)                                         AS total_tokens
  FROM your_cur_database.your_cur_table
  WHERE product_service_code = 'AmazonBedrock'
    AND line_item_usage_start_date >= DATE_ADD('day', -14, CURRENT_DATE)
  GROUP BY DATE(line_item_usage_start_date)
),
cache_ratio AS (
  SELECT
    usage_date,
    ROUND(cache_read_tokens * 100.0 / NULLIF(total_tokens, 0), 1) AS cache_hit_pct,
    AVG(cache_read_tokens * 100.0 / NULLIF(total_tokens, 0)) OVER (
      ORDER BY usage_date
      ROWS BETWEEN 7 PRECEDING AND 1 PRECEDING
    ) AS rolling_7d_cache_hit_pct
  FROM daily_cache
)
SELECT
  usage_date,
  cache_hit_pct,
  ROUND(rolling_7d_cache_hit_pct, 1)      AS baseline_cache_hit_pct,
  ROUND((rolling_7d_cache_hit_pct - cache_hit_pct) / NULLIF(rolling_7d_cache_hit_pct, 0) * 100, 1) AS pct_drop
FROM cache_ratio
WHERE cache_hit_pct < rolling_7d_cache_hit_pct * 0.5  -- >50% relative drop
  AND rolling_7d_cache_hit_pct IS NOT NULL
ORDER BY usage_date DESC;

7. Cost Anomaly Playbook Integration

Anomaly alerts must map to incident severity classifications to ensure the right response cadence and ownership. The following table connects the detection signals above to a P1/P2/P3 framework compatible with standard on-call runbooks.

Severity classification

Severity	Trigger condition	Response time	Owner
P1	Burn rate >10× baseline OR projected daily spend >$10K OR credential leak signal	30 minutes	On-call platform engineer + FinOps lead
P2	Burn rate 5–10× baseline OR unexpected model detected OR agent loop runaway	2 hours	Platform engineering team
P3	AWS CAD alert (24-48h lag) OR cache hit rate collapse without active cost spike	Next business day	FinOps team

30-minute P1 runbook

T+0:00  Alert fires (LiteLLM burn rate OR CloudWatch anomaly alarm)
T+0:01  PagerDuty pages on-call engineer
T+0:05  Engineer acknowledges. Opens Grafana dashboard: litellm_spend_metric by model/key
T+0:08  Identifies anomalous key_hash or team_id from Grafana breakdown
T+0:10  Executes automated remediation (lambda_hard_stop or lambda_kill_key) OR
         manually calls: POST /key/update {"key": "<hash>", "blocked": true}
T+0:12  Confirms spend rate drops in Grafana (rate(litellm_spend_metric[1m]) → 0)
T+0:15  Opens Athena: runs Query A + Query C to assess CUR-level blast radius
T+0:20  Posts incident summary to #incidents Slack channel:
         - Detected at: T+0:00
         - Anomaly type: [pattern A/B/C/D/E]
         - Key/team stopped: [identifier]
         - Estimated impact: $X (from litellm_spend_metric delta)
         - Root cause hypothesis: [routing change / agent loop / credential]
T+0:25  Begins root cause investigation (deploy log correlation, agent trace review)
T+0:30  Posts initial RCA to incident channel. Escalates to P1 if root cause unclear.

Post-incident:
- Update LiteLLM circuit breaker message (lambda_circuit_breaker.py) with team's normal_budget
- Submit CUR-based final cost tally (run Athena Query A scoped to incident window)
- Add anomaly pattern to Grafana dashboard as annotated event
- If credential leak: rotate IAM keys, review CloudTrail for unauthorized calls

Alert routing summary

Source	SNS Topic	Lambda	Slack Channel	PagerDuty
LiteLLM burn rate (>10×)	`bedrock-cost-alerts-p1`	`lambda_hard_stop`	`#incidents`	Yes
LiteLLM burn rate (5–10×)	`bedrock-cost-alerts-p2`	`lambda_tpm_throttle`	`#platform-alerts`	No
CloudWatch OutputTokenCount anomaly	`bedrock-cost-alerts-p2`	`lambda_tpm_throttle`	`#platform-alerts`	No
CloudWatch InvocationThrottles spike	`bedrock-cost-alerts-p2`	None (informational)	`#platform-alerts`	No
Agent loop runaway (custom metric)	`bedrock-cost-alerts-p1`	`lambda_kill_key`	`#incidents`	Yes
AWS Cost Anomaly Detection	`bedrock-cost-alerts-p3`	None	`#finops`	No
Athena scheduled query (new IAM principal)	`bedrock-cost-alerts-p1`	`lambda_kill_key`	`#incidents`	Yes

8. Proactive Cost Guardrails

The cheapest anomaly is the one that never starts. The following guardrails prevent the conditions that lead to anomalies — they are infrastructure, not process.

Guardrail 1: SCP to enforce cost allocation tags on all Bedrock calls

Without cost allocation tags on every Bedrock invocation, CUR-based anomaly detection cannot attribute spend to teams or workloads. This SCP denies Bedrock API calls that lack the Team and Application tags.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyBedrockWithoutCostTags",
      "Effect": "Deny",
      "Action": [
        "bedrock:InvokeModel",
        "bedrock:InvokeModelWithResponseStream"
      ],
      "Resource": "*",
      "Condition": {
        "Null": {
          "aws:RequestTag/Team": "true"
        }
      }
    },
    {
      "Sid": "DenyBedrockWithoutApplicationTag",
      "Effect": "Deny",
      "Action": [
        "bedrock:InvokeModel",
        "bedrock:InvokeModelWithResponseStream"
      ],
      "Resource": "*",
      "Condition": {
        "Null": {
          "aws:RequestTag/Application": "true"
        }
      }
    }
  ]
}

Note: SCPs apply at the AWS Organizations level. Test in a non-production OU first. LiteLLM can inject these tags automatically via extra_headers if the IAM role has iam:TagResource — or via a Lambda@Edge policy — but the most reliable approach is to require them in the SCP and enforce tagging in the LiteLLM config.

Guardrail 2: LiteLLM `max_budget` + `budget_duration` as a circuit breaker

Configure per-team and per-key budgets in LiteLLM. When max_budget is reached, LiteLLM raises BudgetExceededError (HTTP 429) — no further calls go to Bedrock until the budget resets.

# litellm_config.yaml excerpt
general_settings:
  master_key: "sk-master-xxx"
  database_url: "postgresql://user:pass@host/litellm"

litellm_settings:
  max_budget: 500.0           # Global daily budget cap in USD
  budget_duration: "1d"       # Reset daily at 00:00 UTC
  
# Team-level budgets are set via API or UI, not config file:
# POST /team/new
# {
#   "team_alias": "data-science",
#   "max_budget": 100.0,
#   "budget_duration": "1d",
#   "tpm_limit": 500000,
#   "rpm_limit": 1000
# }

# Key-level budgets (for agent workloads):
# POST /key/generate
# {
#   "key_alias": "agent-prod-01",
#   "team_id": "data-science",
#   "max_budget": 20.0,
#   "budget_duration": "1d",
#   "tpm_limit": 100000,
#   "max_parallel_requests": 5
# }

The max_parallel_requests limit on agent keys is underused but highly effective: it caps how many concurrent agent turns can run simultaneously, directly limiting the blast radius of a runaway loop.

Guardrail 3: AWS Budgets action — IAM policy attachment when budget exceeded

When the AWS budget for Bedrock exceeds the threshold, an AWS Budgets action automatically attaches a deny policy to the relevant IAM role, blocking further Bedrock calls at the IAM layer — even if LiteLLM is bypassed.

{
  "BudgetName": "BedrockMonthlyBudget-DataScience",
  "BudgetLimit": {
    "Amount": "500",
    "Unit": "USD"
  },
  "TimeUnit": "MONTHLY",
  "BudgetType": "COST",
  "CostFilters": {
    "Service": ["Amazon Bedrock"],
    "TagKeyValue": ["user:Team$data-science"]
  },
  "NotificationsWithSubscribers": [],
  "AutoAdjustData": {
    "AutoAdjustType": "HISTORICAL",
    "HistoricalOptions": {
      "BudgetAdjustmentPeriod": 3
    }
  }
}

The corresponding Budgets Action:

{
  "BudgetName": "BedrockMonthlyBudget-DataScience",
  "NotificationType": "ACTUAL",
  "ActionType": "APPLY_IAM_POLICY",
  "ActionThreshold": {
    "ActionThresholdValue": 80,
    "ActionThresholdType": "PERCENTAGE"
  },
  "Definition": {
    "IamActionDefinition": {
      "PolicyArn": "arn:aws:iam::123456789012:policy/DenyBedrockPolicy",
      "Roles": ["arn:aws:iam::123456789012:role/DataScienceBedrockRole"]
    }
  },
  "ExecutionRoleArn": "arn:aws:iam::123456789012:role/BudgetsExecutionRole",
  "ApprovalModel": "AUTOMATIC",
  "Subscribers": [
    {
      "SubscriptionType": "SNS",
      "Address": "arn:aws:sns:us-east-1:123456789012:bedrock-cost-alerts"
    }
  ]
}

The DenyBedrockPolicy attached by this action:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyBedrockBudgetBreached",
      "Effect": "Deny",
      "Action": ["bedrock:*"],
      "Resource": "*"
    }
  ]
}

Guardrail 4: Terraform/CDK patterns for guardrail-as-code

Terraform — complete Bedrock cost guardrail module:

# modules/bedrock-guardrails/main.tf
# Deploys: CloudWatch anomaly alarm, SNS topic, Lambda remediation, Budget with action

variable "team_name"          { type = string }
variable "monthly_budget_usd" { type = number }
variable "alert_email"        { type = string }
variable "litellm_base_url"   { type = string }
variable "litellm_master_key" { 
  type      = string
  sensitive = true
}

resource "aws_sns_topic" "bedrock_alerts" {
  name = "bedrock-cost-alerts-${var.team_name}"
}

resource "aws_sns_topic_subscription" "email_alert" {
  topic_arn = aws_sns_topic.bedrock_alerts.arn
  protocol  = "email"
  endpoint  = var.alert_email
}

resource "aws_cloudwatch_metric_alarm" "output_token_anomaly" {
  alarm_name          = "Bedrock-OutputToken-Anomaly-${var.team_name}"
  alarm_description   = "Output token anomaly detected for team ${var.team_name}"
  comparison_operator = "GreaterThanUpperThreshold"
  evaluation_periods  = 2
  treat_missing_data  = "notBreaching"
  threshold_metric_id = "ad1"
  alarm_actions       = [aws_sns_topic.bedrock_alerts.arn]

  metric_query {
    id          = "m1"
    return_data = true
    metric {
      namespace   = "AWS/Bedrock"
      metric_name = "OutputTokenCount"
      period      = 300
      stat        = "Sum"
    }
  }

  metric_query {
    id          = "ad1"
    expression  = "ANOMALY_DETECTION_BAND(m1, 3)"
    label       = "OutputTokenCount (expected)"
    return_data = true
  }
}

resource "aws_iam_role" "lambda_remediation" {
  name = "bedrock-remediation-${var.team_name}"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Principal = { Service = "lambda.amazonaws.com" }
    }]
  })
}

resource "aws_iam_role_policy" "lambda_basic" {
  role = aws_iam_role.lambda_remediation.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"]
        Resource = "arn:aws:logs:*:*:*"
      },
      {
        Effect   = "Allow"
        Action   = ["sns:Publish"]
        Resource = aws_sns_topic.bedrock_alerts.arn
      }
    ]
  })
}

data "archive_file" "lambda_zip" {
  type        = "zip"
  output_path = "/tmp/lambda_tpm_throttle.zip"
  source {
    content  = file("${path.module}/lambda/lambda_tpm_throttle.py")
    filename = "lambda_function.py"
  }
}

resource "aws_lambda_function" "tpm_throttle" {
  function_name    = "bedrock-tpm-throttle-${var.team_name}"
  role             = aws_iam_role.lambda_remediation.arn
  handler          = "lambda_function.lambda_handler"
  runtime          = "python3.12"
  filename         = data.archive_file.lambda_zip.output_path
  source_code_hash = data.archive_file.lambda_zip.output_base64sha256
  timeout          = 30

  environment {
    variables = {
      LITELLM_BASE_URL        = var.litellm_base_url
      LITELLM_MASTER_KEY      = var.litellm_master_key
      TPM_REDUCTION_FACTOR    = "0.25"
    }
  }
}

resource "aws_sns_topic_subscription" "lambda_alert" {
  topic_arn = aws_sns_topic.bedrock_alerts.arn
  protocol  = "lambda"
  endpoint  = aws_lambda_function.tpm_throttle.arn
}

resource "aws_lambda_permission" "sns_invoke" {
  statement_id  = "AllowSNSInvoke"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.tpm_throttle.function_name
  principal     = "sns.amazonaws.com"
  source_arn    = aws_sns_topic.bedrock_alerts.arn
}

resource "aws_budgets_budget" "bedrock_monthly" {
  name         = "bedrock-monthly-${var.team_name}"
  budget_type  = "COST"
  limit_amount = tostring(var.monthly_budget_usd)
  limit_unit   = "USD"
  time_unit    = "MONTHLY"

  cost_filter {
    name   = "Service"
    values = ["Amazon Bedrock"]
  }

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 80
    threshold_type             = "PERCENTAGE"
    notification_type          = "ACTUAL"
    subscriber_sns_topic_arns  = [aws_sns_topic.bedrock_alerts.arn]
  }

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 100
    threshold_type             = "PERCENTAGE"
    notification_type          = "ACTUAL"
    subscriber_sns_topic_arns  = [aws_sns_topic.bedrock_alerts.arn]
  }
}

CDK TypeScript — equivalent pattern:

// lib/bedrock-guardrails-stack.ts
import * as cdk from 'aws-cdk-lib';
import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch';
import * as actions from 'aws-cdk-lib/aws-cloudwatch-actions';
import * as sns from 'aws-cdk-lib/aws-sns';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as budgets from 'aws-cdk-lib/aws-budgets';
import { Construct } from 'constructs';

interface BedrockGuardrailsProps {
  teamName: string;
  modelId: string;
  monthlyBudgetUsd: number;
  litellmBaseUrl: string;
  litellmMasterKey: cdk.SecretValue;
}

export class BedrockGuardrailsStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props: BedrockGuardrailsProps & cdk.StackProps) {
    super(scope, id, props);

    const alertTopic = new sns.Topic(this, 'BedrockAlerts', {
      topicName: `bedrock-cost-alerts-${props.teamName}`,
    });

    const outputTokenMetric = new cloudwatch.Metric({
      namespace: 'AWS/Bedrock',
      metricName: 'OutputTokenCount',
      dimensionsMap: { ModelId: props.modelId },
      period: cdk.Duration.minutes(5),
      statistic: 'Sum',
    });

    const anomalyAlarm = new cloudwatch.Alarm(this, 'OutputTokenAnomaly', {
      alarmName: `Bedrock-OutputToken-Anomaly-${props.teamName}`,
      metric: outputTokenMetric,
      threshold: 0, // overridden by anomaly detection expression
      evaluationPeriods: 2,
      comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_UPPER_THRESHOLD,
    });

    // Note: CDK does not natively support ANOMALY_DETECTION_BAND expressions
    // in high-level constructs. Use CfnAlarm for full control:
    const cfnAlarm = new cloudwatch.CfnAlarm(this, 'OutputTokenAnomalyCfn', {
      alarmName: `Bedrock-OutputToken-AnomalyBand-${props.teamName}`,
      comparisonOperator: 'GreaterThanUpperThreshold',
      evaluationPeriods: 2,
      treatMissingData: 'notBreaching',
      thresholdMetricId: 'ad1',
      alarmActions: [alertTopic.topicArn],
      metrics: [
        {
          id: 'm1',
          returnData: true,
          metricStat: {
            metric: {
              namespace: 'AWS/Bedrock',
              metricName: 'OutputTokenCount',
              dimensions: [{ name: 'ModelId', value: props.modelId }],
            },
            period: 300,
            stat: 'Sum',
          },
        },
        {
          id: 'ad1',
          expression: 'ANOMALY_DETECTION_BAND(m1, 3)',
          label: 'OutputTokenCount (expected)',
          returnData: true,
        },
      ],
    });

    const remediationFn = new lambda.Function(this, 'TpmThrottle', {
      functionName: `bedrock-tpm-throttle-${props.teamName}`,
      runtime: lambda.Runtime.PYTHON_3_12,
      handler: 'lambda_function.lambda_handler',
      code: lambda.Code.fromAsset('lambda/tpm_throttle'),
      timeout: cdk.Duration.seconds(30),
      environment: {
        LITELLM_BASE_URL: props.litellmBaseUrl,
        LITELLM_MASTER_KEY: props.litellmMasterKey.unsafeUnwrap(),
        TPM_REDUCTION_FACTOR: '0.25',
      },
    });

    alertTopic.addSubscription(
      new cdk.aws_sns_subscriptions.LambdaSubscription(remediationFn)
    );
  }
}

Summary: Detection Stack by Latency

Layer	Mechanism	Latency	Automated Remediation
L1 — Sub-minute	LiteLLM Prometheus burn rate	15–30s	Yes (Lambda via SNS)
L2 — 5-minute	CloudWatch OutputTokenCount anomaly band	5–10 min	Yes (Lambda via SNS)
L3 — 5-minute	CloudWatch InvocationThrottles static alarm	5 min	Informational
L4 — Hourly	LiteLLM SpendLogs Athena (scheduled Lambda)	60 min	Yes (Lambda direct)
L5 — Daily	Athena CUR anomaly queries (new principal, model drift)	24h	Alert-driven manual
L6 — Lagging	AWS Cost Anomaly Detection	24–48h	No

The correct operational posture: L1 and L2 handle real-time incidents with automated remediation. L3 provides throttle storm visibility. L4 catches patterns that accumulate over hours. L5 and L6 handle audit-grade retrospective review and serve as validation that the earlier layers worked. Do not design an incident response process that depends on L6 alerting as the primary signal.

What it actually does

Monitor types

Recommended configuration

SNS → Slack/PagerDuty routing

Limitations for Bedrock specifically

2. Near-Real-Time Anomaly Detection via LiteLLM

Prometheus metric structure

Grafana burn rate alert

Python burn rate calculation against LiteLLM SpendLogs

3. CloudWatch Metric-Based Anomaly Detection

Bedrock CloudWatch metrics (2026)

CloudWatch Anomaly Detection alarm (JSON)

InvocationThrottles spike detection (static threshold)

InvocationLatency degradation as indirect cost signal

CloudWatch Anomaly Detection ML bands vs. static thresholds

Cost of CloudWatch alarms for Bedrock

4. EventBridge + Lambda Automated Remediation

Architecture overview

Pattern 1: Budget breach → Lambda → LiteLLM team budget to $0 (hard stop)

Pattern 2: Anomaly alert → Lambda → reduce TPM quota on virtual key

Pattern 3: Runaway agent loop detection → Lambda → disable API key

Pattern 4: Circuit breaker — SNS → SQS → Lambda with exponential backoff before re-enabling

5. Bedrock-Specific Anomaly Patterns

Pattern A: Agent loop runaway

Pattern B: Prompt injection attack

Pattern C: Credential leak / unauthorized use

Pattern D: Cache miss storm

Pattern E: Model routing misconfiguration

6. Athena Anomaly Detection Queries

Query A: Day-over-day spend spike (>3× rolling 7-day average)

Query B: Unexpected model appearing in usage

Query C: IAM principal with first-time Bedrock usage

Query D: Sudden cache hit rate collapse

7. Cost Anomaly Playbook Integration

Severity classification

30-minute P1 runbook

Alert routing summary

8. Proactive Cost Guardrails

Guardrail 1: SCP to enforce cost allocation tags on all Bedrock calls

Guardrail 2: LiteLLM max_budget + budget_duration as a circuit breaker

Guardrail 3: AWS Budgets action — IAM policy attachment when budget exceeded

Guardrail 4: Terraform/CDK patterns for guardrail-as-code

Summary: Detection Stack by Latency

Guardrail 2: LiteLLM `max_budget` + `budget_duration` as a circuit breaker