← Agent Frameworks 🕐 9 min read
Agent Frameworks

Bedrock FinOps Automation — Anomaly Detection, Right-Sizing, Budget Enforcement (2026)

Table of Contents · March 2026

> **Audience:** Platform engineers and FinOps leads operating production LLM workloads on AWS Bedrock, whether via LiteLLM proxy or direct SDK calls.

Audience: Platform engineers and FinOps leads operating production LLM workloads on AWS Bedrock, whether via LiteLLM proxy or direct SDK calls. This note is opinionated and specific — it assumes a multi-account AWS Organization, existing CUR 2.0 delivery to S3, and at least one team running token-volume workloads exceeding $5k/month on Bedrock.

  1. AWS Cost Anomaly Detection for Bedrock
  2. Automated Reserved Tier Right-Sizing
  3. Budget-Gated CI/CD
  4. Multi-Account Showback Automation
  5. LiteLLM Budget Enforcement Automation
  6. Terraform / CDK IaC Patterns

1. AWS Cost Anomaly Detection for Bedrock

1.1 Service Monitor Configuration

Cost Anomaly Detection (CAD) operates at the service level via the ce:CreateAnomalyMonitor API. For Bedrock workloads, create a DIMENSIONAL monitor scoped to SERVICE = "Amazon Bedrock":

{
  "MonitorName": "Bedrock-Production",
  "MonitorType": "DIMENSIONAL",
  "MonitorDimension": "SERVICE"
}

With a service monitor, CAD’s ML model trains on historical Bedrock spend patterns per account. It does not, by default, isolate spend by model ID — that requires cost allocation tags (covered in §1.4).

IAM permissions required on the caller role:

ce:CreateAnomalyMonitor
ce:CreateAnomalySubscription
ce:GetAnomalies
sns:CreateTopic
sns:SetTopicAttributes

1.2 Threshold Configuration — Percentage vs Absolute

The aws_ce_anomaly_subscription resource accepts either an ABSOLUTE_VALUE or PERCENTAGE threshold via the threshold_expression block (provider ≥ 5.x) or the deprecated threshold float for absolute-only.

Production guidance:

  • Use ABSOLUTE_VALUE for high-spend services where a 20% spike might be acceptable but a $500 overage is not. For Bedrock workloads spending $10k/month, a $300 absolute threshold catches real incidents without alert fatigue.
  • Use PERCENTAGE during ramp-up phases when baseline spend is still growing. A 40% spike over a 7-day rolling baseline is a meaningful signal even when absolute dollars are low.
  • Combine both: subscribe with absolute for paging, subscribe with percentage for Slack info-channel.

Alert frequency options: IMMEDIATE, DAILY, WEEKLY. For production Bedrock, use IMMEDIATE for the SNS→PagerDuty path and DAILY for the Slack digest.

1.3 EventBridge → SNS → Slack Routing

CAD emits events to EventBridge under the aws.ce source with detail-type "Cost Anomaly Detection Alert". The event payload includes anomalyTotalImpact, rootCauses[], and anomalyStartDate.

EventBridge rule (JSON):

{
  "source": ["aws.ce"],
  "detail-type": ["Cost Anomaly Detection Alert"],
  "detail": {
    "anomalyTotalImpact": {
      "numericGreaterThan": 100
    }
  }
}

Route this rule to an SNS topic. A Lambda subscriber formats and posts to Slack:

import json, os, urllib.request

def handler(event, context):
    detail = event["detail"]
    impact = detail.get("anomalyTotalImpact", {})
    root   = detail.get("rootCauses", [{}])[0]

    msg = {
        "text": (
            f":rotating_light: *Bedrock Cost Anomaly*\n"
            f"Impact: *${impact.get('totalImpact', 0):.2f}* "
            f"({impact.get('totalImpactPercentage', 0):.1f}% above expected)\n"
            f"Service: {root.get('service', 'n/a')} | "
            f"Region: {root.get('region', 'n/a')} | "
            f"Account: {root.get('linkedAccount', 'n/a')}\n"
            f"Started: {detail.get('anomalyStartDate', 'n/a')}"
        )
    }

    req = urllib.request.Request(
        os.environ["SLACK_WEBHOOK_URL"],
        data=json.dumps(msg).encode(),
        headers={"Content-Type": "application/json"},
    )
    urllib.request.urlopen(req)

SNS topic policy — required to allow the CAD service principal to publish:

{
  "Sid": "AllowCostAnomalyDetection",
  "Effect": "Allow",
  "Principal": { "Service": "costalerts.amazonaws.com" },
  "Action": "SNS:Publish",
  "Resource": "arn:aws:sns:us-east-1:ACCOUNT_ID:bedrock-anomaly-alerts"
}

1.4 Granular Attribution with Cost Allocation Tags — The Per-Model Gap

Critical limitation: CAD service monitors cannot differentiate between Claude Sonnet and Claude Haiku spikes. You see AmazonBedrock total anomalies, not per-model anomalies. Two mechanisms close this gap:

Option A — Application Inference Profiles (AIPs) with cost allocation tags: Create an AIP per team/model combination. AIPs are logical wrappers that accept arbitrary resource tags:

import boto3

bedrock = boto3.client("bedrock")
bedrock.create_inference_profile(
    inferenceProfileName="team-search-claude-sonnet",
    description="Search team Sonnet 3.7 profile",
    modelSource={"copyFrom": "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-7-sonnet-20250219-v1:0"},
    tags=[
        {"key": "Team", "value": "search"},
        {"key": "Model", "value": "claude-sonnet-37"},
        {"key": "CostCenter", "value": "eng-platform"},
    ]
)

Activate these tags in the Billing console under Cost Allocation Tags (takes 24–48 hours to propagate to CUR 2.0). Now run aws ce create-cost-category-definition to build a model-level cost category.

Option B — IAM principal-based attribution (2026 GA feature): AWS now automatically records the full IAM ARN of the caller in CUR 2.0’s line_item_iam_principal column. If your LiteLLM proxy or application assumes a different role per team (via sts:AssumeRole with session tags), each team’s Bedrock spend appears as a distinct line item with no instrumentation changes required.

Session tag passthrough pattern:

sts = boto3.client("sts")
assumed = sts.assume_role(
    RoleArn="arn:aws:iam::ACCOUNT:role/bedrock-inference",
    RoleSessionName="team-search-user-42",
    Tags=[
        {"Key": "team", "Value": "search"},
        {"Key": "user_id", "Value": "user-42"},
    ],
    TransitiveTagKeys=["team"],
)

The resulting session credentials, when used for Bedrock calls, generate CUR 2.0 rows attributed to assumed-role/bedrock-inference/team-search-user-42 with iamPrincipal/team=search as a usable tag dimension.

1.5 Practical Alert Thresholds from Production

From published production deployments and AWS guidance:

Tier Threshold type Value Channel Frequency
P1 paging Absolute $500 single anomaly PagerDuty IMMEDIATE
P2 Slack alert Absolute $100 single anomaly #finops-alerts IMMEDIATE
Daily digest Percentage 30% above baseline #finops-digest DAILY
Weekly report Any anomaly detected — Email to FinOps lead WEEKLY

Known CAD limitations in Bedrock context:

  • CAD trains on ~60 days of history. New workloads have noisy baselines for the first 2–3 weeks.
  • CAD is a spend anomaly detector, not a quota detector. A throttle storm that costs nothing (because throttled requests don’t incur charges) is invisible to CAD. Use CloudWatch InvocationThrottles alarms (§2) for that.
  • Budget data in CAD refreshes up to 24 hours behind. It cannot stop a runaway job in real time. Treat CAD as a billing-layer control, not an application-layer guardrail.

2. Automated Reserved Tier Right-Sizing

2.1 Key CloudWatch Metrics

All Bedrock metrics live in the AWS/Bedrock CloudWatch namespace. Dimensions: ModelId (foundation model or cross-region inference profile ARN), and optionally ProvisionedModelArn for provisioned throughput.

Metric Unit Use
EstimatedTPMQuotaUsage Percent Fraction of your TPM quota consumed in the last minute. Fires on every successful request. Use this to detect when you are approaching quota before throttling starts.
InputTokenCount Count Tokens sent per invocation. Sum over 1-minute periods to get TPM.
OutputTokenCount Count Tokens generated per invocation.
InvocationThrottles Count Requests rejected because the workload exceeded quota. Non-zero value means user-visible errors.
InvocationLatency Milliseconds End-to-end latency including queuing.
TimeToFirstToken Milliseconds TTFT for streaming calls. Added March 2026.

EstimatedTPMQuotaUsage includes cache write tokens and output burndown multipliers, so it is more accurate than computing InputTokenCount / quota_limit yourself.

2.2 Right-Sizing Decision Logic

The core automation logic runs on a 5-minute CloudWatch cron or a Lambda invoked by an EventBridge schedule. The algorithm:

current_tpm_usage_pct = avg(EstimatedTPMQuotaUsage, 1h)
throttle_rate_pct     = sum(InvocationThrottles, 1h) / sum(Invocations, 1h) * 100

if throttle_rate_pct > 2.0:
    action = INCREASE_RESERVATION
    signal = "throttle_pressure"
elif current_tpm_usage_pct < 60.0 for 7 consecutive days:
    action = DECREASE_RESERVATION
    signal = "underutilization"
else:
    action = HOLD

Thresholds:

  • Underutilization: < 60% average TPM usage over 7 days. Source: AWS FinOps recommendations for Provisioned Throughput.
  • Throttle pressure: > 2% of invocations throttled in any 1-hour window. Even brief throttle storms cause cascading retries.

Important constraint: Provisioned Throughput by Model Units (the commitment-based form) does not support changing the model unit count after creation — you must create a new purchase and delete the old one. Provisioned Throughput by tokens (the newer per-token reservation form for select models) does support UpdateProvisionedModelThroughput. Confirm which type applies to your model before building automation around it.

2.3 CloudWatch Alarm → Lambda → SNS → Slack Runbook

Create two alarms per model: one for over-utilization (throttle risk), one for underutilization.

Over-utilization alarm:

{
  "AlarmName": "Bedrock-Claude-Sonnet-Throttle-High",
  "Namespace": "AWS/Bedrock",
  "MetricName": "InvocationThrottles",
  "Dimensions": [{"Name": "ModelId", "Value": "anthropic.claude-3-7-sonnet-20250219-v1:0"}],
  "Statistic": "Sum",
  "Period": 300,
  "EvaluationPeriods": 3,
  "Threshold": 10,
  "ComparisonOperator": "GreaterThanOrEqualToThreshold",
  "AlarmActions": ["arn:aws:sns:us-east-1:ACCOUNT:bedrock-ops"],
  "TreatMissingData": "notBreaching"
}

Underutilization alarm (uses EstimatedTPMQuotaUsage):

{
  "AlarmName": "Bedrock-Claude-Sonnet-Quota-Underused",
  "Namespace": "AWS/Bedrock",
  "MetricName": "EstimatedTPMQuotaUsage",
  "Dimensions": [{"Name": "ModelId", "Value": "anthropic.claude-3-7-sonnet-20250219-v1:0"}],
  "ExtendedStatistic": "p95",
  "Period": 3600,
  "EvaluationPeriods": 168,
  "Threshold": 60.0,
  "ComparisonOperator": "LessThanThreshold",
  "AlarmActions": ["arn:aws:sns:us-east-1:ACCOUNT:bedrock-ops"]
}

168 evaluation periods × 3600 seconds = 7 days of data required before alarm can fire. This prevents noise from weekend dips.

Lambda runbook triggered by the throttle alarm:

import boto3, json, os

bedrock = boto3.client("bedrock")
sns     = boto3.client("sns")

def handler(event, context):
    alarm_name = event["Records"][0]["Sns"]["Message"]
    alarm_data = json.loads(alarm_name)

    model_id    = _extract_model_from_alarm(alarm_data)
    current_rpm = _get_service_quota(model_id, quota_code="L-XXXXXXXX")  # replace with actual quota code
    new_rpm     = int(current_rpm * 1.5)

    # Post to Slack with runbook link — do NOT auto-increase without approval
    sns.publish(
        TopicArn=os.environ["OPS_TOPIC_ARN"],
        Subject=f"Bedrock throttle alert: {model_id}",
        Message=(
            f"Model {model_id} is throttling.\n"
            f"Current quota: {current_rpm} RPM\n"
            f"Recommended new quota: {new_rpm} RPM\n"
            f"Runbook: https://wiki.internal/bedrock-quota-increase\n"
            f"Auto-request: run `make bedrock-quota-increase MODEL={model_id} RPM={new_rpm}`"
        )
    )

def _get_service_quota(model_id, quota_code):
    sq = boto3.client("service-quotas")
    resp = sq.get_service_quota(ServiceCode="bedrock", QuotaCode=quota_code)
    return resp["Quota"]["Value"]

def _extract_model_from_alarm(alarm_data):
    for dim in alarm_data.get("Trigger", {}).get("Dimensions", []):
        if dim["name"] == "ModelId":
            return dim["value"]
    return "unknown"

Automated quota increase via Service Quotas API:

sq = boto3.client("service-quotas")
sq.request_service_quota_increase(
    ServiceCode="bedrock",
    QuotaCode="L-XXXXXXXX",   # quota code for the specific model RPM/TPM
    DesiredValue=new_rpm,
)

Quota increases for Bedrock RPM/TPM are reviewed by AWS and typically approved within 1–2 business days for moderate increases (2x). For > 5x increases, expect a business justification request.

2.4 Provisioned Throughput Automation via boto3

For workloads that have committed to Provisioned Throughput (and are using a model that supports unit count changes):

bedrock = boto3.client("bedrock")

def rightsize_provisioned_throughput(provisioned_model_arn: str, new_model_units: int):
    """
    Update provisioned model units. Only works for models that support
    UpdateProvisionedModelThroughput capacity changes.
    Check beforehand: describe_provisioned_model_throughput to verify commitmentDuration.
    """
    resp = bedrock.describe_provisioned_model_throughput(
        provisionedModelId=provisioned_model_arn
    )

    current_units = resp["modelUnits"]
    status        = resp["status"]  # InService | Creating | Updating | Failed

    if status != "InService":
        raise RuntimeError(f"Cannot resize: status is {status}")

    if current_units == new_model_units:
        return {"action": "noop", "units": current_units}

    bedrock.update_provisioned_model_throughput(
        provisionedModelId=provisioned_model_arn,
        desiredModelUnits=new_model_units,
    )
    return {"action": "resized", "from": current_units, "to": new_model_units}

Commitment duration caveat: Provisioned Throughput with a 1-month or 6-month commitment cannot be cancelled before the term ends. Downsizing within a commitment period may not be supported. Use on-demand (no-term) Provisioned Throughput for workloads where you need resize flexibility.

3. Budget-Gated CI/CD

3.1 Architecture Overview

The pattern: before a model deployment job runs, query the AWS Budgets API to check current Bedrock spend against the monthly threshold. If spend exceeds the gate, block the deployment and alert. This prevents deploying new model endpoints during a cost spike without human sign-off.

Important limitation: AWS Budgets data refreshes once every 24 hours. This gate catches yesterday’s overage, not today’s real-time runaway. Combine with a CloudWatch alarm on EstimatedTPMQuotaUsage for real-time enforcement.

3.2 GitHub Actions Integration

# .github/workflows/bedrock-deploy.yml
name: Bedrock Model Deploy

on:
  workflow_dispatch:
    inputs:
      model_id:
        description: "Bedrock model to deploy"
        required: true
      environment:
        description: "Target environment"
        required: true
        default: "production"

jobs:
  budget-check:
    runs-on: ubuntu-latest
    outputs:
      budget_ok: ${{ steps.check.outputs.budget_ok }}
    steps:
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_DEPLOY_ROLE_ARN }}
          aws-region: us-east-1

      - name: Check Bedrock budget gate
        id: check
        env:
          BUDGET_NAME: ${{ vars.BEDROCK_MONTHLY_BUDGET_NAME }}
          ACCOUNT_ID: ${{ secrets.AWS_ACCOUNT_ID }}
          SPEND_THRESHOLD_PCT: "85"
        run: |
          python3 - <<'EOF'
          import boto3, json, sys, os

          budgets = boto3.client("budgets", region_name="us-east-1")
          account_id   = os.environ["ACCOUNT_ID"]
          budget_name  = os.environ["BUDGET_NAME"]
          threshold_pct = float(os.environ["SPEND_THRESHOLD_PCT"])

          resp = budgets.describe_budget(
              AccountId=account_id,
              BudgetName=budget_name,
          )
          budget = resp["Budget"]

          limit    = float(budget["BudgetLimit"]["Amount"])
          actual   = float(budget["CalculatedSpend"]["ActualSpend"]["Amount"])
          forecast = float(budget["CalculatedSpend"].get("ForecastedSpend", {}).get("Amount", actual))
          pct      = (actual / limit) * 100

          print(f"Bedrock budget: ${actual:.2f} / ${limit:.2f} ({pct:.1f}%)")
          print(f"Forecasted: ${forecast:.2f}")

          if pct >= threshold_pct:
              print(f"::error::Budget gate BLOCKED: {pct:.1f}% of monthly Bedrock budget consumed.")
              print("budget_ok=false", file=open(os.environ["GITHUB_OUTPUT"], "a"))
              sys.exit(1)
          else:
              print(f"Budget gate PASSED: {pct:.1f}% consumed.")
              print("budget_ok=true", file=open(os.environ["GITHUB_OUTPUT"], "a"))
          EOF

  deploy:
    needs: budget-check
    if: needs.budget-check.outputs.budget_ok == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Deploy Bedrock configuration
        run: |
          echo "Deploying model: ${{ inputs.model_id }} to ${{ inputs.environment }}"
          # CDK deploy, cdk synth, or boto3 calls here

3.3 Budget Action — Automatic IAM Policy Attach

For environments where you want automatic enforcement (not just a gate), configure a Budget Action to attach an IAM deny policy when spend crosses a threshold. This is the hard stop:

IAM policy document — deny Bedrock inference:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyBedrockInference",
      "Effect": "Deny",
      "Action": [
        "bedrock:InvokeModel",
        "bedrock:InvokeModelWithResponseStream",
        "bedrock:Converse",
        "bedrock:ConverseStream"
      ],
      "Resource": "*"
    }
  ]
}

Attach this as a budget action via boto3:

import boto3

budgets = boto3.client("budgets")

budgets.create_budget_action(
    AccountId="123456789012",
    BudgetName="Bedrock-Monthly-Production",
    NotificationType="ACTUAL",
    ActionType="APPLY_IAM_POLICY",
    ActionThreshold={
        "ActionThresholdValue": 95.0,
        "ActionThresholdType": "PERCENTAGE",
    },
    Definition={
        "IamActionDefinition": {
            "PolicyArn": "arn:aws:iam::123456789012:policy/DenyBedrockInference",
            "Roles": ["arn:aws:iam::123456789012:role/bedrock-inference-role"],
        }
    },
    ExecutionRoleArn="arn:aws:iam::123456789012:role/budgets-execution-role",
    ApprovalModel="AUTOMATIC",
    Subscribers=[
        {
            "SubscriptionType": "SNS",
            "Address": "arn:aws:sns:us-east-1:123456789012:bedrock-alerts"
        }
    ]
)

IAM permissions required on the budget execution role:

iam:AttachRolePolicy
iam:DetachRolePolicy
budgets:ModifyBudget

The AUTOMATIC approval model fires without human approval. Use MANUAL if you want a two-step confirmation flow via the AWS console before the policy attaches.

3.4 CDK Pattern for Budget Gate Infrastructure

# cdk/bedrock_budget_gate.py
from aws_cdk import (
    aws_budgets as budgets,
    aws_iam as iam,
    aws_sns as sns,
    Stack,
)
from constructs import Construct

class BedrockBudgetGate(Stack):
    def __init__(self, scope: Construct, id: str,
                 monthly_limit_usd: float = 10000.0,
                 gate_threshold_pct: float = 85.0,
                 hard_stop_pct: float = 95.0,
                 **kwargs):
        super().__init__(scope, id, **kwargs)

        alert_topic = sns.Topic(self, "BedrockBudgetAlerts",
            display_name="Bedrock Budget Alerts")

        deny_policy = iam.ManagedPolicy(self, "DenyBedrockInference",
            statements=[
                iam.PolicyStatement(
                    effect=iam.Effect.DENY,
                    actions=[
                        "bedrock:InvokeModel",
                        "bedrock:InvokeModelWithResponseStream",
                        "bedrock:Converse",
                        "bedrock:ConverseStream",
                    ],
                    resources=["*"],
                )
            ]
        )

        inference_role = iam.Role.from_role_name(
            self, "InferenceRole", "bedrock-inference-role")

        budgets.CfnBudget(self, "BedrockMonthlyBudget",
            budget=budgets.CfnBudget.BudgetDataProperty(
                budget_name="Bedrock-Monthly-Production",
                budget_type="COST",
                time_unit="MONTHLY",
                budget_limit=budgets.CfnBudget.SpendProperty(
                    amount=str(monthly_limit_usd),
                    unit="USD",
                ),
                cost_filters={"Service": ["Amazon Bedrock"]},
            ),
            notifications_with_subscribers=[
                budgets.CfnBudget.NotificationWithSubscribersProperty(
                    notification=budgets.CfnBudget.NotificationProperty(
                        notification_type="ACTUAL",
                        comparison_operator="GREATER_THAN",
                        threshold=gate_threshold_pct,
                        threshold_type="PERCENTAGE",
                    ),
                    subscribers=[
                        budgets.CfnBudget.SubscriberProperty(
                            subscription_type="SNS",
                            address=alert_topic.topic_arn,
                        )
                    ]
                )
            ]
        )

4. Multi-Account Showback Automation

4.1 CUR 2.0 Setup for Bedrock Attribution

CUR 2.0 (Cost and Usage Report, new format) is the foundation. Bedrock-specific columns available as of 2026:

CUR 2.0 column Content
line_item_product_code AmazonBedrock
line_item_iam_principal Full ARN of the IAM principal that made the call (new 2026 feature)
product_model_id Bedrock model ID (e.g., anthropic.claude-3-7-sonnet-20250219-v1:0)
resource_tags/user:Team Custom cost allocation tag value
line_item_usage_type e.g., USE1-Claude-InputTokens
line_item_usage_amount Token count
line_item_unblended_cost USD cost

Enable CUR 2.0 delivery to S3, then run a Glue Crawler to catalog it. Athena queries can reference the table cur_database.cur_report_name.

4.2 Athena Scheduled Queries for Monthly Chargeback

Monthly chargeback by team (using cost allocation tags):

-- Bedrock monthly chargeback by Team tag
-- Run on the 2nd of each month for prior month
SELECT
    DATE_FORMAT(line_item_usage_start_date, '%Y-%m')              AS billing_month,
    resource_tags_user_team                                        AS team,
    product_model_id                                               AS model,
    line_item_product_code                                         AS service,
    SUM(line_item_usage_amount)                                    AS total_tokens,
    SUM(line_item_unblended_cost)                                  AS total_cost_usd,
    COUNT(DISTINCT line_item_iam_principal)                        AS distinct_callers
FROM cur_database.aws_cur
WHERE
    line_item_product_code = 'AmazonBedrock'
    AND line_item_line_item_type = 'Usage'
    AND DATE_FORMAT(line_item_usage_start_date, '%Y-%m')
        = DATE_FORMAT(DATE_ADD('month', -1, CURRENT_DATE), '%Y-%m')
GROUP BY 1, 2, 3, 4
ORDER BY total_cost_usd DESC;

Per-model daily spend trend (for anomaly investigation):

SELECT
    CAST(line_item_usage_start_date AS DATE) AS usage_date,
    product_model_id,
    SUM(line_item_unblended_cost)            AS daily_cost_usd,
    SUM(line_item_usage_amount)              AS daily_tokens
FROM cur_database.aws_cur
WHERE
    line_item_product_code = 'AmazonBedrock'
    AND line_item_usage_start_date >= CURRENT_DATE - INTERVAL '30' DAY
GROUP BY 1, 2
ORDER BY 1 DESC, daily_cost_usd DESC;

4.3 Cost Categories API — Division Mapping

Cost Categories allow you to define cost attribution rules that show up in Cost Explorer and CUR 2.0. Create a BEDROCK_DIVISION category:

import boto3

ce = boto3.client("ce", region_name="us-east-1")

ce.create_cost_category_definition(
    Name="BEDROCK_DIVISION",
    RuleVersion="CostCategoryExpression.v1",
    Rules=[
        {
            "Value": "Search",
            "Rule": {
                "Tags": {
                    "Key": "Team",
                    "Values": ["search", "search-platform"],
                    "MatchOptions": ["EQUALS"],
                }
            },
            "Type": "REGULAR",
        },
        {
            "Value": "Recommendations",
            "Rule": {
                "Tags": {
                    "Key": "Team",
                    "Values": ["recs", "recommendations"],
                    "MatchOptions": ["EQUALS"],
                }
            },
            "Type": "REGULAR",
        },
        {
            "Value": "Shared-Infrastructure",
            "Rule": {
                "And": [
                    {
                        "Dimensions": {
                            "Key": "SERVICE",
                            "Values": ["Amazon Bedrock"],
                        }
                    },
                    {
                        "Not": {
                            "Tags": {
                                "Key": "Team",
                                "Values": ["search", "recs"],
                                "MatchOptions": ["EQUALS"],
                            }
                        }
                    }
                ]
            },
            "Type": "REGULAR",
        }
    ],
    DefaultValue="Untagged",
    SplitChargeRules=[
        {
            "Source": "Shared-Infrastructure",
            "Targets": ["Search", "Recommendations"],
            "Method": "PROPORTIONAL",
        }
    ],
)

SplitChargeRules with PROPORTIONAL method distributes shared Bedrock costs (e.g., shared RAG infrastructure) proportionally based on each team’s tagged spend. This is the right pattern for shared embedding services.

4.4 S3 Export → BI Pipeline + Lambda Cron Delivery

Lambda cron for automated monthly report delivery:

import boto3, io, csv, os
from datetime import date, timedelta

def handler(event, context):
    """
    Runs on the 2nd of each month via EventBridge.
    1. Queries Athena for prior-month Bedrock chargeback
    2. Writes CSV to S3
    3. Posts summary to Slack
    """
    athena   = boto3.client("athena")
    s3       = boto3.client("s3")

    prior_month = (date.today().replace(day=1) - timedelta(days=1)).strftime("%Y-%m")
    output_key  = f"finops-reports/bedrock-chargeback-{prior_month}.csv"
    output_uri  = f"s3://{os.environ['REPORTS_BUCKET']}/{output_key}"

    query = f"""
    SELECT
        resource_tags_user_team     AS team,
        product_model_id            AS model,
        SUM(line_item_usage_amount) AS tokens,
        SUM(line_item_unblended_cost) AS cost_usd
    FROM {os.environ['CUR_TABLE']}
    WHERE line_item_product_code = 'AmazonBedrock'
      AND DATE_FORMAT(line_item_usage_start_date, '%Y-%m') = '{prior_month}'
    GROUP BY 1, 2
    ORDER BY cost_usd DESC
    """

    start = athena.start_query_execution(
        QueryString=query,
        QueryExecutionContext={"Database": os.environ["ATHENA_DATABASE"]},
        ResultConfiguration={
            "OutputLocation": f"s3://{os.environ['ATHENA_RESULTS_BUCKET']}/queries/"
        },
        WorkGroup=os.environ.get("ATHENA_WORKGROUP", "primary"),
    )

    exec_id = start["QueryExecutionId"]
    _wait_for_query(athena, exec_id)

    results = athena.get_query_results(QueryExecutionId=exec_id)
    rows    = _parse_athena_results(results)

    # Write CSV to S3
    buf = io.StringIO()
    writer = csv.DictWriter(buf, fieldnames=["team", "model", "tokens", "cost_usd"])
    writer.writeheader()
    writer.writerows(rows)
    s3.put_object(
        Bucket=os.environ["REPORTS_BUCKET"],
        Key=output_key,
        Body=buf.getvalue().encode(),
        ContentType="text/csv",
    )

    # Post summary to Slack
    total = sum(float(r["cost_usd"]) for r in rows)
    top_teams = sorted(rows, key=lambda r: float(r["cost_usd"]), reverse=True)[:5]
    _post_slack(prior_month, total, top_teams, output_uri)


def _wait_for_query(athena, exec_id, max_attempts=60):
    import time
    for _ in range(max_attempts):
        resp = athena.get_query_execution(QueryExecutionId=exec_id)
        state = resp["QueryExecution"]["Status"]["State"]
        if state in ("SUCCEEDED",):
            return
        if state in ("FAILED", "CANCELLED"):
            raise RuntimeError(f"Athena query {state}: "
                               f"{resp['QueryExecution']['Status'].get('StateChangeReason')}")
        time.sleep(5)
    raise TimeoutError("Athena query timed out")


def _parse_athena_results(results):
    cols = [c["VarCharValue"] for c in results["ResultSet"]["Rows"][0]["Data"]]
    rows = []
    for row in results["ResultSet"]["Rows"][1:]:
        values = [d.get("VarCharValue", "") for d in row["Data"]]
        rows.append(dict(zip(cols, values)))
    return rows


def _post_slack(month, total, top_teams, report_uri):
    import json, urllib.request
    lines = [f"• {r['team']} | {r['model'].split('.')[-1]} | ${float(r['cost_usd']):.2f}"
             for r in top_teams]
    msg = {
        "text": (
            f":bar_chart: *Bedrock Chargeback Report — {month}*\n"
            f"Total: *${total:,.2f}*\n\n"
            f"Top 5 by cost:\n" + "\n".join(lines) + f"\n\n"
            f"Full report: {report_uri}"
        )
    }
    req = urllib.request.Request(
        os.environ["SLACK_WEBHOOK_URL"],
        data=json.dumps(msg).encode(),
        headers={"Content-Type": "application/json"},
    )
    urllib.request.urlopen(req)

EventBridge rule for monthly trigger:

{
  "ScheduleExpression": "cron(0 8 2 * ? *)",
  "Description": "Trigger Bedrock monthly chargeback report on 2nd of each month at 08:00 UTC"
}

5. LiteLLM Budget Enforcement Automation

5.1 Architecture Overview

LiteLLM proxy operates as a budget enforcement layer between application code and Bedrock. All inference traffic passes through the proxy, which tracks spend in a PostgreSQL backend (required for multi-replica deployments). The enforcement hierarchy:

Global proxy budget (litellm_settings.max_budget)
  └─ Team budget (team.max_budget)
       └─ Virtual key budget (key.max_budget)
            └─ Per-user budget (user.max_budget)

Each level enforces independently. A key budget does not override a team budget; both are checked on every request.

5.2 Core config.yaml Configuration

# litellm/config.yaml — production Bedrock deployment
model_list:
  - model_name: "claude-sonnet-37"
    litellm_params:
      model: "bedrock/anthropic.claude-3-7-sonnet-20250219-v1:0"
      aws_region_name: "us-east-1"
      # Use cross-region inference profile for higher quota
      model: "bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0"

  - model_name: "claude-haiku-35"
    litellm_params:
      model: "bedrock/us.anthropic.claude-3-5-haiku-20241022-v1:0"
      aws_region_name: "us-east-1"

litellm_settings:
  # Global monthly budget across all keys — hard stop
  max_budget: 50000.0          # USD
  budget_duration: "30d"

  # Alert channels
  alerting:
    - slack
    - email

  # Slack alerting config
  alerting_args:
    daily_report_frequency: 86400        # seconds between daily digests
    report_check_interval: 3600          # how often to check for budget alerts (1h)
    budget_alert_ttl: 3600               # deduplicate identical alerts within 1h window
    slack_daily_report_frequency: 86400

  # Spend report to #finops-alerts every 24h
  spend_report_frequency: "1d"

general_settings:
  master_key: "sk-${LITELLM_MASTER_KEY}"
  database_url: "postgresql://${DB_USER}:${DB_PASS}@${DB_HOST}:5432/litellm"

  # Optional: enable spend tracking on all requests
  enable_jwt_auth: false
  store_model_in_db: true

Environment variables:

SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T.../B.../xxx
LITELLM_MASTER_KEY=sk-xxx
DB_USER=litellm
DB_PASS=xxx
DB_HOST=postgres.internal

5.3 Virtual Key Budget Configuration

Create keys via the LiteLLM admin API. The /key/generate endpoint accepts:

curl -X POST https://litellm.internal/key/generate \
  -H "Authorization: Bearer sk-${LITELLM_MASTER_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "key_alias": "team-search-production",
    "team_id": "team-search",
    "max_budget": 2000.0,
    "soft_budget": 1600.0,
    "budget_duration": "30d",
    "models": ["claude-sonnet-37", "claude-haiku-35"],
    "metadata": {
      "cost_center": "eng-platform",
      "owner": "search-team@company.com"
    }
  }'

Soft budget (soft_budget) triggers an alert at the specified dollar amount without blocking requests. Hard budget (max_budget) blocks requests with HTTP 429 once exhausted. Set soft_budget at 80% of max_budget for a meaningful warning window.

5.4 Multi-Threshold Budget Alerts

LiteLLM v1.83+ supports multi-threshold alerting on virtual keys, firing at configurable percentages rather than only at 80%:

import httpx, os

async def create_key_with_thresholds(team_id: str, budget_usd: float):
    async with httpx.AsyncClient() as client:
        resp = await client.post(
            "https://litellm.internal/key/generate",
            headers={"Authorization": f"Bearer {os.environ['LITELLM_MASTER_KEY']}"},
            json={
                "team_id": team_id,
                "max_budget": budget_usd,
                "soft_budget": budget_usd * 0.8,     # 80% alert
                "budget_duration": "30d",
                "budget_reset_at": None,              # auto-resets per budget_duration
                "metadata": {
                    "alert_thresholds": [50, 80, 95]  # pct — for webhook callbacks
                }
            }
        )
        return resp.json()

The alert_thresholds in metadata is used by custom callback handlers. Wire a callback:

# callbacks/budget_callback.py
from litellm.integrations.custom_logger import CustomLogger
import httpx, os

class BudgetThresholdCallback(CustomLogger):

    THRESHOLDS = [0.50, 0.80, 0.95]

    async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
        key_metadata = kwargs.get("metadata", {})
        key_alias    = key_metadata.get("user_api_key_alias", "unknown")
        spend        = kwargs.get("response_cost", 0.0)
        max_budget   = kwargs.get("user_api_key_max_budget")

        if not max_budget:
            return

        pct = spend / max_budget

        for threshold in self.THRESHOLDS:
            if abs(pct - threshold) < 0.01:   # within 1% of threshold
                await self._alert_slack(key_alias, spend, max_budget, pct)
                break

    async def _alert_slack(self, key_alias, spend, budget, pct):
        async with httpx.AsyncClient() as client:
            await client.post(
                os.environ["SLACK_WEBHOOK_URL"],
                json={
                    "text": (
                        f":money_with_wings: *LiteLLM Budget Alert* — `{key_alias}`\n"
                        f"Spend: *${spend:.2f}* / *${budget:.2f}* ({pct*100:.0f}%)\n"
                        f"{'*Hard limit approaching — requests will be blocked soon.*' if pct >= 0.95 else ''}"
                    )
                }
            )

Register the callback in config.yaml:

litellm_settings:
  success_callback:
    - /app/callbacks/budget_callback.BudgetThresholdCallback

5.5 Auto-Rotation on Budget Exhaustion

When a virtual key exhausts its budget, LiteLLM returns HTTP 429. For workloads that cannot tolerate downtime, implement automatic key rotation:

Pattern 1 — Client-side rotation with fallback key:

import litellm, os

PRIMARY_KEY   = os.environ["LITELLM_KEY_PRIMARY"]
FALLBACK_KEY  = os.environ["LITELLM_KEY_FALLBACK"]  # lower quota, same team

async def invoke_with_rotation(messages: list, model: str = "claude-sonnet-37"):
    for key in [PRIMARY_KEY, FALLBACK_KEY]:
        try:
            response = await litellm.acompletion(
                model=model,
                messages=messages,
                api_key=key,
                api_base="https://litellm.internal",
            )
            return response
        except litellm.BudgetExceededError:
            continue
    raise RuntimeError("All keys exhausted — escalate to FinOps")

Pattern 2 — LiteLLM-side team rotation via admin API:

async def rotate_exhausted_key(exhausted_key_hash: str, team_id: str):
    """
    Called by a Lambda triggered by LiteLLM webhook on BudgetExceeded event.
    Creates a new key for the team and posts credentials to Slack.
    """
    async with httpx.AsyncClient() as client:
        # Disable exhausted key
        await client.post(
            "https://litellm.internal/key/block",
            headers={"Authorization": f"Bearer {os.environ['LITELLM_MASTER_KEY']}"},
            json={"key": exhausted_key_hash},
        )

        # Create new key (budget resets)
        new_key_resp = await client.post(
            "https://litellm.internal/key/generate",
            headers={"Authorization": f"Bearer {os.environ['LITELLM_MASTER_KEY']}"},
            json={
                "team_id": team_id,
                "max_budget": float(os.environ["TEAM_MONTHLY_BUDGET"]),
                "soft_budget": float(os.environ["TEAM_MONTHLY_BUDGET"]) * 0.8,
                "budget_duration": "30d",
                "key_alias": f"{team_id}-auto-rotated",
            }
        )

        new_key = new_key_resp.json()["key"]
        await _post_new_key_to_slack(team_id, new_key)
        # In practice: write to AWS Secrets Manager, not Slack
        await _store_in_secrets_manager(team_id, new_key)

Posting new credentials via Secrets Manager (recommended over Slack):

async def _store_in_secrets_manager(team_id: str, new_key: str):
    secrets = boto3.client("secretsmanager")
    secret_name = f"litellm/team-keys/{team_id}"
    try:
        secrets.put_secret_value(
            SecretId=secret_name,
            SecretString=json.dumps({"api_key": new_key, "rotated_at": str(date.today())})
        )
    except secrets.exceptions.ResourceNotFoundException:
        secrets.create_secret(
            Name=secret_name,
            SecretString=json.dumps({"api_key": new_key}),
        )

Application code pulls the key from Secrets Manager at startup, so rotation is transparent.

5.6 Provider Budget Routing

LiteLLM v1.75+ supports per-provider budget routing: when a provider (Bedrock, in this case) exhausts a configured daily budget, LiteLLM automatically routes to the next provider in the list:

# config.yaml — provider budget routing
router_settings:
  provider_budget_config:
    bedrock:
      budget_limit: 500.0       # USD per budget_duration
      budget_duration: "1d"     # resets daily
    openai:
      budget_limit: 200.0
      budget_duration: "1d"

model_list:
  - model_name: "claude-sonnet-37"
    litellm_params:
      model: "bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0"

  - model_name: "claude-sonnet-37"   # same logical name, different provider
    litellm_params:
      model: "openai/gpt-4o"         # fallback if Bedrock daily budget exhausted

When Bedrock’s budget_limit is hit, LiteLLM routes subsequent requests to openai/gpt-4o without the application seeing an error. This is a cost management pattern, not a reliability pattern — it changes the model being used.

6. Terraform / CDK IaC Patterns

6.1 Complete Terraform Module — Bedrock FinOps Stack

# terraform/bedrock_finops/main.tf

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.50"
    }
  }
}

variable "monthly_budget_usd"        { default = 10000 }
variable "anomaly_alert_threshold"   { default = 300 }   # USD absolute
variable "slack_webhook_ssm_path"    { default = "/finops/slack-webhook-url" }
variable "environment"               { default = "production" }

# ─── SNS Topic ───────────────────────────────────────────────────────────────

resource "aws_sns_topic" "bedrock_finops" {
  name = "bedrock-finops-${var.environment}"
}

resource "aws_sns_topic_policy" "allow_cost_anomaly" {
  arn = aws_sns_topic.bedrock_finops.arn
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Sid       = "AllowCostAnomalyDetection"
      Effect    = "Allow"
      Principal = { Service = "costalerts.amazonaws.com" }
      Action    = "SNS:Publish"
      Resource  = aws_sns_topic.bedrock_finops.arn
    }]
  })
}

# ─── Cost Anomaly Detection ──────────────────────────────────────────────────

resource "aws_ce_anomaly_monitor" "bedrock" {
  name         = "Bedrock-${var.environment}"
  monitor_type = "DIMENSIONAL"
  monitor_dimension = "SERVICE"
}

resource "aws_ce_anomaly_subscription" "bedrock_immediate" {
  name      = "Bedrock-Immediate-Alert-${var.environment}"
  frequency = "IMMEDIATE"
  monitor_arn_list = [aws_ce_anomaly_monitor.bedrock.arn]

  threshold_expression {
    and {
      dimension {
        key           = "ANOMALY_TOTAL_IMPACT_ABSOLUTE"
        match_options = ["GREATER_THAN_OR_EQUAL"]
        values        = [tostring(var.anomaly_alert_threshold)]
      }
    }
  }

  subscriber {
    address = aws_sns_topic.bedrock_finops.arn
    type    = "SNS"
  }
}

resource "aws_ce_anomaly_subscription" "bedrock_daily" {
  name      = "Bedrock-Daily-Digest-${var.environment}"
  frequency = "DAILY"
  monitor_arn_list = [aws_ce_anomaly_monitor.bedrock.arn]

  threshold_expression {
    and {
      dimension {
        key           = "ANOMALY_TOTAL_IMPACT_PERCENTAGE"
        match_options = ["GREATER_THAN_OR_EQUAL"]
        values        = ["40"]
      }
    }
  }

  subscriber {
    address = aws_sns_topic.bedrock_finops.arn
    type    = "SNS"
  }
}

# ─── CloudWatch Alarms ───────────────────────────────────────────────────────

locals {
  bedrock_models = {
    "sonnet-37"  = "anthropic.claude-3-7-sonnet-20250219-v1:0"
    "haiku-35"   = "anthropic.claude-3-5-haiku-20241022-v1:0"
  }
}

resource "aws_cloudwatch_metric_alarm" "throttle_high" {
  for_each = local.bedrock_models

  alarm_name          = "Bedrock-${each.key}-ThrottleHigh-${var.environment}"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  evaluation_periods  = 3
  metric_name         = "InvocationThrottles"
  namespace           = "AWS/Bedrock"
  period              = 300
  statistic           = "Sum"
  threshold           = 10
  alarm_description   = "Bedrock throttling detected — consider quota increase"
  treat_missing_data  = "notBreaching"

  dimensions = {
    ModelId = each.value
  }

  alarm_actions = [aws_sns_topic.bedrock_finops.arn]
  ok_actions    = [aws_sns_topic.bedrock_finops.arn]
}

resource "aws_cloudwatch_metric_alarm" "quota_underused" {
  for_each = local.bedrock_models

  alarm_name          = "Bedrock-${each.key}-QuotaUnderutilized-${var.environment}"
  comparison_operator = "LessThanThreshold"
  evaluation_periods  = 168     # 7 days at 1h resolution
  metric_name         = "EstimatedTPMQuotaUsage"
  namespace           = "AWS/Bedrock"
  period              = 3600
  extended_statistic  = "p95"
  threshold           = 60.0
  alarm_description   = "Bedrock quota p95 < 60% for 7 days — consider reducing reservation"
  treat_missing_data  = "notBreaching"

  dimensions = {
    ModelId = each.value
  }

  alarm_actions = [aws_sns_topic.bedrock_finops.arn]
}

# ─── AWS Budget ──────────────────────────────────────────────────────────────

resource "aws_budgets_budget" "bedrock_monthly" {
  name         = "Bedrock-Monthly-${var.environment}"
  budget_type  = "COST"
  limit_amount = tostring(var.monthly_budget_usd)
  limit_unit   = "USD"
  time_unit    = "MONTHLY"

  cost_filter {
    name   = "Service"
    values = ["Amazon Bedrock"]
  }

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 80
    threshold_type             = "PERCENTAGE"
    notification_type          = "ACTUAL"
    subscriber_sns_topic_arns  = [aws_sns_topic.bedrock_finops.arn]
  }

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 95
    threshold_type             = "PERCENTAGE"
    notification_type          = "ACTUAL"
    subscriber_sns_topic_arns  = [aws_sns_topic.bedrock_finops.arn]
  }

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 100
    threshold_type             = "PERCENTAGE"
    notification_type          = "FORECASTED"
    subscriber_sns_topic_arns  = [aws_sns_topic.bedrock_finops.arn]
  }
}

# ─── IAM Deny Policy for Budget Action ───────────────────────────────────────

resource "aws_iam_policy" "deny_bedrock_inference" {
  name        = "DenyBedrockInference-${var.environment}"
  description = "Applied automatically by Budget Action at 95% spend"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Sid    = "DenyBedrockInference"
      Effect = "Deny"
      Action = [
        "bedrock:InvokeModel",
        "bedrock:InvokeModelWithResponseStream",
        "bedrock:Converse",
        "bedrock:ConverseStream",
      ]
      Resource = "*"
    }]
  })
}

resource "aws_iam_role" "budget_execution" {
  name = "budgets-bedrock-execution-${var.environment}"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "budgets.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy" "budget_execution_permissions" {
  role = aws_iam_role.budget_execution.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = ["iam:AttachRolePolicy", "iam:DetachRolePolicy"]
      Resource = "arn:aws:iam::*:role/bedrock-inference-role"
    }]
  })
}

resource "aws_budgets_budget_action" "deny_on_95pct" {
  budget_name        = aws_budgets_budget.bedrock_monthly.name
  action_type        = "APPLY_IAM_POLICY"
  approval_model     = "AUTOMATIC"
  notification_type  = "ACTUAL"
  execution_role_arn = aws_iam_role.budget_execution.arn

  action_threshold {
    action_threshold_type  = "PERCENTAGE"
    action_threshold_value = 95
  }

  definition {
    iam_action_definition {
      policy_arn = aws_iam_policy.deny_bedrock_inference.arn
      roles      = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/bedrock-inference-role"]
    }
  }

  subscriber {
    address           = aws_sns_topic.bedrock_finops.arn
    subscription_type = "SNS"
  }
}

data "aws_caller_identity" "current" {}

# ─── Lambda for Slack routing ────────────────────────────────────────────────

data "aws_ssm_parameter" "slack_webhook" {
  name            = var.slack_webhook_ssm_path
  with_decryption = true
}

resource "aws_lambda_function" "finops_router" {
  function_name = "bedrock-finops-router-${var.environment}"
  runtime       = "python3.12"
  handler       = "handler.main"
  role          = aws_iam_role.lambda_exec.arn
  filename      = "${path.module}/lambda_router.zip"
  timeout       = 30

  environment {
    variables = {
      SLACK_WEBHOOK_URL = data.aws_ssm_parameter.slack_webhook.value
    }
  }
}

resource "aws_sns_topic_subscription" "lambda_to_sns" {
  topic_arn = aws_sns_topic.bedrock_finops.arn
  protocol  = "lambda"
  endpoint  = aws_lambda_function.finops_router.arn
}

resource "aws_lambda_permission" "allow_sns" {
  statement_id  = "AllowSNSInvoke"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.finops_router.function_name
  principal     = "sns.amazonaws.com"
  source_arn    = aws_sns_topic.bedrock_finops.arn
}

resource "aws_iam_role" "lambda_exec" {
  name = "bedrock-finops-lambda-${var.environment}"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "lambda.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy_attachment" "lambda_basic" {
  role       = aws_iam_role.lambda_exec.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

6.2 Outputs

output "finops_sns_topic_arn" {
  value = aws_sns_topic.bedrock_finops.arn
}

output "anomaly_monitor_arn" {
  value = aws_ce_anomaly_monitor.bedrock.arn
}

output "deny_bedrock_policy_arn" {
  value = aws_iam_policy.deny_bedrock_inference.arn
}

6.3 Minimum IAM Permissions Checklist

The role running terraform apply for this module requires:

ce:CreateAnomalyMonitor
ce:DeleteAnomalyMonitor
ce:CreateAnomalySubscription
ce:DeleteAnomalySubscription
ce:CreateCostCategoryDefinition
ce:UpdateCostCategoryDefinition
budgets:CreateBudget
budgets:ModifyBudget
budgets:CreateBudgetAction
budgets:UpdateBudgetAction
budgets:DeleteBudgetAction
sns:CreateTopic
sns:SetTopicAttributes
sns:Subscribe
sns:DeleteTopic
cloudwatch:PutMetricAlarm
cloudwatch:DeleteAlarms
iam:CreatePolicy
iam:DeletePolicy
iam:CreateRole
iam:AttachRolePolicy
iam:DetachRolePolicy
lambda:CreateFunction
lambda:UpdateFunctionCode
lambda:AddPermission
ssm:GetParameter

6.4 Athena Workgroup and Glue Catalog via Terraform

resource "aws_athena_workgroup" "finops" {
  name = "bedrock-finops"

  configuration {
    result_configuration {
      output_location = "s3://${aws_s3_bucket.athena_results.bucket}/queries/"
      encryption_configuration {
        encryption_option = "SSE_S3"
      }
    }
    bytes_scanned_cutoff_per_query     = 10737418240   # 10 GB safety cutoff
    publish_cloudwatch_metrics_enabled = true
  }
}

resource "aws_glue_crawler" "cur_crawler" {
  name          = "cur-bedrock-crawler"
  role          = aws_iam_role.glue_crawler.arn
  database_name = aws_glue_catalog_database.cur.name

  s3_target {
    path = "s3://${var.cur_bucket_name}/${var.cur_prefix}/"
  }

  schedule = "cron(0 6 * * ? *)"   # daily at 06:00 UTC

  schema_change_policy {
    delete_behavior = "LOG"
    update_behavior = "UPDATE_IN_DATABASE"
  }
}

resource "aws_glue_catalog_database" "cur" {
  name = "aws_billing"
}

Operational Runbook — Decision Tree

Bedrock cost alert fires
│
├─ Source: Cost Anomaly Detection
│   ├─ Check root cause in AWS Cost Explorer → drill by Service → Bedrock
│   ├─ If anomaly is model-specific: query CUR 2.0 by product_model_id for the date range
│   ├─ If anomaly is team-specific: query by resource_tags_user_team or line_item_iam_principal
│   └─ Escalate if > $1000: engage FinOps lead + model team owner
│
├─ Source: InvocationThrottles alarm
│   ├─ Check EstimatedTPMQuotaUsage for the same model in the same window
│   ├─ If quota usage was < 80% before throttle: likely burst spike → add retry-with-jitter
│   ├─ If quota usage was > 90% sustained: submit Service Quotas increase request
│   └─ Consider cross-region inference profile for automatic burst absorption
│
├─ Source: LiteLLM BudgetExceeded (HTTP 429 from proxy)
│   ├─ Check which team/key exhausted budget in LiteLLM UI or /spend/logs API
│   ├─ If legitimate workload growth: request budget increase from FinOps lead
│   ├─ If anomalous: block key immediately, investigate usage logs
│   └─ For planned events (bulk jobs): pre-create a separate high-budget key with expiry
│
└─ Source: Budget Action fired (IAM Deny attached)
    ├─ All Bedrock inference now blocked for the inference role
    ├─ To re-enable: detach the deny policy manually or via budgets:ExecuteBudgetAction (REVERSE)
    └─ Root-cause before re-enabling — do not reverse without understanding the overage

Known Gaps and Caveats (2026)

Gap Workaround
CAD cannot alert on per-model spend spikes without cost allocation tags Use AIPs with Team + Model tags; activate tags in Billing console
AWS Budgets data is 24h stale — cannot stop real-time runaway jobs LiteLLM hard limits are the real-time control; Budgets is the billing-layer backstop
Provisioned Throughput by Model Units cannot be resized after purchase Use no-term (on-demand) Provisioned Throughput for workloads where you need resize flexibility
EstimatedTPMQuotaUsage is per-account, not per-model, in some regions Verify metric dimensionality in your region; use ModelId dimension where available
LiteLLM spend tracking requires PostgreSQL; SQLite is not suitable for multi-replica production Budget PostgreSQL with connection pooling (PgBouncer) for proxy fleets > 3 replicas
IAM principal cost attribution requires caller to use distinct roles or session tags Single-role gateway patterns hide attribution; session tags are the minimum viable fix
Cost Categories SplitChargeRules max 10 sources For organizations with > 10 shared Bedrock services, aggregate before splitting