← Agent Frameworks 🕐 22 min read
Agent Frameworks

Bedrock Inference Cost Optimization Playbook — Prioritized Checklist (2026)

Tier 1 — Zero-Cost Visibility Wins (Do in First Week) · March 2026

Nova Micro vs Claude Sonnet 4.6: **107× price differential** on output tokens. Routing even 20% of traffic from Sonnet to Nova Micro for simple tasks produces material savings at any scale.

Audience: FinOps practitioners inheriting an existing Bedrock deployment. Assumes active production traffic. Not a setup guide — a triage and optimization guide.

Orientation: Most teams overspend 50–100% in their first 6 months with Bedrock. The highest-ROI moves cost nothing to implement and take under a week. This playbook sequences them by effort-to-impact ratio.

Key pricing anchors (June 2026):

Model Input ($/MTok) Output ($/MTok) Cache Read ($/MTok)
Claude Sonnet 4.6 $3.00 $15.00 $0.30
Claude Haiku 4.5 $1.00 $5.00 $0.10
Nova Pro $0.80 $3.20 $0.08
Nova Lite $0.06 $0.24 $0.006
Nova Micro $0.035 $0.14 $0.0035

Nova Micro vs Claude Sonnet 4.6: 107× price differential on output tokens. Routing even 20% of traffic from Sonnet to Nova Micro for simple tasks produces material savings at any scale.

These require no code changes, no architecture decisions, and zero direct cost. They are prerequisites for everything else — without them you are optimizing blind.

1.1 Enable CUR 2.0 with IAM Principal Opt-In

What: AWS Cost and Usage Report 2.0 exposes a line_item_iam_principal column that shows the full IAM ARN for every Bedrock inference call. As of April 2026, this includes user and role attribution natively.

How:

  1. Billing console → Data Exports → Create export → select “Cost and Usage Report 2.0”
  2. Check “Include caller identity (IAM principal) allocation data”
  3. Choose S3 destination bucket, set Parquet format for Athena efficiency
  4. Allow 24–48 hours for first full day to populate

Why it matters: Without this, every Bedrock call looks like one undifferentiated cost line from the service. With it, you can see which IAM role (app, team, microservice) is driving spend — the prerequisite for any chargeback or blame-free accountability conversation.

Common gap: Many teams have CUR 1.0 enabled but not CUR 2.0. CUR 1.0 does not include IAM principal data. Check your existing exports — if they predate April 2026, recreate them.

1.2 Activate Cost Allocation Tags on IAM Roles

What: Tag IAM roles and users with team, project, env, cost-center attributes, then activate those tags as cost allocation tags in Billing.

How:

  1. Tag every IAM role that calls Bedrock: aws iam tag-role --role-name <role> --tags Key=team,Value=ml-platform Key=project,Value=rag-pipeline Key=env,Value=prod
  2. Billing console → Cost Allocation Tags → activate each tag key
  3. Allow 24 hours for tags to begin appearing in CUR data

Naming discipline matters: Standardize tag key casing and allowed values across the org before activating. team=mlplatform, team=MLPlatform, and team=ml-platform appear as three separate buckets. Enforce via Service Control Policies if at org scale.

LiteLLM integration note: If you run an LLM gateway (LiteLLM or similar), the gateway calls Bedrock under a single IAM role, collapsing all per-user attribution. Fix: configure the gateway to call sts:AssumeRole per user/tenant with --role-session-name set to the user identity and --tags set to their attributes. The session name and tags propagate to CUR.

1.3 Create a Basic Athena View Over CUR 2.0

What: A saved query that surfaces daily Bedrock spend by model, IAM principal, and region. Costs nothing to run; you pay only per-query Athena fees (~$5/TB scanned, typically <$1/month for this query pattern).

Starter query:

SELECT
  line_item_usage_start_date,
  line_item_resource_id,
  line_item_iam_principal,
  product_region,
  line_item_usage_type,
  SUM(line_item_blended_cost) AS daily_cost,
  SUM(line_item_usage_amount) AS token_volume
FROM cur2_data
WHERE product_service_name = 'Amazon Bedrock'
  AND line_item_usage_start_date >= date_add('day', -30, current_date)
GROUP BY 1, 2, 3, 4, 5
ORDER BY 6 DESC;

Save as a named view. This becomes the foundation for dashboards, anomaly investigation, and optimization tracking.

Key dimensions to add when you have time:

  • line_item_usage_type contains Input, Output, CacheRead, CacheWrite tokens — split these to see cache efficiency per workload
  • Join against your resource tag data to get team/project attribution

1.4 Set Up Cost Anomaly Detection

What: AWS Cost Anomaly Detection uses ML to detect unusual spend patterns and alerts before you see them in the monthly bill.

How:

  1. Cost Explorer → Cost Anomaly Detection → Create monitor
  2. Select AWS Service monitor type, choose Amazon Bedrock
  3. Set alert threshold: absolute ($50 for dev accounts, $500+ for production) or percentage (20% week-over-week)
  4. Subscribe via SNS → Slack or PagerDuty webhook

Also create: a second monitor scoped to bedrock:modelId cost allocation tag to catch per-model anomalies (e.g., a new team spinning up Opus 4 workloads without governance approval).

What this catches in practice:

  • A new feature deployment that inadvertently calls the wrong model
  • A feedback loop where outputs become inputs (agent amplification)
  • Dev/staging environment leaking into production billing
  • A batch job that re-ran after a failure, doubling token spend

Budget guardrails: After anomaly detection, also set AWS Budgets alerts at 80% and 100% of monthly Bedrock allocation. These are coarse but trigger for billing teams who may not monitor anomaly detection.

1.5 Pull the First Diagnostic Report

Before touching any code, run this diagnostic to identify where optimization effort should go:

Questions to answer from CUR 2.0 data:

  1. What is the top-5 IAM principals by Bedrock cost? Are they expected?
  2. What is the model distribution? What % of spend is on Opus or Sonnet vs Haiku/Nova?
  3. What is the cache hit rate by workload? (CacheRead tokens / total input tokens)
  4. What % of calls happen during business hours vs overnight? (batch opportunity signal)
  5. What is the input/output token ratio? High output ratio = summarization/generation workload, harder to reduce. High input ratio = RAG or long-context workload, better caching opportunity.

Write down answers. These become the prioritization filter for Tiers 2–4.

Tier 2 — Quick Wins (< 1 Week to Implement, Significant ROI)

2.1 Model Right-Sizing Audit

The 107× cost spread across the model catalog makes right-sizing the single highest-ROI intervention for most teams.

Step 1 — Classify every workload by task type:

Task Type Appropriate Model Over-served by
Classification, routing, intent detection Nova Micro or Nova Lite Anything above Haiku
Structured extraction from documents Nova Lite or Haiku 4.5 Sonnet
RAG Q&A (factual, bounded) Haiku 4.5 or Nova Pro Sonnet
Customer-facing chat (complex reasoning) Sonnet 4.6 Opus (usually)
Code generation, multi-step reasoning Sonnet 4.6 Opus (often)
Research synthesis, novel reasoning Opus 4.5 Correct tier

Step 2 — Run a shadow comparison test:

  • For each workload currently on Sonnet or above: replay 200 representative prompts against the next tier down (Haiku or Nova Pro)
  • Score against your existing eval set or by human review of 20 sample outputs
  • Accept downgrade if accuracy delta < 3% on your task; reject if it matters

Step 3 — Migrate and tag:

  • Update model IDs in code for accepted downgrades
  • Tag the workload IAM role with approved-model-tier=nova-lite to prevent future drift back to premium tiers
  • Track cost delta in CUR after 1 week to confirm savings

Typical outcome: Teams find 30–50% of Sonnet traffic is classification, routing, or bounded Q&A tasks that Nova Pro handles with <1% accuracy loss. At a 5× cost reduction per token (Sonnet → Nova Pro) on those workloads, blended monthly bill drops 20–35%.

Intelligent Prompt Routing (Bedrock native): AWS offers a managed version of this via Intelligent Prompt Routing — configure a router ARN that spans Claude Sonnet and Haiku, and Bedrock automatically routes each request to the cheaper model when the prompt complexity permits. Reported savings: 30–94% on applicable workloads, with no accuracy regression on straightforward prompts. Use as a complement to manual right-sizing, not a replacement — the router only operates within a model family and does not cross to Nova.

2.2 Prompt Caching Audit

The highest-ROI single feature for RAG, agent, and document-processing workloads.

Cache read pricing (June 2026): ~90% cheaper than standard input tokens. Claude Sonnet cache read = $0.30/MTok vs $3.00/MTok standard.

Identify caching candidates from CUR data:

  • High input token volume per call (>2,000 tokens)
  • Repeated calls with similar prefixes (same system prompt, same document context, same tool definitions)
  • Low or zero CacheRead tokens in current usage (means caching is off or misconfigured)

What to cache:

  1. System prompts — if your system prompt is >500 tokens and the same across many requests, cache it. Set cache_control: {"type": "ephemeral"} on the system message block.
  2. Tool/function definitions — agent frameworks pass tool schemas on every call. Cache the tool block.
  3. Document context (RAG) — if the same document is referenced across multiple turns, structure your prompt so the document appears before the user query and mark it for caching.
  4. Few-shot examples — classification prompts with many examples benefit enormously from caching the example block.

Cache TTL: 1-hour TTL (default as of January 2026 for Claude 4.x and Nova models). For documents accessed less frequently than hourly, caching provides diminishing returns — evaluate batch inference instead (Section 2.3).

Cache write overhead: Cache writes cost 25% more than standard input tokens. The math is positive after approximately 1.3 re-reads of the cached content. Any content used more than twice per hour is worth caching.

Expected impact: Workloads with long repeated context (RAG with static docs, agent loops with large tool definitions) commonly see 50–80% reduction in input token costs after caching is enabled. The Caylent benchmark shows 85% latency improvement as a secondary benefit.

2.3 Batch Inference Migration

50% discount vs on-demand, zero code complexity for offline workloads.

Identify batch candidates: Any workload where the user or system does not need a synchronous response:

  • Nightly document summarization
  • Bulk content classification or tagging
  • Scheduled report generation
  • Offline evaluation jobs
  • Data enrichment pipelines

How to migrate:

  1. Prepare input: JSONL file with one request per line, each with recordId and modelInput
  2. Upload to S3
  3. Submit via bedrock:CreateModelInvocationJob API or Bedrock console
  4. Poll for completion (typical SLA: within 24 hours; usually 2–6 hours for sub-100K request jobs)
  5. Read output JSONL from S3

Limitations: No streaming, no tool use in batch mode (as of June 2026), response latency up to 24 hours. Not suitable for user-facing workloads.

Typical batch-eligible traffic: In mature deployments, 30–60% of Bedrock token volume qualifies for batch. Moving that volume to batch produces a 15–30% reduction in total monthly bill.

2.4 Flex Tier Migration for Latency-Insensitive Workloads

~50% discount vs standard on-demand for best-effort, non-latency-sensitive traffic.

Flex tier characteristics:

  • Best-effort delivery: requests may be throttled during peak demand
  • No guaranteed throughput or latency
  • No SLA commitment from AWS
  • Roughly 50% of on-demand pricing

Good candidates:

  • Background enrichment pipelines (latency tolerance: minutes to hours)
  • Internal tools used during off-peak hours
  • Dev and staging environments
  • A/B test traffic that doesn’t need real-time scoring

Bad candidates:

  • User-facing chat (latency matters)
  • Any workload with a retry budget that would multiply costs under throttling
  • Workloads where a queued job failure creates downstream pipeline failures

Implementation: Use Flex inference profile ARNs when calling InvokeModel. Bedrock exposes these per-model; check the Bedrock console under “Inference Profiles” for your target models.

Tier 3 — Medium Effort Wins (1–4 Weeks)

3.1 Application Inference Profiles for Attribution

What: Application Inference Profiles (AIPs) are named, tagged wrappers around a foundation model. Calls made through an AIP carry the AIP’s cost allocation tags into CUR.

Why this matters for FinOps: Without AIPs, all calls to anthropic.claude-sonnet-4-6 from all teams look identical in billing. With AIPs, you create team-a-sonnet-prod, team-b-sonnet-dev, and each generates separately attributable cost lines.

How to set up:

  1. Create one AIP per application/team/environment combination that matters for attribution
  2. Attach tags: team, project, env, cost-center
  3. Update application code to use AIP ARN instead of base model ID — it’s a drop-in substitution
  4. Activate AIP tags as cost allocation tags in Billing

Governance benefit: Enforce via IAM policy. Grant each team’s IAM role access only to their own AIP ARNs. This prevents accidental or unauthorized model tier upgrades (a team using Sonnet when they should use Haiku).

Setup time: 1–2 days for a team of 5–10 application owners. Ongoing maintenance is low.

3.2 LiteLLM Deployment for Centralized Governance

What: LiteLLM is an open-source LLM proxy that provides a unified OpenAI-compatible API surface in front of Bedrock (and other providers). Deployed internally, it becomes the single choke point for all LLM spend.

FinOps capabilities:

  • Per-user and per-team spend tracking in a local database
  • Budget limits per team/user with hard cutoffs
  • Model routing policy enforcement (prevent teams from calling Opus without approval)
  • Audit log of every LLM call with token counts, model, latency, user identity
  • Automatic fallback: if primary model is throttled, route to backup model
  • Cost dashboards without requiring Athena/CUR setup

Architecture:

Application → LiteLLM Proxy (ECS/K8s) → Bedrock API
                    ↓
            Per-user budget DB (Redis/Postgres)
            Audit log (S3 or CloudWatch)

IAM chargeback fix for gateway pattern: Configure LiteLLM to assume a per-team Bedrock role using the team’s identifier as the session name and cost-center as a session tag. This propagates team identity into CUR even though the gateway is a single IAM principal.

When to deploy: When you have ≥5 teams or applications independently calling Bedrock and governance through AIP tagging alone is insufficient. Also valuable when you need multi-provider routing (Bedrock + OpenAI + Azure) under one policy layer.

Effort: 3–5 days to deploy, configure per-team budgets, and migrate application endpoints.

3.3 Prompt Compression for Token-Heavy Workloads

What: Systematic reduction of token volume in prompts before caching is applied. Reduces both cache write costs and the uncached token costs for workloads that don’t qualify for caching.

Techniques, roughly ordered by ROI:

  1. System prompt audit — Review every system prompt for verbosity. Typical enterprise system prompts have 20–40% removable redundancy: repeated instructions, excessive examples, boilerplate disclaimers. A 3,000-token prompt reduced to 1,800 tokens cuts input costs 40% before any other change.

  2. RAG chunk sizing — Many RAG implementations pass 3–5 full document chunks per query. Reduce chunk size (1,000 → 600 tokens), add a reranking step to pass only the top 2 chunks, and implement a confidence gate to skip retrieval for high-confidence cached answers. Typical result: 30–50% reduction in RAG input token volume.

  3. Context window pruning — Multi-turn conversations accumulate context. Implement a sliding window (keep last N turns) or a compression step (summarize older turns into a brief). Without this, token costs grow linearly with conversation length; with it, costs stabilize.

  4. Prompt templating discipline — Enforce that prompts use templates with variable slots rather than string concatenation. Templates expose token volume to static analysis; concatenated strings do not. Static analysis of templates lets you catch bloat before it ships.

  5. AWS Prompt Compression for RAG — AWS Builder Center documents a native Bedrock prompt compression pattern for Knowledge Base queries that selectively compresses the retrieved context before injection. Reduces retrieved context by 30–60% with minimal answer quality degradation on factual Q&A.

Tooling: tiktoken (OpenAI tokenizer, approximates Bedrock token counts within 5%) for offline analysis. For Claude specifically, use the Anthropic token counting API endpoint before submitting to get exact counts.

3.4 Cross-Region Inference Profiles (CRIS) for Quota Bursting vs Reserved Tier Decision

What: Cross-Region Inference dynamically routes requests across multiple AWS regions, expanding effective throughput limits by up to 2× the in-region quota.

Two CRIS types:

  • Geographic CRIS — stays within a geographic boundary (e.g., US only). Useful for data residency requirements.
  • Global CRIS — routes worldwide. Approximately 10% cost savings vs geographic for Claude Sonnet 4.5 (as of June 2026) because off-peak regions cost less.

FinOps relevance — the CRIS vs Reserved Tier decision:

Situation Recommendation
Occasional bursts >2× baseline, need quota headroom CRIS (no commitment)
Sustained load ≥80% utilization on a specific model/region Reserved (Provisioned) Throughput
Mixed: steady base + occasional bursts Reserved base + CRIS for overflow
Unpredictable growth trajectory CRIS until pattern stabilizes, then re-evaluate

CRIS has no additional routing cost. You pay the source region on-demand rate. No commitment required. This makes it the default for any team not yet confident in their usage pattern.

Compliance caveat: If your data cannot leave a specific jurisdiction, Global CRIS is off the table. Geographic CRIS with an explicit region boundary list is the alternative.

CRIS token cost tracking note: When using cross-region routing, CUR records the cost at the source region’s rate, but the line_item_resource_id will reflect the region the request was actually served from. Add a region filter to your Athena views to avoid double-counting when CRIS routes a request from us-east-1 to eu-west-1 — both will appear in your CUR if you have multi-region CUR exports enabled. Recommended: use a single primary region CUR export and use the product_region column to track actual serve location.

Tier 4 — Strategic Investments (1–3 Months)

4.1 Fine-Tuning and Model Distillation Assessment

When it pays: High-volume (≥100K requests/month), well-defined, repetitive tasks where a smaller model consistently falls short of acceptable quality on the raw task but could be trained to match.

Two paths:

Custom fine-tuning:

  • Upload labeled training data to Bedrock
  • Fine-tune a base model (Nova Lite, Haiku) on your specific task
  • Inference from the fine-tuned model under Provisioned Throughput
  • Cost structure: one-time training cost + monthly storage + hourly MU cost for inference
  • ROI break-even: typically 2–4 months at 100K requests/month; shorter at higher volumes

Model Distillation (Bedrock native):

  • Use a frontier teacher model (Sonnet or Opus) to generate synthetic training data for a student model (Nova Micro or Haiku)
  • AWS reports: distilled models are up to 75% cheaper and 500% faster than the teacher model, with <2% accuracy loss on RAG-style tasks
  • Cost structure: teacher inference cost for data generation + fine-tuning cost + Provisioned Throughput for student inference
  • Best for: structured extraction, classification, RAG Q&A with bounded answer space

Distillation ROI example (illustrative):

  • 500K monthly requests on Claude Sonnet 4.6 at average 2K input / 500 output tokens = ~$18,750/month
  • Post-distillation on Nova Micro via Provisioned Throughput: ~$3,000–4,000/month
  • Distillation setup cost: ~$2,000–5,000 one-time
  • Break-even: ~1 month

Do not distill if: the task requires novel reasoning, open-ended generation, or handles a highly variable input distribution. Distillation compresses quality on edge cases.

4.2 Reserved Tier (Provisioned Throughput) Commitment

What: Provisioned Throughput reserves dedicated Model Units (MUs) at a fixed hourly rate, providing predictable performance and 15–40% cost reduction vs on-demand at sustained utilization.

Commitment options:

  • 1-month: ~15–20% discount vs on-demand
  • 6-month: ~30–40% discount vs on-demand

The utilization math:

  • Provisioned throughput is economical at ≥80% sustained utilization
  • At 60% utilization, the fixed hourly cost roughly matches on-demand
  • Below 60%, on-demand or batch are cheaper

Qualification criteria for Reserved Tier:

  1. The workload has been in production ≥3 months with stable volume
  2. P50 utilization on the target model exceeds 75% during business hours
  3. The workload runs on a single model version (Reserved is model-version specific; a model upgrade requires a new reservation)
  4. No pending major model migrations (e.g., upgrading from Sonnet 4.5 to 4.6) within the reservation window

Recommended path: Start with a 1-month commitment to validate the pattern before locking into 6 months. If the utilization holds, convert to 6-month for maximum discount.

Mixed strategy: Provision for your steady-state load (P50); use on-demand + CRIS for the P90–P99 burst. This captures reservation discount on the base without over-committing.

4.3 OpenCost/Kubecost for Kubernetes AI Platform Cost Visibility

Context: If your team runs the Bedrock-calling applications on EKS, the Bedrock API cost is only part of the picture. GPU-backed preprocessing, vector database pods, embedding generation, and inference orchestration all appear as undifferentiated compute cost in the K8s layer.

OpenCost: CNCF sandbox project; open source; provides namespace/pod/deployment cost allocation by mapping K8s resource requests to cloud pricing. Free but requires self-hosting and lacks enterprise features.

Kubecost: Enterprise product built on OpenCost internals; adds chargeback reports, savings recommendations, multi-cluster support, and finance-accurate billing reconciliation. $0 for clusters under 25 nodes; paid above that.

Integration pattern for AI platforms:

  1. Deploy OpenCost or Kubecost on EKS
  2. Label all AI-platform namespaces: team=ml-platform, project=rag-pipeline
  3. Allocate Bedrock API costs via the CUR 2.0 IAM tag data (separate pipeline) and reconcile against K8s-level infrastructure costs
  4. Export combined report to cost dashboard (Grafana, QuickSight)

What this unlocks: True cost-per-inference visibility that includes compute, storage, networking, and API costs in a single number. Required for FinOps maturity level 3+ and for accurate chargeback to product teams.

Note on accuracy: Both tools price at on-demand list rates by default. For finance-accurate showback that reconciles with the actual AWS invoice (including Savings Plans, RIs, and Flex pricing), layer in Cost and Usage Report reconciliation.

4.4 FinOps Governance Org and Chargeback Program

The last-mile problem: All the technical optimizations above are undermined if no human owns the cost signal and no team feels the financial consequence of their model choices.

Minimum viable governance program:

  1. Model tier policy — Published document: which model tier is approved for which task class. Enforcement via IAM (AIP scoping) + LiteLLM budget limits. Exceptions require a named approver.

  2. Monthly review cadence — 30-minute monthly review with each team lead: show their Bedrock spend trend, cache hit rate, model distribution. Not a blame meeting — a signal review.

  3. Experimentation budget — A named budget line for R&D Bedrock usage, separate from production. This prevents experimentation cost from polluting production optimization signals. Typical size: 10–15% of production Bedrock spend.

  4. Chargeback or showback — At minimum, showback (show teams their costs, no actual billing transfer). At maturity, chargeback (transfer Bedrock cost to team P&L). Showback alone produces 15–25% voluntary cost reduction as teams become cost-aware.

  5. Savings tracking — Maintain a running ledger of optimizations implemented and dollars saved. This justifies FinOps headcount and creates organizational incentive to continue.

Anti-Patterns to Avoid

The 10 most common Bedrock cost mistakes encountered in production deployments:

1. Defaulting to Sonnet for All Tasks

The most expensive and most common mistake. Teams pick Claude Sonnet during prototyping because it works well, then ship it to production without a right-sizing step. Even a brief shadow test would move 30–50% of typical traffic to cheaper models.

2. Prompt Caching Not Enabled on Long-Context Workloads

Cache reads are 90% cheaper than standard input. Any RAG workload with >1,000-token context blocks, any agent with large tool definitions, and any system prompt over 500 tokens is a missed caching opportunity. Check CUR for CacheRead token volume — if it’s zero on high-input-token workloads, caching is off.

3. Batch-Eligible Traffic Running On-Demand

30–60% of enterprise Bedrock traffic is async-eligible (nightly summaries, bulk classification, data enrichment). Running this on-demand instead of batch pays a 2× premium for no benefit.

4. CloudWatch Full-Payload Logging at Scale

CloudWatch Logs ingestion costs $0.50/GB. A workload generating 1GB/day of prompt and response payloads adds $180/month in log costs — often unnoticed because it appears as a CloudWatch line item, not a Bedrock line item. Log metadata (token counts, model, latency) instead of full payloads. Sample full payloads at 1–5% for debugging.

5. Provisioned Throughput Under 80% Utilization

If reserved capacity sits below 60% utilization, you are paying the fixed hourly rate for idle MUs. Calculate your actual utilization before reserving. At sub-80% utilization, on-demand is cheaper.

6. Agent Amplification Without Token Budget Limits

Agent loops that call tools and re-invoke the model can multiply token costs by 5–20× vs a single call. Without a max_tokens ceiling on intermediate steps and a max_iterations limit on the agent loop, a single malformed request can cost $5–50. Set both limits. Monitor for outlier token volume per session.

7. Missing Cost Allocation Tags on IAM Roles

Without tags on the IAM roles calling Bedrock, every spend analysis answer is “Amazon Bedrock, $X” with no attribution. This delays optimization by weeks because you cannot identify where to act. Tag roles before deploying workloads; retrofitting is painful.

8. Treating Development and Production as One Cost Pool

Dev and staging traffic is often the fastest-growing and least-governed category. Run dev workloads on Nova Micro or Haiku regardless of what production uses. Enforce via AIP scoping: dev IAM roles can only access dev-tier AIPs.

9. Ignoring Knowledge Base Infrastructure Costs

Bedrock Knowledge Bases add 30–80% to inference costs through vector store hosting (OpenSearch Serverless starts at ~$700/month), embedding generation (charged per token), and S3 sync operations. Many teams see “Bedrock API” costs flatten after optimization but miss that Knowledge Base infrastructure costs continue growing. Include these in the total cost model.

10. Skipping the IAM Principal Opt-In in CUR 2.0

This requires an explicit checkbox at CUR creation time (Section 1.1). Many CUR 2.0 reports were set up before April 2026 without it, or teams created CUR 2.0 from the default wizard without reading the options. If your CUR data shows no line_item_iam_principal column, recreate the export with the opt-in enabled.

Quick Wins ROI Table

Typical savings ranges by technique and monthly Bedrock spend level. Ranges reflect real-world variance based on workload type and baseline configuration.

Technique Effort $10K/mo savings $50K/mo savings $200K/mo savings
Model right-sizing (Sonnet → Haiku/Nova) Low 20–40% 20–40% 20–40%
Prompt caching (RAG/agent workloads) Low 40–80%* 40–80%* 40–80%*
Batch inference migration Low 10–20% 15–25% 20–30%
Flex tier (latency-insensitive) Low 5–15% 8–20% 10–25%
Intelligent Prompt Routing Low 10–30% 15–35% 20–40%
Prompt compression Medium 10–30% 10–30% 10–30%
Provisioned Throughput (≥80% utilization) Medium 15–20% 20–30% 30–40%
CloudWatch log optimization Low 2–5% 3–8% 5–15%**
Model distillation High — 40–70%* 50–80%*
Chargeback/showback program Medium 10–20%*** 15–25%*** 15–25%***

* Applies only to eligible workloads (repeated context, high volume). Will not apply to all traffic. ** Log cost as % of total Bedrock bill grows at higher volumes. *** Behavioral change effect; takes 1–3 months to materialize.

Blended potential: A team running $50K/month with no prior optimization that implements Tiers 1–2 fully can typically reach $25–35K/month within 30 days. Tier 3 additions can push to $18–22K/month. Tier 4 (at scale, with distillation) can reach $12–18K/month.

Decision Flowchart: Given a Cost Problem Type, Which Technique Addresses It

COST PROBLEM TYPE
│
├── "I don't know where the money is going"
│   └── Tier 1: CUR 2.0 + IAM tags + Athena view
│
├── "One model is consuming most of the budget"
│   ├── Check: Is the model appropriate for the task?
│   │   ├── No → Model right-sizing (2.1) or Intelligent Prompt Routing
│   │   └── Yes → Check utilization
│   │       ├── ≥80% sustained → Provisioned Throughput (4.2)
│   │       └── Variable/bursty → CRIS (3.4) + on-demand mix
│
├── "Input token costs are high"
│   ├── Check: Does the workload repeat the same context?
│   │   ├── Yes (same doc, same system prompt, same tools) → Prompt caching (2.2)
│   │   └── No → Prompt compression (3.3) + RAG chunk optimization
│
├── "Output token costs are high"
│   ├── Check: Is this batch-eligible (no real-time requirement)?
│   │   ├── Yes → Batch inference (2.3) for 50% savings
│   │   └── No → Model right-sizing; output tokens are harder to reduce
│   └── Set max_tokens limits on all model calls to prevent runaway output
│
├── "Costs are unpredictable / spiking unexpectedly"
│   ├── Immediate: Cost Anomaly Detection (1.4) + budget alerts
│   ├── Root cause: Check for agent amplification (unbounded loops)
│   └── Governance: LiteLLM budget limits (3.2) per team/user
│
├── "Different teams have no visibility into their own spend"
│   ├── Quick: Application Inference Profiles + cost allocation tags (3.1)
│   └── Full: LiteLLM proxy with per-team dashboards (3.2)
│
├── "High sustained volume on a well-defined repetitive task"
│   ├── Volume ≥100K req/month + task is well-defined → Distillation (4.1)
│   └── Volume ≥80% sustained utilization → Provisioned Throughput (4.2)
│
├── "Costs are growing faster than usage"
│   ├── Check CloudWatch log ingestion (anti-pattern #4)
│   ├── Check Knowledge Base infrastructure costs (anti-pattern #9)
│   └── Check for agent amplification without max_iterations (anti-pattern #6)
│
└── "No human owns the cost signal"
    └── FinOps governance program (4.4): model policy + monthly review + chargeback

First-30-Days Execution Sequence

For a practitioner inheriting a Bedrock deployment cold:

Days 1–2: Enable CUR 2.0 with IAM principal opt-in. Create Athena view. Set up anomaly detection. Do not change anything in production.

Days 3–4: Run diagnostic queries. Identify top-5 cost drivers by model and IAM principal. Map each to a workload owner.

Days 5–7: Schedule model right-sizing shadow tests for the top-3 cost-driving workloads. Verify prompt caching status for each.

Week 2: Enable prompt caching on any uncached high-input-token workload. Shadow test confirms correct behavior. Deploy to production.

Week 3: Identify batch-eligible workloads. Migrate the clearest candidates (nightly jobs, bulk pipelines).

Week 4: AIP tagging rollout. Briefing to workload owners on their cost trend and what changed.

Month 2: LiteLLM deployment (if ≥5 teams). Model distillation assessment for top-volume repetitive workload.

Month 3: Provisioned Throughput commitment for any workload with ≥80% utilization confirmed over 60 days. Begin FinOps review cadence.

Ongoing (monthly): Review Athena cost trend. Check cache hit rates per workload (should be rising as prompt templates stabilize). Verify no new workloads bypassed AIP tagging. Track blended cost-per-request as the KPI — model count, token volume, and spend should grow sublinearly relative to request volume as optimizations compound.

Reference Pricing Summary (June 2026)

Feature Cost
Claude Sonnet 4.6 cache read $0.30/MTok input
Batch inference discount 50% vs on-demand
Flex tier discount ~50% vs on-demand
Provisioned Throughput (1-month) ~15–20% vs on-demand
Provisioned Throughput (6-month) ~30–40% vs on-demand
Knowledge Base (OpenSearch Serverless) ~$700/month minimum
CloudWatch log ingestion $0.50/GB
Athena query ~$5/TB scanned
CUR 2.0 export S3 storage cost only
Cost Anomaly Detection Free
Application Inference Profiles Free
Cross-Region Inference routing No additional cost

Optimization Maturity Levels

Use this to self-assess where a deployment currently sits and what the next priority is:

Level 0 — Spend Blindness

  • No CUR 2.0 export, no cost allocation tags
  • All Bedrock cost appears as a single undifferentiated line item
  • No anomaly detection, no budgets
  • Typical for: deployments under 3 months, or teams that inherited infra without a handoff

Level 1 — Visibility

  • CUR 2.0 with IAM principal opt-in enabled
  • Cost allocation tags active on IAM roles
  • Athena view queried weekly
  • Anomaly detection alerting to Slack
  • Typical savings vs Level 0: 0% (visibility only, but enables everything else)

Level 2 — Quick Wins Applied

  • Model right-sizing completed for top-3 workloads
  • Prompt caching enabled on all RAG/agent/long-context workloads
  • Batch inference running for all async workloads
  • Flex tier applied to dev/staging
  • Typical savings vs Level 1: 25–50% of prior bill

Level 3 — Governed

  • Application Inference Profiles deployed for all production workloads
  • LiteLLM or equivalent gateway enforcing model tier policy
  • Prompt compression applied; token budgets set
  • Monthly FinOps review cadence established
  • Typical savings vs Level 2: additional 10–25%

Level 4 — Strategic

  • Provisioned Throughput for all workloads with ≥80% utilization
  • Model distillation running for top-volume repetitive tasks
  • Full chargeback program operational with team P&L accountability
  • OpenCost/Kubecost providing K8s-level infrastructure cost attribution
  • Typical savings vs Level 3: additional 15–35% at scale

Most teams starting from Level 0 reach Level 2 within 30 days and Level 3 within 90 days with a dedicated practitioner. Level 4 requires organizational buy-in beyond FinOps engineering and typically takes 6+ months to operationalize fully.

Sources: AWS Bedrock pricing page, AWS Cloud Financial Management blog, Caylent Bedrock pricing explainer, AWS Builder Center architecture guides, DEV Community prompt caching implementation guides, TeamAWS model distillation analysis.