← Agent Frameworks 🕐 17 min read
Agent Frameworks

Bedrock Prompt Caching — Cost Patterns, Break-Even Math, and Operational Optimization (2026)

1. How Bedrock Prompt Caching Works · March 2026

Prompt caching allows Bedrock to store the computed KV (key-value) attention state of a prompt prefix so subsequent requests that share that prefix can skip the re-computation.

Audience: FinOps practitioners and AI platform engineers managing enterprise Bedrock workloads.
Status: Current as of June 2026. Pricing figures reference us-east-1 on-demand rates unless noted.

Mechanism

Prompt caching allows Bedrock to store the computed KV (key-value) attention state of a prompt prefix so subsequent requests that share that prefix can skip the re-computation. The model receives only the new tokens (the dynamic tail of the request) and retrieves the cached state for the static head.

From the billing and API perspective, each request tokenizes into at most four distinct segments:

Segment What it is
Cache write input tokens Tokens in the stable prefix that are being written to cache for the first time (or re-written after TTL expiry)
Cache read input tokens Tokens in the stable prefix that were found in the cache and served from there
Standard input tokens Tokens in the dynamic tail that are not cached — processed normally
Output tokens All generated tokens

Cache hits and misses are mutually exclusive: a given token in the stable prefix contributes to either a write or a read, never both in the same request.

Cache Scoping and Isolation

Bedrock caches are isolated along four dimensions:

  • AWS Account — no cross-account cache sharing. Teams on separate accounts each maintain independent caches, even if they run identical prompts against the same model.
  • Model — anthropic.claude-sonnet-4-5-v1:0 and anthropic.claude-sonnet-4-6-v1:0 maintain separate cache namespaces.
  • AWS Region — cache state is regional. Cross-region inference profiles (e.g., us.anthropic.claude-sonnet-4-6-v1:0) can route to multiple regions and lose cache hits when a request lands in a different region than the previous one.
  • Prefix content — the cache key is the exact byte sequence of the cached content plus model ID. One character difference = cache miss.

Multi-tenant implication: Different internal teams sharing a single AWS account and the same model CAN share cache hits if they use identical prompt prefixes. This is a meaningful architecture decision for platform teams building internal LLM gateways.

TTL Options

As of January 2026, Bedrock supports two TTL tiers for Claude models:

TTL Cache write premium Cache read discount Use case
5-minute (standard) 1.25× standard input 0.10× standard input High-frequency, stateless APIs; RAG loops; chat sessions
1-hour (extended) 2.00× standard input 0.10× standard input Document analysis workflows, long-running agentic sessions, start-of-business cache warming

TTL operates on a sliding window: each successful cache read resets the timer. A frequently-hit cache entry stays alive indefinitely as long as at least one request hits it within each TTL window.

Nova models currently support only the 5-minute TTL with a fixed write premium.

Supported Models (June 2026)

Model family Cache support Min cacheable tokens Tool caching
Claude Sonnet 4.5 / 4.6 GA (April 2025) 1,024 Yes
Claude Opus 4.x GA 1,024 Yes
Claude Haiku 3.5 GA 2,048 Yes
Claude Haiku 4.5 / Opus 4.5 GA 4,096 Yes
Amazon Nova Micro / Lite / Pro / Premier GA 1,024 No
Nova 2 models Preview (early 2026) 1,024 No
Meta Llama, Mistral Not supported — —

Minimum cacheable block size is evaluated against the cumulative token count across all cached sections (tools definition + system prompt + message history), not per-section. A 500-token system prompt paired with a 600-token tools block clears the 1,024-token threshold for Sonnet 4.6.

Requests that mark a block for caching but fall below the minimum threshold are processed without error and without caching — the cache_control annotation is silently ignored and you are billed at standard input rates.

How Cache Points Are Marked (API)

Claude on Bedrock uses the same cache_control annotation as the direct Anthropic API:

import boto3, json

client = boto3.client("bedrock-runtime", region_name="us-east-1")

body = {
    "anthropic_version": "bedrock-2023-05-31",
    "max_tokens": 1024,
    "system": [
        {
            "type": "text",
            "text": "<your stable system prompt — 1000+ tokens>",
            "cache_control": {"type": "ephemeral"}   # marks cache point
        }
    ],
    "messages": [
        {"role": "user", "content": "What is the capital of France?"}
    ]
}

response = client.invoke_model(
    modelId="anthropic.claude-sonnet-4-6-v1:0",
    body=json.dumps(body),
    contentType="application/json",
    accept="application/json"
)

usage = json.loads(response["body"].read())["usage"]
# usage keys: input_tokens, output_tokens,
#             cache_creation_input_tokens, cache_read_input_tokens

For Nova models, the syntax uses a cachePoint block in the messages array rather than inline cache_control on content blocks — a distinction that tripped LiteLLM’s Bedrock adapter (see §6).

2. CUR 2.0 Representation

The Four Token-Type Columns

In AWS Cost and Usage Report 2.0 (Data Exports), each Bedrock invocation that uses prompt caching can generate up to four separate line items with distinct line_item_usage_type values:

line_item_usage_type suffix Maps to Priced at
InputTokens Standard (non-cached) input tokens 1× standard input rate
OutputTokens Generated output tokens standard output rate
CacheReadInputTokens Cache read tokens ~0.1× standard input rate
CacheWriteInputTokens Cache write tokens 1.25× or 2.0× standard input rate (by TTL)

The full line_item_usage_type string includes the region and model identifier prefix. Example values in us-east-1:

USE1-Claude-Sonnet-4-6:InputTokens
USE1-Claude-Sonnet-4-6:OutputTokens
USE1-Claude-Sonnet-4-6:CacheReadInputTokens
USE1-Claude-Sonnet-4-6:CacheWriteInputTokens

The Reconciliation Error

The most common FinOps mistake: engineers building Bedrock cost dashboards sum only InputTokens + OutputTokens and exclude the cache token types. This produces:

  1. Understated cost — cache write tokens at 1.25× or 2.0× are expensive. Excluding them makes heavy-write workloads look cheaper than they are.
  2. Overstated token volume — the actual tokens processed by the model is InputTokens + CacheWriteInputTokens (both touch the model). Cache reads do not.
  3. False attribution — if you alert on input token spend without including cache writes, a workload shift from standard to caching-heavy can look like a 40% cost decrease while actually being a 10% cost increase.

Correct CUR Query Pattern

-- Athena query against CUR 2.0 parquet data
SELECT
    line_item_resource_id,
    DATE_TRUNC('day', line_item_usage_start_date) AS usage_day,

    -- Standard tokens
    SUM(CASE WHEN line_item_usage_type LIKE '%InputTokens'
             AND line_item_usage_type NOT LIKE '%CacheReadInputTokens'
             AND line_item_usage_type NOT LIKE '%CacheWriteInputTokens'
             THEN line_item_usage_amount ELSE 0 END) AS standard_input_tokens,

    SUM(CASE WHEN line_item_usage_type LIKE '%OutputTokens'
             THEN line_item_usage_amount ELSE 0 END) AS output_tokens,

    -- Cache tokens
    SUM(CASE WHEN line_item_usage_type LIKE '%CacheReadInputTokens'
             THEN line_item_usage_amount ELSE 0 END) AS cache_read_tokens,

    SUM(CASE WHEN line_item_usage_type LIKE '%CacheWriteInputTokens'
             THEN line_item_usage_amount ELSE 0 END) AS cache_write_tokens,

    -- Effective cost (all four types)
    SUM(line_item_unblended_cost) AS total_cost,

    -- Cache hit rate
    ROUND(
        SUM(CASE WHEN line_item_usage_type LIKE '%CacheReadInputTokens'
                 THEN line_item_usage_amount ELSE 0 END)
        / NULLIF(
            SUM(CASE WHEN line_item_usage_type LIKE '%CacheReadInputTokens'
                     THEN line_item_usage_amount ELSE 0 END)
            + SUM(CASE WHEN line_item_usage_type LIKE '%CacheWriteInputTokens'
                       THEN line_item_usage_amount ELSE 0 END),
            0
        ) * 100, 2
    ) AS cache_hit_rate_pct

FROM cur_db.bedrock_cur
WHERE product_service_name = 'Amazon Bedrock'
  AND line_item_usage_start_date >= DATE '2026-06-01'
GROUP BY 1, 2
ORDER BY 2 DESC, 5 DESC;

CUR 2.0 IAM Principal Attribution

When INCLUDE_MEMBER_LINKED_ACCOUNTS and IAM principal data are enabled in your Data Export, the column line_item_iam_principal surfaces the IAM ARN of the Bedrock caller. This enables per-team or per-application cache efficiency reporting without tagging changes:

SELECT
    line_item_iam_principal,
    SUM(CASE WHEN line_item_usage_type LIKE '%CacheReadInputTokens'
             THEN line_item_usage_amount ELSE 0 END) AS cache_reads,
    SUM(CASE WHEN line_item_usage_type LIKE '%CacheWriteInputTokens'
             THEN line_item_usage_amount ELSE 0 END) AS cache_writes,
    SUM(line_item_unblended_cost) AS cost
FROM cur_db.bedrock_cur
WHERE product_service_name = 'Amazon Bedrock'
GROUP BY 1
ORDER BY 4 DESC;

3. Break-Even Math

Pricing Reference (Claude on Bedrock, us-east-1, June 2026)

Token type Claude Sonnet 4.5 Claude Sonnet 4.6
Standard input $3.00 / MTok $3.00 / MTok
Standard output $15.00 / MTok $15.00 / MTok
Cache write (5-min TTL) $3.75 / MTok (1.25×) $3.75 / MTok
Cache write (1-hr TTL) $6.00 / MTok (2.0×) $6.00 / MTok
Cache read $0.30 / MTok (0.10×) $0.30 / MTok

Prices match as of June 2026. Sonnet 4.5 and 4.6 carry identical pricing on Bedrock on-demand.

Break-Even Formula

Define:

  • P_i = standard input price per token
  • P_w = cache write price per token (1.25 × P_i for 5-min TTL)
  • P_r = cache read price per token (0.10 × P_i)
  • N = number of tokens in the stable prefix (the cached block)
  • R = number of requests that hit the cache after the first write
  • Total_cached_cost = P_w × N + P_r × N × R
  • Total_uncached_cost = P_i × N × (1 + R)

Break-even when Total_cached_cost = Total_uncached_cost:

P_w × N + P_r × N × R = P_i × N × (1 + R)

P_w + P_r × R = P_i × (1 + R)

P_w - P_i = P_i × R - P_r × R

(P_w - P_i) = R × (P_i - P_r)

R_breakeven = (P_w - P_i) / (P_i - P_r)

For 5-minute TTL (P_w = 1.25 P_i, P_r = 0.10 P_i):

R_breakeven = (1.25 P_i - P_i) / (P_i - 0.10 P_i)
             = 0.25 P_i / 0.90 P_i
             = 0.278
             ≈ 0.28 additional reads per write

Translated: you break even when roughly 1 in 4 subsequent requests hit the cached prefix. In practice, a workload where the same system prompt is used for more than ~3 requests per TTL window will save money with 5-minute caching.

For 1-hour TTL (P_w = 2.0 P_i, P_r = 0.10 P_i):

R_breakeven = (2.0 P_i - P_i) / (P_i - 0.10 P_i)
             = 1.0 P_i / 0.90 P_i
             = 1.11
             ≈ 1.1 additional reads per write

For 1-hour TTL, you need at least ~2 total requests (1 write + 1.1 reads) per cache entry to break even. The higher write premium means this TTL tier only pencils out when cache hit counts per prefix per hour exceed ~2.

Worked Example: RAG System Prompt (5-min TTL)

Setup:

  • Stable system prompt: 2,000 tokens
  • RAG retrieval chunks prepended to each request: 3,000 tokens (variable content — not cached)
  • User query: ~50 tokens
  • Requests per minute (steady state): 20
  • Request pattern: same system prompt, different queries

Without caching (baseline):

  • Input tokens per request: 2,000 (system) + 3,000 (RAG) + 50 (query) = 5,050
  • Cost per request: 5,050 × $3.00 / 1,000,000 = $0.01515
  • Cost per minute: 20 × $0.01515 = $0.303

With caching (system prompt cached, 5-min TTL):

  • First request (write): 2,000 cache write + 3,050 standard input
    • = (2,000 × $3.75 + 3,050 × $3.00) / 1,000,000
    • = ($7.50 + $9.15) / 1,000 = $0.01665
  • Subsequent 19 requests (reads): 2,000 cache read + 3,050 standard input each
    • = (2,000 × $0.30 + 3,050 × $3.00) / 1,000,000
    • = ($0.60 + $9.15) / 1,000 = $0.009750 per request
    • 19 requests: 19 × $0.009750 = $0.18525

Total per minute with caching: $0.01665 + $0.18525 = $0.2019 Saving: $0.303 − $0.2019 = $0.1011/min = 33% reduction

Cache hit rate in this scenario: 19/20 = 95% — well above break-even.

Worked Example: Infrequent Long-Document Analysis (1-hr TTL)

Setup:

  • Document: 50,000 tokens cached
  • Requests against this document per hour: 3 (low volume analyst use case)
  • Queries per document: ~200 tokens each

Without caching:

  • Each request: 50,200 tokens input
  • 3 requests: 150,600 tokens × $3.00/MTok = $0.452

With 1-hour caching:

  • 1 write: 50,000 × $6.00/MTok = $0.300, plus 200 standard = $0.0006 → $0.3006
  • 2 reads: 2 × (50,000 × $0.30/MTok + 200 × $3.00/MTok) = 2 × ($0.015 + $0.0006) = $0.0312
  • Total: $0.3006 + $0.0312 = $0.3318

Saving: $0.452 − $0.3318 = $0.120 (27% reduction) Even at 3 requests/hour, the 1-hour TTL is profitable for very large documents.

When Caching Costs More

  • Single-request workloads (batch jobs that reuse no prefix): every request pays the write premium, zero reads recoup it. Net cost increase of 25% on cached tokens.
  • High-variability system prompts (personalized per-user instructions that change every request): the prefix never matches, every request is a write miss, you pay 1.25× for nothing.
  • Very short sessions below TTL with unique prefixes: e.g., a chatbot where each user conversation starts with a unique user-specific context block. The write cost never amortizes.
  • Cross-region inference with low traffic: if traffic is sparse and routes randomly across regions, a cache write in us-east-1 does not benefit a read routed to us-west-2.

4. Operational Patterns

4.1 Cache Warming

Cache warming = proactively issuing a “priming” request at the start of a usage window to establish the cache before real user traffic arrives. This prevents the first real user request from paying the write cost and experiencing write-latency.

Pattern: Lambda-based cache warmer on a schedule

# lambda_warmer.py — triggered by EventBridge at 08:45 UTC Mon-Fri
import boto3, json, os

SYSTEM_PROMPT = open("/opt/system_prompt.txt").read()  # ~3,000 tokens
MODEL_ID = "anthropic.claude-sonnet-4-6-v1:0"

def handler(event, context):
    client = boto3.client("bedrock-runtime", region_name="us-east-1")
    body = {
        "anthropic_version": "bedrock-2023-05-31",
        "max_tokens": 1,           # minimize output cost
        "system": [
            {
                "type": "text",
                "text": SYSTEM_PROMPT,
                "cache_control": {"type": "ephemeral"}
            }
        ],
        "messages": [
            {"role": "user", "content": "warmup"}
        ]
    }

    resp = client.invoke_model(
        modelId=MODEL_ID,
        body=json.dumps(body),
        contentType="application/json",
        accept="application/json"
    )
    usage = json.loads(resp["body"].read())["usage"]
    print(f"Warmer result: {json.dumps(usage)}")
    # Expected: cache_creation_input_tokens > 0, cache_read_input_tokens = 0

    return {"status": "warmed", "usage": usage}

For 5-minute TTL, the warmer must run at least every 4 minutes during active hours to keep the cache alive. For 1-hour TTL, a single scheduled Lambda invocation per hour suffices.

Cost of the warmer itself (3,000-token system prompt, 5-min TTL):

  • One warming request: 3,000 × $3.75/MTok = $0.01125
  • 60 warming requests/hour × 9 business hours × 20 business days/month: 10,800 requests
  • Monthly warming cost: 10,800 × $0.01125 = $121.50/month

This is break-even against a workload of ~3+ real requests per minute during those hours. At 20 RPM it pays off easily; at 0.5 RPM it does not.

4.2 Cache Invalidation Triggers

The cache is invalidated (entry expires or misses) in these conditions:

Trigger Behavior
TTL expiry with no hits Entry silently evicted; next request pays write cost
Single-character change to cached content Cache miss; full write cost on the modified prefix
Model version change New model ID = new cache namespace; all entries are cold
Region routing change Cross-region inference routes to different AZ/region; cache miss
Tool definition reordering Tool JSON is part of the prefix; reordering = cache miss
Adding/removing a message before the cache point Shifts the prefix; cache miss for all content after the change
Changing section ordering (tools → system → messages chain) Invalidates downstream sections

Practical implication for feature flags: If your system prompt is assembled dynamically and a feature flag changes the content (even trailing whitespace), every request from that point forward pays the write premium until the cache re-warms. Instrument cache_creation_input_tokens on feature flag changes.

4.3 Multi-Tenant Caching Architecture

Question: Can Team A’s cache write benefit Team B’s subsequent identical request?

Answer: Only if both teams share the same AWS account. Cache isolation is per-account, per-model, per-region.

Architecture Cache sharing Notes
Single AWS account, shared Bedrock role Full sharing between teams Most cache-efficient; requires IAM boundary enforcement
Per-team AWS accounts No sharing Each team warms its own cache
LiteLLM gateway on single account Sharing enabled via gateway Gateway consolidates requests; use user parameter for attribution, not account separation
AWS Organizations with separate member accounts No sharing Each member account is isolated

Platform engineering recommendation: If multiple teams run the same RAG application pattern (identical system prompts), consolidating them behind a single-account LiteLLM gateway maximizes cache utilization. Per-team cost attribution is handled through LiteLLM’s metadata.team_id tagging, not through account separation.

4.4 Prompt Structure for Maximum Cache Hits

The cache key covers the exact byte sequence from the beginning of the request to the cache_control marker. Structure your prompt so the stable content is fully in front, and dynamic content is fully behind.

Optimal structure (stable → dynamic):

[CACHED SECTION — marked with cache_control]
1. Tool definitions (rarely change)
2. System prompt instructions (stable per application version)
3. Domain knowledge / reference documents (stable per version)
4. RAG context retrieved for a query (semi-stable: same top-k for same query)

[UNCACHED SECTION — after cache point]
5. Conversation history (grows per turn)
6. Current user message (unique per request)

Anti-pattern: Inserting a timestamp or request ID into the system prompt. Even a single dynamic token at position 0 prevents any caching.

# BAD — timestamp breaks caching
system = f"You are a helpful assistant. Current time: {datetime.now()}. ..."

# GOOD — timestamp stays in user message, system is static
system = "You are a helpful assistant..."
user_message = f"[{datetime.now()}] User query: {query}"

4.5 RAG Chunk Ordering for Cache Efficiency

In RAG pipelines, the retrieved chunks are prepended to the user query. If you want to cache the retrieved context (not just the system prompt), you need retrieval to be deterministic for a given query.

Problem: Most vector databases return chunks in relevance-score order, which is stable for a given query but changes as the index is updated. A reindex (daily embedding refresh, new document ingestion) changes chunk ordering and busts the cache.

Mitigation patterns:

  1. Sort chunks by document ID then chunk offset after retrieval. This makes ordering deterministic and index-update-resilient, at the cost of slightly suboptimal relevance ordering.
  2. Separate cache points: use one cache_control for the system prompt (highly stable) and a second for the RAG chunks (stable within a session). This allows the system prompt cache to survive RAG cache misses.
  3. Cache the RAG retrieval result at application layer (Redis/ElastiCache) keyed by the query embedding. If the same query recurs within the cache TTL, serve the same chunks in the same order, guaranteeing a Bedrock cache hit.

4.6 Conversation History Caching

In multi-turn conversations, the conversation history grows with each turn. Cache the longest stable prefix you have.

Pattern: sliding cache point on conversation turns

def build_cached_request(system_prompt, history, new_message):
    """
    Cache everything up to but not including the current turn.
    On turn N, the prefix (system + turns 1..N-1) is cacheable.
    The current turn is uncached.
    """
    messages = []

    # Add all prior turns as a single cached block
    if history:
        history_text = "\n\n".join([
            f"{m['role'].upper()}: {m['content']}" for m in history
        ])
        messages.append({
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": history_text,
                    "cache_control": {"type": "ephemeral"}
                }
            ]
        })
        # Bedrock requires alternating roles; inject a minimal assistant ack
        messages.append({
            "role": "assistant",
            "content": "Understood. Continuing the conversation."
        })

    # Current turn — not cached
    messages.append({"role": "user", "content": new_message})

    return {
        "anthropic_version": "bedrock-2023-05-31",
        "max_tokens": 2048,
        "system": [
            {
                "type": "text",
                "text": system_prompt,
                "cache_control": {"type": "ephemeral"}
            }
        ],
        "messages": messages
    }

Cost profile of conversation caching: On a 10-turn conversation, turns 2–10 each read the prior history from cache. The total cache read savings (9 reads × history length) typically exceed the write cost by turn 3–4 for conversations with histories longer than ~1,000 tokens.

4.7 When NOT to Use Caching

Scenario Reason to skip
Batch processing pipelines (one-off documents) No repeated prefix; every request pays write premium with zero reads
Per-user personalized system prompts that change per session Cache key unique per user; writes never amortize
Prompts shorter than the model’s minimum cacheable threshold Cache annotation silently ignored; standard rates apply
Latency-sensitive single-turn APIs with < 1,000 token prompts Cache write adds ~10–20ms overhead; no latency benefit from cache read on short prompts
Experimental/dev environments with frequent prompt iteration Cache busted constantly; you pay 25% extra for every iteration
Cross-region inference with very low RPM (< 2/min) High probability of cross-region misses; break-even not reached

5. LiteLLM Configuration for Bedrock Caching

Basic Setup

LiteLLM’s proxy natively passes cache_control annotations through to Bedrock when enable_prompt_caching is set in the model configuration:

# litellm_config.yaml
model_list:
  - model_name: claude-sonnet-4-6
    litellm_params:
      model: bedrock/anthropic.claude-sonnet-4-6-v1:0
      aws_region_name: us-east-1
      enable_prompt_caching: true   # instructs LiteLLM to pass cache_control through

litellm_settings:
  success_callback: ["prometheus"]
  failure_callback: ["prometheus"]

With this config, any request that includes cache_control in the message body will be passed through to Bedrock. LiteLLM does not add cache points automatically — the caller must annotate the messages.

Known Bugs and Fixes

Bug 1: extraneous key [cachePoint] (Issue #17479)

  • Symptom: BadRequestError: extraneous key [cachePoint] is not permitted when using prompt caching with Claude via Bedrock.
  • Root cause: LiteLLM was injecting Nova-style cachePoint blocks into Claude requests. Nova and Claude use different cache annotation syntax on Bedrock.
  • Status: Open as of June 2026; affects LiteLLM versions through at least v1.55.x. Workaround: pin to v1.52.3 or apply the patch in PR #17490.

Bug 2: cacheWriteInputTokens excluded from cost calculation (PR #15292)

  • Symptom: Negative computed costs for Bedrock Claude requests with prompt caching; cost dashboards in LiteLLM UI show implausible numbers.
  • Root cause: Bedrock Anthropic models return cacheWriteInputTokens separately; LiteLLM was not including this in prompt_tokens, causing the token math to produce a negative net.
  • Fixed in: LiteLLM v1.50.1.
  • Action: If you are on a version < 1.50.1, upgrade. Validate with a known-cost test prompt.

Bug 3: TTL parameter stripped (Issue #19848)

  • Symptom: 1-hour TTL requests sent through LiteLLM revert to 5-minute TTL. The ttl parameter in the Bedrock cache_control body is stripped before the request is forwarded.
  • Status: Open as of June 2026. Workaround: call Bedrock directly for workloads requiring 1-hour TTL caching, bypassing LiteLLM for those requests.

Bug 4: Router affinity TTL hardcoded to 5 minutes (Issue #28427)

  • Symptom: LiteLLM’s router cache affinity (which routes repeated requests to the same Bedrock endpoint to maximize cache hits) uses a hardcoded 5-minute window regardless of the TTL configured on the model. If you configure 1-hour TTL and traffic is sparse, the router affinity expires before the Bedrock cache does.
  • Status: Open as of June 2026. Workaround: set router_settings.cache_responses_ttl manually in litellm_config.yaml to match your Bedrock TTL.

LiteLLM Python SDK Usage with Cache Control

import litellm

# Ensure cache_control is passed through to Bedrock
response = litellm.completion(
    model="bedrock/anthropic.claude-sonnet-4-6-v1:0",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": STABLE_SYSTEM_CONTEXT,   # 2000+ tokens
                    "cache_control": {"type": "ephemeral"}
                },
                {
                    "type": "text",
                    "text": user_query
                }
            ]
        }
    ],
    aws_region_name="us-east-1",
    enable_prompt_caching=True
)

# Inspect cache usage
usage = response.usage
print(f"cache_creation_input_tokens: {usage.get('cache_creation_input_tokens', 0)}")
print(f"cache_read_input_tokens:     {usage.get('cache_read_input_tokens', 0)}")
print(f"input_tokens (uncached):     {usage.input_tokens}")

6. Cost Monitoring

CloudWatch Metrics

Bedrock automatically publishes prompt caching metrics to CloudWatch under the AWS/Bedrock namespace. No additional configuration required.

CloudWatch Metric Dimension Description
CacheReadInputTokens ModelId, Region Tokens served from cache (reads)
CacheWriteInputTokens ModelId, Region Tokens written to cache
InputTokens ModelId, Region Standard (uncached) input tokens
OutputTokens ModelId, Region Generated output tokens

All four metrics support Sum aggregation. Use CacheReadInputTokens / (CacheReadInputTokens + CacheWriteInputTokens) as the cache hit rate signal.

CloudWatch Metric Math for Cache Hit Rate:

METRIC_MATH:
  Expression: m1/(m1+m2)*100
  Label: CacheHitRate_Pct

  m1: AWS/Bedrock CacheReadInputTokens  (Sum, 1min)
  m2: AWS/Bedrock CacheWriteInputTokens (Sum, 1min)

CloudWatch Alarm: cache hit rate drop

{
  "AlarmName": "BedrockCacheHitRateLow",
  "AlarmDescription": "Cache hit rate dropped below 50% — potential cache invalidation event",
  "Metrics": [
    {
      "Id": "m1",
      "MetricStat": {
        "Metric": {
          "Namespace": "AWS/Bedrock",
          "MetricName": "CacheReadInputTokens",
          "Dimensions": [{"Name": "ModelId", "Value": "anthropic.claude-sonnet-4-6-v1:0"}]
        },
        "Period": 300,
        "Stat": "Sum"
      }
    },
    {
      "Id": "m2",
      "MetricStat": {
        "Metric": {
          "Namespace": "AWS/Bedrock",
          "MetricName": "CacheWriteInputTokens",
          "Dimensions": [{"Name": "ModelId", "Value": "anthropic.claude-sonnet-4-6-v1:0"}]
        },
        "Period": 300,
        "Stat": "Sum"
      }
    },
    {
      "Id": "e1",
      "Expression": "m1/(m1+m2)*100",
      "Label": "CacheHitRate"
    }
  ],
  "ComparisonOperator": "LessThanThreshold",
  "Threshold": 50,
  "EvaluationPeriods": 3,
  "TreatMissingData": "notBreaching"
}

LiteLLM Prometheus Metrics

LiteLLM exposes prompt caching metrics at http://<proxy-host>:4000/metrics in Prometheus exposition format.

Metric name Type Description
litellm_input_cached_tokens_metric Counter Provider-side cache read tokens (Bedrock CacheReadInputTokens)
litellm_input_cache_creation_tokens_metric Counter Provider-side cache write tokens (Bedrock CacheWriteInputTokens)
litellm_input_tokens_metric Counter Standard input tokens
litellm_output_tokens_metric Counter Output tokens

Important distinction: litellm_input_cached_tokens_metric tracks provider-side cache reads (tokens served from Bedrock’s KV cache). This is separate from litellm_cached_tokens_metric, which tracks LiteLLM’s own response-level semantic cache (entire response served from LiteLLM’s Redis cache, no provider request made at all).

Prometheus scrape config:

# prometheus.yml
scrape_configs:
  - job_name: 'litellm'
    static_configs:
      - targets: ['litellm-proxy:4000']
    metrics_path: '/metrics'
    scrape_interval: 15s

Grafana Dashboard Queries

Cache Hit Rate (PromQL):

# Cache hit rate as a percentage (rate over 5-minute window)
rate(litellm_input_cached_tokens_metric[5m])
/
(
  rate(litellm_input_cached_tokens_metric[5m])
  + rate(litellm_input_cache_creation_tokens_metric[5m])
) * 100

Cache Write Cost Ratio (what fraction of total input spend is write premium):

# Assuming write tokens cost 1.25x standard; compute write premium spend
(
  rate(litellm_input_cache_creation_tokens_metric[5m]) * 3.75
)
/
(
  rate(litellm_input_tokens_metric[5m]) * 3.00
  + rate(litellm_input_cached_tokens_metric[5m]) * 0.30
  + rate(litellm_input_cache_creation_tokens_metric[5m]) * 3.75
) * 100

Cost savings vs. no caching baseline:

# Effective cost per minute
(
  rate(litellm_input_tokens_metric[1m]) * 3.00
  + rate(litellm_output_tokens_metric[1m]) * 15.00
  + rate(litellm_input_cached_tokens_metric[1m]) * 0.30
  + rate(litellm_input_cache_creation_tokens_metric[1m]) * 3.75
) / 1000000

# Baseline cost per minute (hypothetical no-cache: all tokens at standard input + output)
(
  (
    rate(litellm_input_tokens_metric[1m])
    + rate(litellm_input_cached_tokens_metric[1m])
    + rate(litellm_input_cache_creation_tokens_metric[1m])
  ) * 3.00
  + rate(litellm_output_tokens_metric[1m]) * 15.00
) / 1000000

Grafana dashboard import: The official LiteLLM dashboard is available at Grafana Labs as ID 24965. It includes token volume panels but does not yet break out cache-specific panels by default — add the above queries as custom panels.

Per-Application Cache Efficiency Report

When LiteLLM is configured with metadata.team_id or metadata.application_id, Prometheus labels carry these dimensions:

# Cache hit rate by team
sum by (team) (rate(litellm_input_cached_tokens_metric{team=~".+"}[5m]))
/
(
  sum by (team) (rate(litellm_input_cached_tokens_metric{team=~".+"}[5m]))
  + sum by (team) (rate(litellm_input_cache_creation_tokens_metric{team=~".+"}[5m]))
) * 100

7. Extended Context and Nova Model Caching

Amazon Nova Prompt Caching Specifics

Nova models (Micro, Lite, Pro, Premier) support prompt caching with behavioral differences from Claude:

Dimension Claude (Bedrock) Nova (Bedrock)
Cache annotation syntax cache_control: {"type": "ephemeral"} in content blocks cachePoint: {"type": "default"} in messages array
TTL options 5-minute or 1-hour 5-minute only
Tool definition caching Supported Not supported
Implicit caching No Yes — Nova caches repetitive prefixes automatically
Minimum cacheable tokens 1,024 1,024
Max prefix size Not published ~20,000 tokens
Cross-region TTL behavior Sliding window per region Sliding window per region

Nova implicit caching means that for Nova models you do not need to add cachePoint annotations to get some caching benefit — the model service automatically detects repeated prefix content and caches it. However, explicit cache points give you control over what is cached and allow you to inspect cacheWriteInputTokens / cacheReadInputTokens in the response.

Nova cache write pricing:

  • Nova Micro: $0.042/MTok write, $0.0035/MTok read (reported; 12× read discount)
  • Nova Lite: $0.075/MTok write, $0.006/MTok read
  • Nova Pro: $0.500/MTok write, $0.040/MTok read

Real-world benchmark (3 Nova models with caching + model routing): 49% cost reduction from caching alone, 97% reduction when combined with routing smaller queries to Micro/Lite.

Large Document Caching

For document analysis workloads, the 1-hour TTL (Claude) or implicit caching (Nova) enables cost-effective repeated analysis of large documents:

  • A 50,000-token document (≈37,500 words) cached with 1-hour TTL: break-even at ~2 queries per document per hour.
  • For PDF analysis, Bedrock’s Document API passes PDFs as base64-encoded content. The PDF bytes are tokenized server-side; the resulting tokens are cacheable using the same cache_control mechanism.

Caching structured data for Claude:

# Cache a large reference document alongside the system prompt
body = {
    "anthropic_version": "bedrock-2023-05-31",
    "max_tokens": 2048,
    "system": [
        {
            "type": "text",
            "text": SYSTEM_INSTRUCTIONS,
            "cache_control": {"type": "ephemeral"}   # cache point 1: instructions
        }
    ],
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "document",
                    "source": {
                        "type": "base64",
                        "media_type": "application/pdf",
                        "data": base64_pdf_content
                    },
                    "cache_control": {"type": "ephemeral"}   # cache point 2: document
                },
                {
                    "type": "text",
                    "text": user_query   # uncached — changes per request
                }
            ]
        }
    ]
}

8. Operational Checklist

Implementation Checklist

  • [ ] Stable prompt content (system instructions, tool definitions, reference docs) is positioned before any dynamic content
  • [ ] cache_control: {"type": "ephemeral"} is applied to the end of the stable section
  • [ ] Combined stable tokens exceed the model’s minimum cacheable threshold (1,024 for Sonnet 4.6)
  • [ ] System prompt contains no per-request dynamic values (timestamps, request IDs, user-specific data)
  • [ ] RAG chunk ordering is deterministic (sorted by doc ID + offset, not raw similarity score)
  • [ ] Cache warming Lambda is scheduled at model selection for 5-min TTL workloads
  • [ ] LiteLLM version ≥ 1.50.1 (cacheWriteInputTokens cost fix)
  • [ ] LiteLLM version pinned or patched for the cachePoint/cache_control mismatch bug (Issue #17479) if using the proxy

Monitoring Checklist

  • [ ] CloudWatch dashboard includes CacheReadInputTokens and CacheWriteInputTokens panels
  • [ ] Alarm set on cache hit rate < 50% for 3 consecutive 5-minute windows
  • [ ] CUR 2.0 Athena queries include all four token type columns
  • [ ] LiteLLM Prometheus metrics scraped and litellm_input_cached_tokens_metric visualized in Grafana
  • [ ] Per-application or per-team cache hit rate tracked separately

Red Flags

Signal Likely cause Action
cache_creation_input_tokens > 0 on every request Cache miss on every call; no reads accumulating Check for dynamic content in cached prefix; verify TTL is sufficient for traffic rate
CUR shows cost increase after enabling caching Write premium not offset by reads; hit rate < break-even Compute actual hit rate from CUR; may need higher traffic volume or larger stable prefix
LiteLLM shows negative cost for Bedrock requests Running LiteLLM < 1.50.1; bug #15292 Upgrade LiteLLM
Cache hit rate drops sharply on business days Feature flag or environment variable being injected into system prompt Audit prompt assembly code; grep for non-deterministic injections
1-hour TTL requests revert to 5-minute behavior LiteLLM TTL stripping bug #19848 Call Bedrock directly for 1-hour TTL workloads, bypassing LiteLLM

9. Summary: Decision Framework

Is your stable prompt prefix > minimum cacheable tokens?
  No  → Caching unavailable; standard pricing applies
  Yes → Continue

Is the same prefix reused across multiple requests?
  Rarely (< 1.1 reuses per write for 1-hr TTL, < 0.28 for 5-min TTL) → Skip caching
  Yes → Continue

What is your traffic pattern?
  Burst at start of business hours, low overnight → Cache warming + 5-min TTL
  Continuous high-RPM (20+ RPM) → 5-min TTL; warmer optional
  Infrequent document analysis (< 5 RPM) → 1-hour TTL; warmer on first access

Are you running cross-region inference?
  Yes, with sparse traffic → Expect 20–40% cache miss rate; factor into break-even
  Yes, with dense traffic → Likely still profitable; monitor per-region hit rates

Multi-tenant: do teams share an account?
  No (separate accounts) → Each team warms and manages its own cache
  Yes → Shared cache possible; consolidate on shared IAM role for same-prefix workloads

Are you routing through LiteLLM?
  Check version ≥ 1.50.1 (cost fix)
  Check for cachePoint bug if using recent versions (Issue #17479)
  Workaround 1-hour TTL stripping (Issue #19848) if needed

Sources