← Agent Frameworks 🕐 12 min read
Agent Frameworks

Prompt Engineering and Context Optimization as LLM Cost Levers (2026)

1. Prompt Caching Economics · March 2026

Anthropic's prompt caching creates a server-side snapshot of a prefix up to a breakpoint marker. Subsequent requests that share that prefix pay the read price instead of the write price.

Audience: Platform engineers and AI leads managing Bedrock or OpenAI spend at scale.
Problem: Token costs are not linear — they are architectural. The same workload can cost 3–10x more depending on how prompts are structured, how context accumulates, and which model handles which task. This note covers each lever with formulas and working code.

How It Works

Anthropic’s prompt caching creates a server-side snapshot of a prefix up to a breakpoint marker. Subsequent requests that share that prefix pay the read price instead of the write price.

Tier Cache Write Cache Read Base Input
Claude 3.5 Sonnet $3.75/MTok (1.25×) $0.30/MTok (0.1×) $3.00/MTok
Claude 3 Haiku $0.30/MTok (1.25×) $0.03/MTok (0.1×) $0.25/MTok
Claude 3 Opus $18.75/MTok (1.25×) $1.50/MTok (0.1×) $15.00/MTok

On Bedrock, cache read is slightly higher than the direct Anthropic API (approximately $0.0003 per 1,000 tokens, roughly 90% discount vs on-demand), but the write multiplier stays at 1.25×.

Minimum cacheable prefix: 1,024 tokens for Claude models.
Cache TTL: 5 minutes (default). Bedrock now supports up to 1-hour TTL via the ttl parameter — but see the LiteLLM bug below before relying on this.

Break-Even Formula

A cache write costs 1.25× the base input price for N_prefix tokens. Each cache read saves (1 - 0.1) × P_input × N_prefix = 0.9 × P_input × N_prefix. The write cost extra is 0.25 × P_input × N_prefix.

Break-even reads = 0.25 / 0.9 ≈ 0.278 → round up to 1 read per write to clear overhead

But you also need to account for the number of requests per TTL window:

def cache_break_even(
    prefix_tokens: int,
    input_price_per_mtok: float,  # e.g. 3.0 for Sonnet
    ttl_minutes: int = 5
) -> dict:
    """How many requests must hit the cache before it pays off."""
    write_overhead = 0.25 * input_price_per_mtok * prefix_tokens / 1_000_000
    read_savings_per_req = 0.90 * input_price_per_mtok * prefix_tokens / 1_000_000
    break_even_requests = write_overhead / read_savings_per_req
    return {
        "break_even_requests": break_even_requests,  # always ~0.278 → rounds to 1
        "write_overhead_usd": write_overhead,
        "read_savings_per_req_usd": read_savings_per_req,
        "note": "Economical if >1 request hits cache per write cycle"
    }

# Example: 4,000-token system prompt on Sonnet
result = cache_break_even(4000, 3.0)
# write_overhead: $0.000003, read_savings_per_req: $0.0000108 → breaks even at 0.28 requests
# At 10 requests/window: saves $0.000105/window → $0.0126/hour on a 100 rps workload

The Bedrock breakeven hit ratio is ~30%. If fewer than 30% of requests share the cached prefix within the TTL window, you are spending more on writes than you recover from reads. High-traffic system prompts break even within the first request. One-shot document analysis rarely does.

System Prompt Caching — Highest-Leverage Pattern

The system prompt is the most stable prefix in any deployment. It changes rarely (deploys, not per-user). Cache it unconditionally.

# Anthropic SDK direct
import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LARGE_SYSTEM_PROMPT,  # your 2000-4000 token system context
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[{"role": "user", "content": user_message}]
)

For multi-turn conversations, also cache the growing message history by placing a cache_control breakpoint after the most recent assistant turn.

CUR Instrumentation on Bedrock

Enable Cost and Usage Report with resource-level granularity. The four columns you need:

lineItem/UsageType  → contains model identifier + token type
product/usagetype   → e.g. "USE1-Claude3.5Sonnet-cache-read-input-token-count"
                           "USE1-Claude3.5Sonnet-cache-write-input-token-count"
                           "USE1-Claude3.5Sonnet-input-token-count"
                           "USE1-Claude3.5Sonnet-output-token-count"

Critical: If you only sum input-token-count and output-token-count, your totals will not match your bill. Cache write tokens are billed separately at 1.25× and appear in their own usage type. Teams consistently hit 15–25% reconciliation gaps from this oversight. Always aggregate all four token columns per model.

Athena query to break down cache efficiency:

SELECT
  regexp_extract(line_item_usage_type, '(Claude[^-]+)', 1) AS model,
  SUM(CASE WHEN line_item_usage_type LIKE '%cache-read%' THEN line_item_usage_amount ELSE 0 END) AS cache_reads,
  SUM(CASE WHEN line_item_usage_type LIKE '%cache-write%' THEN line_item_usage_amount ELSE 0 END) AS cache_writes,
  SUM(CASE WHEN line_item_usage_type LIKE '%input-token%'
       AND line_item_usage_type NOT LIKE '%cache%' THEN line_item_usage_amount ELSE 0 END) AS uncached_inputs,
  ROUND(SUM(line_item_unblended_cost), 4) AS total_cost_usd
FROM cur_table
WHERE line_item_product_code = 'AmazonBedrock'
  AND line_item_usage_start_date >= DATE_TRUNC('day', CURRENT_DATE - INTERVAL '7' DAY)
GROUP BY 1
ORDER BY total_cost_usd DESC;

LiteLLM / Bedrock Converse API Bug — Known Active Issue

LiteLLM translates OpenAI-format cache_control markers to Bedrock’s native cachePoint format. Several bugs remain active as of mid-2026:

  1. TTL stripping (issue #17250, #15880): LiteLLM silently strips the ttl parameter from Bedrock requests. The 1-hour TTL option is supported by AWS but unreachable via LiteLLM. Workaround: use the Bedrock SDK directly for long-TTL use cases.

  2. Assistant and tool message cache points dropped (issue #12695): Cache breakpoints in assistant turns and tool result messages are dropped before the request reaches Bedrock. This hurts agentic loops that rely on caching tool outputs.

  3. Content array format regression (issue #23247): When messages use the content-block array format (required for vision/multi-modal), text parts are reconstructed with only type and text, silently dropping cache_control. Affects any multi-modal pipeline.

  4. Tool configuration caching unsupported (issue #21969): Bedrock Converse supports caching toolConfig.tools for Claude models, but LiteLLM has no cache_control_injection_points support for tool arrays.

Mitigation: For production Bedrock + caching, pin LiteLLM versions and test cache hit rates via CUR. Alternatively, use the Anthropic Bedrock SDK directly for cache-critical paths.

2. Context Window Bloat — The Hidden Cost Driver

Why Context Grows Quadratically in Cost Terms

In a multi-turn conversation, each API call sends the full history. After N turns of T tokens each:

Total tokens billed = T + 2T + 3T + ... + NT = T × N(N+1)/2

A 10-turn conversation with 500 tokens per turn bills 500 × 55 = 27,500 input tokens, not 5,000. At Sonnet pricing, that is $0.083 vs $0.015. A 50-turn support conversation multiplies cost by 25× relative to a naive single-call estimate.

Agentic loops compound this: every tool call result appended to the context inflates subsequent calls. An agent with 10 tool calls per task, each returning 500 tokens, adds 5,000 tokens to every subsequent step.

Cost Impact at Scale

Effective Context Tokens/Request Daily Cost (10M tokens/day, Sonnet) vs Baseline
4K 4,000 $12.00 1×
8K 8,000 $24.00 2×
32K 32,000 $96.00 8×
128K 128,000 $384.00 32×

“10M tokens/day” refers to output tokens for context — real input tokens grow proportionally. These figures assume the context is the dominant cost driver (typical in multi-turn or RAG pipelines).

Memory Strategy Comparison

Full history (default): No implementation cost, O(N²) token cost. Acceptable only for short sessions (<5 turns).

Sliding window: Keep the last K turns. Simple to implement. Loses older context entirely — problematic for reference-heavy tasks.

def sliding_window_messages(messages: list, max_turns: int = 6) -> list:
    """Keep system message + last max_turns conversation turns."""
    system = [m for m in messages if m["role"] == "system"]
    conversation = [m for m in messages if m["role"] != "system"]
    return system + conversation[-max_turns * 2:]  # pairs of user/assistant

LangChain ConversationSummaryBufferMemory: Maintains a running LLM-generated summary of older turns, plus verbatim recent turns. Costs one extra summarization call when the buffer limit is hit, but keeps context roughly constant.

from langchain.memory import ConversationSummaryBufferMemory
from langchain_anthropic import ChatAnthropic

llm = ChatAnthropic(model="claude-3-haiku-20240307")  # Use Haiku for summaries — not Sonnet
memory = ConversationSummaryBufferMemory(
    llm=llm,
    max_token_limit=2000,  # summarize when rolling buffer exceeds this
    return_messages=True
)

Cost note: use Haiku (not Sonnet) for the summarization step. The summary model does not need reasoning capability — it needs compression. Haiku costs 12× less than Sonnet.

LLMLingua / LongLLMLingua — Semantic Compression

LLMLingua (Microsoft) uses a small proxy LLM to score token importance and drop low-scoring tokens from the context before sending to the main model. This is distinct from summarization — it preserves original tokens but at lower density.

Measured performance:

  • LLMLingua-2: 3–6× faster than LLMLingua-1, trained via GPT-4 distillation
  • Practical compression ratios: 4× to 10× with <5% task performance degradation on QA benchmarks
  • LongLLMLingua: 21.4% performance improvement on NaturalQuestions at 4× token reduction (GPT-3.5-Turbo)
  • LooGLE benchmark: 94% cost reduction reported at the extreme end (10–20× compression)
from llmlingua import PromptCompressor

compressor = PromptCompressor(
    model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank",
    use_llmlingua2=True,
    device_map="cpu"  # runs locally, no API cost
)

compressed = compressor.compress_prompt(
    context_tokens,
    rate=0.5,           # target 50% of original tokens
    force_tokens=["\n", ".", "?"],  # never compress sentence boundaries
    drop_consecutive=True
)
# compressed["compressed_prompt"] is the reduced context
# compressed["ratio"] reports actual compression achieved

Quality tradeoff: At 2× compression, quality degradation is negligible for extraction and classification tasks. At 5× compression, expect 3–8% drop on reasoning tasks. At 10×, only use for coarse retrieval or pre-filtering, not final generation.

When to use what:

Scenario Strategy
Chat sessions, <5 turns Full history
Chat sessions, >5 turns SummaryBufferMemory (Haiku summarizer)
RAG retrieval context LLMLingua-2 at 2–4×
Agentic tool output accumulation Sliding window + tool output truncation
Long document analysis LongLLMLingua before send

3. Model Routing by Complexity

The Cost Multiplier Reality

Claude 3.5 Sonnet costs 12× more than Claude 3 Haiku per token. GPT-4o costs roughly 6× more than GPT-4o-mini. For classification, extraction, and slot-filling tasks, the cheaper model performs comparably or identically. The cost reduction from right-sizing model selection on classification tasks is typically 70–80%.

Task Type Appropriate Tier Example Models
Classification, intent detection Tier 1 (nano) Haiku, GPT-4o-mini, Llama 3 8B
Extraction, structured parsing Tier 1–2 Haiku, GPT-4o-mini
Single-step reasoning, summarization Tier 2 (mid) Sonnet, GPT-4o-mini
Multi-step reasoning, code generation Tier 2–3 Sonnet, GPT-4o
Complex synthesis, adversarial review Tier 3 (large) Opus, GPT-4o

LiteLLM Complexity Router Configuration

LiteLLM’s built-in complexity router scores requests with rule-based heuristics (prompt length, structural cues, keyword signals) and routes to configured tiers. It operates in sub-millisecond latency with zero external API calls.

# litellm_config.yaml
router_settings:
  routing_strategy: "complexity-based-routing"

model_list:
  - model_name: tier-1-fast
    litellm_params:
      model: bedrock/anthropic.claude-3-haiku-20240307-v1:0
      aws_region_name: us-east-1

  - model_name: tier-2-reasoning
    litellm_params:
      model: bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
      aws_region_name: us-east-1

  - model_name: tier-3-complex
    litellm_params:
      model: bedrock/anthropic.claude-3-opus-20240229-v1:0
      aws_region_name: us-east-1

complexity_based_routing:
  simple:
    threshold: 30
    model: tier-1-fast
  medium:
    threshold: 60
    model: tier-2-reasoning
  complex:
    threshold: 100
    model: tier-3-complex

Known issue (March 2026): The complexity router fails when messages use the content-block array format (multi-modal). Plain string content works correctly. File issue #23247 for tracking.

Building a Custom Routing Classifier

The built-in complexity router uses surface-level signals. For higher precision, build a task-type classifier:

Option A: Rule-based (fastest, no model cost)

import re

EXTRACTION_PATTERNS = re.compile(
    r'\b(extract|parse|identify|list all|find all|what is the|return the)\b',
    re.IGNORECASE
)
REASONING_PATTERNS = re.compile(
    r'\b(analyze|evaluate|compare|recommend|explain why|design|architect)\b',
    re.IGNORECASE
)

def route_request(prompt: str, context_tokens: int) -> str:
    if context_tokens < 500 and EXTRACTION_PATTERNS.search(prompt):
        return "tier-1-fast"
    elif REASONING_PATTERNS.search(prompt) or context_tokens > 8000:
        return "tier-3-complex"
    else:
        return "tier-2-reasoning"

Option B: Embedding similarity (medium cost, high precision)

from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

# Pre-embed labeled examples offline
TIER_CENTROIDS = {
    "tier-1": np.load("centroids/tier1.npy"),
    "tier-2": np.load("centroids/tier2.npy"),
    "tier-3": np.load("centroids/tier3.npy"),
}

def route_by_embedding(prompt_embedding: np.ndarray) -> str:
    scores = {
        tier: cosine_similarity([prompt_embedding], [centroid])[0][0]
        for tier, centroid in TIER_CENTROIDS.items()
    }
    return max(scores, key=scores.get)

Use text-embedding-3-small at $0.02/MTok for routing — the embedding cost is negligible against the model cost savings.

Option C: Fine-tuned small classifier

Fine-tune a distilbert or sentence-transformer model on labeled routing examples (500–2000 labeled prompts). Achieves >95% accuracy at sub-1ms inference cost. Use ONNX runtime for CPU deployment alongside the LiteLLM proxy.

4. Few-Shot vs Zero-Shot Economics

Token Cost of Few-Shot Examples

Each few-shot example adds tokens to every request. Three examples at 300 tokens each add 900 input tokens per call. At Sonnet pricing and 10M calls/day, that is $27/day for the examples alone — before any actual task content.

Approach Extra Input Tokens Benefit When to Use
Zero-shot 0 Cheapest Simple tasks with structured output
1-shot ~300 Formatting anchor When output format is critical
3-shot ~900 Consistency When model misunderstands task
5-shot ~1,500 Edge case coverage Complex classification with rare classes

Structured Output as Few-Shot Replacement

JSON mode / structured output (response_format: {type: "json_object"}) often eliminates the need for format-anchoring few-shot examples. The model produces valid JSON without examples demonstrating the format.

# Instead of 3 few-shot examples showing JSON format:
response = client.messages.create(
    model="claude-3-haiku-20240307",
    max_tokens=256,
    system="""Extract entities from the text. Return ONLY valid JSON matching this schema:
{"entities": [{"name": string, "type": "person"|"org"|"location", "confidence": 0.0-1.0}]}
No other text.""",
    messages=[{"role": "user", "content": document_text}]
)

This approach removes 600–1,500 input tokens per request while maintaining format consistency, at the cost of occasionally requiring output validation and retry logic.

Chain-of-Thought Cost Analysis

CoT reasoning tokens are output tokens, which cost 3–5× more per token than input on most providers (Sonnet: $3/MTok input vs $15/MTok output). A 500-token reasoning chain costs $0.0075 per request. At 1M requests/day, that is $7,500/day — for reasoning tokens that are discarded and never returned to the user.

When to skip CoT:

  • Classification and extraction tasks (accuracy gain is marginal)
  • Tasks with deterministic correct answers (the model either knows or doesn’t)
  • High-volume, low-stakes inference (summarization, tagging, routing)

When CoT is worth the cost:

  • Multi-step math or logic
  • Code generation with edge cases
  • Tasks where a wrong confident answer causes downstream failures

Extended thinking (Claude 3.7+): Anthropic’s extended thinking produces separate reasoning tokens billed at output token rates. These are valuable for complex reasoning but should be explicitly disabled for simpler subtasks. Pass thinking: {type: "disabled"} explicitly in task decomposition architectures.

Output Token Minimization

Output tokens cost 5× input tokens on Sonnet. Techniques to reduce output length:

  1. Constrained schemas: Ask for a JSON object with specific fields rather than prose. Prose explanations add 200–500 tokens. A structured response with the same information is 50–150 tokens.

  2. Explicit length directives: "Respond in 3 sentences or fewer." Models follow this reliably on extraction tasks.

  3. Binary/categorical outputs for routing: "Answer YES or NO only." For gate-keeping steps, this is a 99% output token reduction.

  4. Enumerate, don’t explain: "List only. No explanations." Cuts output by 60–80% on list generation tasks.

5. Batch and Async Patterns

Bedrock Batch Inference: 50% Discount

Bedrock batch inference processes JSONL input asynchronously from S3 at 50% of on-demand token pricing. Supported across Anthropic Claude, Meta Llama, Mistral, and Amazon Nova models. Maximum latency: 24 hours.

Economics at scale:

Daily Volume On-Demand Cost (Sonnet) Batch Cost Annual Savings
10M tokens $30/day $15/day $5,475/year
100M tokens $300/day $150/day $54,750/year
1B tokens $3,000/day $1,500/day $547,500/year

Use cases for batch:

  • Nightly document processing pipelines (contracts, reports, transcripts)
  • Offline evaluation/scoring runs against labeled datasets
  • Bulk classification or tagging of existing document corpora
  • Embedding backfills when adding new content to a vector store

S3 → Bedrock Batch Pattern

import boto3
import json
import uuid
from datetime import datetime

bedrock = boto3.client("bedrock", region_name="us-east-1")

def submit_batch_job(
    prompts: list[str],
    model_id: str = "anthropic.claude-3-5-sonnet-20241022-v2:0",
    s3_bucket: str = "my-batch-bucket"
) -> str:
    job_id = str(uuid.uuid4())[:8]
    input_key = f"batch-inputs/{job_id}.jsonl"
    output_prefix = f"batch-outputs/{job_id}/"

    # Write JSONL to S3
    s3 = boto3.client("s3")
    records = "\n".join(
        json.dumps({
            "recordId": str(i),
            "modelInput": {
                "anthropic_version": "bedrock-2023-05-31",
                "max_tokens": 1024,
                "messages": [{"role": "user", "content": prompt}]
            }
        })
        for i, prompt in enumerate(prompts)
    )
    s3.put_object(Bucket=s3_bucket, Key=input_key, Body=records.encode())

    # Submit batch job
    response = bedrock.create_model_invocation_job(
        jobName=f"batch-{job_id}-{datetime.now():%Y%m%d}",
        modelId=model_id,
        inputDataConfig={
            "s3InputDataConfig": {
                "s3Uri": f"s3://{s3_bucket}/{input_key}"
            }
        },
        outputDataConfig={
            "s3OutputDataConfig": {
                "s3Uri": f"s3://{s3_bucket}/{output_prefix}"
            }
        }
    )
    return response["jobArn"]

SQS → Lambda → Bedrock Batch — Production Pattern

For production pipelines where documents arrive continuously but latency tolerance is 1–24 hours:

Documents → SQS FIFO queue
    → EventBridge Scheduler (triggers every 4 hours)
    → Lambda aggregator (reads SQS, builds JSONL, writes to S3)
    → Bedrock CreateModelInvocationJob
    → S3 output prefix
    → Lambda consumer (reads JSONL results, writes to downstream store)
    → SQS dead-letter queue (failed records)

Key design decisions:

  • 4-hour aggregation window balances batch efficiency against freshness requirements. Reduce to 1 hour if SLA requires. Never go below 500 records per batch job — setup overhead makes smaller batches uneconomical.
  • FIFO queue + message deduplication prevents duplicate submissions.
  • Lambda consumer must be idempotent — Bedrock may write partial output files before completing.

LiteLLM Batch Submission

import litellm

batch_requests = [
    {"model": "bedrock/claude-3-5-sonnet", "messages": [{"role": "user", "content": p}]}
    for p in prompts
]

# LiteLLM's async gather pattern (not true Bedrock batch — uses concurrent on-demand calls)
import asyncio

async def batch_complete(requests):
    tasks = [litellm.acompletion(**req) for req in requests]
    return await asyncio.gather(*tasks, return_exceptions=True)

results = asyncio.run(batch_complete(batch_requests))

Note: LiteLLM’s gather pattern uses concurrent on-demand API calls, not Bedrock’s native batch inference. You pay on-demand pricing. For true 50% discount, use the Bedrock CreateModelInvocationJob API directly.

6. Embedding Cost Optimization

Model Cost vs Quality (2026)

Model Price (standard) Price (batch) MTEB Retrieval Notes
OpenAI text-embedding-3-small $0.02/MTok $0.01/MTok 62.3% Best cost/quality
OpenAI text-embedding-3-large $0.13/MTok $0.065/MTok 64.6% 6.5× more for +2.3 pts
Amazon Titan Embeddings v2 $0.02/MTok N/A ~60% No batch discount
Cohere Embed 4 $0.12/MTok N/A 63.1% Multilingual strength
Voyage-4-lite $0.02/MTok N/A ~63% Lowest latency

For most production RAG pipelines: text-embedding-3-small via the Batch API at $0.01/MTok is the correct default. The quality gap vs large models is 2–3 NDCG points, which is imperceptible in end-user retrieval quality for most document types.

Cache Embeddings for Stable Documents

Embeddings for documents that do not change should be computed once and stored. The cost per re-embedding is pure waste.

import hashlib
import numpy as np
from functools import lru_cache

class EmbeddingCache:
    def __init__(self, store):  # store: Redis, DynamoDB, local file
        self.store = store

    def embed(self, text: str, model: str = "text-embedding-3-small") -> np.ndarray:
        cache_key = hashlib.sha256(f"{model}:{text}".encode()).hexdigest()
        cached = self.store.get(cache_key)
        if cached is not None:
            return np.frombuffer(cached, dtype=np.float32)

        # Only call API on cache miss
        embedding = self._call_api(text, model)
        self.store.set(cache_key, embedding.tobytes(), ex=86400 * 30)  # 30-day TTL
        return embedding

Re-embedding triggers: Only re-embed on content change (hash mismatch), not on schedule. Static knowledge bases (product catalogs, legal docs, policy manuals) can go months without re-embedding.

Matryoshka Embeddings — Dimensionality Reduction

Matryoshka Representation Learning (MRL) trains embeddings to front-load semantic information into the earliest dimensions, enabling truncation without retraining.

from openai import OpenAI

client = OpenAI()

# Full 1536 dimensions — maximum quality
full = client.embeddings.create(
    model="text-embedding-3-large",
    input=documents,
    dimensions=1536
)

# Truncate to 768 — 94.6% of full performance, 50% storage reduction
reduced = client.embeddings.create(
    model="text-embedding-3-large",
    input=documents,
    dimensions=768  # native API support for MRL truncation
)

Storage and cost impact at scale:

Dimensions Storage per 1M docs vs Full Quality Retention
1536 ~6 GB (float32) 1× 100%
768 ~3 GB 0.5× 94.6%
256 ~1 GB 0.17× ~90%

For 10M document corpora: reducing from 1536 to 768 dimensions cuts vector store storage cost in half with no meaningful retrieval quality loss. At Pinecone/Weaviate pricing (~$0.08/GB/month), this saves approximately $240/month per 10M documents.

Re-Embedding Frequency Strategy

Static documents (policies, manuals):     Embed once. Re-embed on content version bump.
Semi-static (product catalog):            Weekly batch re-embedding. Use Batch API.
Dynamic (news, tickets, chat):            Real-time embedding at ingest. Monitor drift quarterly.
Experimental (new model evaluation):      Parallel embed a 1% sample before full migration.

7. Measurement Framework

Token Efficiency Ratio

The core metric for prompt engineering quality:

TER = useful_output_tokens / total_input_tokens

Where useful_output_tokens is the parsed, structured output that your application actually consumes (not chain-of-thought, not preamble). A well-optimized pipeline has TER > 0.3. A bloated one often sits at 0.05–0.10.

def compute_ter(response) -> float:
    """Compute token efficiency ratio from an Anthropic response."""
    total_input = response.usage.input_tokens
    # Approximate useful output: strip CoT blocks if present
    output_text = response.content[0].text
    useful_tokens = response.usage.output_tokens

    # If CoT is embedded in prose, estimate: structured JSON ÷ total output
    import json
    try:
        parsed = json.loads(output_text)
        useful_tokens = len(json.dumps(parsed).split()) * 1.3  # rough token estimate
    except json.JSONDecodeError:
        pass

    return useful_tokens / max(total_input, 1)

Retry Rate as Quality Proxy

Failed structured output parses that trigger retries are a quality signal — and a cost multiplier. Each retry doubles the cost of that request.

import litellm
from litellm import completion
import json

def tracked_completion(prompt: str, model: str, max_retries: int = 2) -> dict:
    metrics = {"attempts": 0, "total_input_tokens": 0, "total_output_tokens": 0}

    for attempt in range(max_retries + 1):
        metrics["attempts"] += 1
        response = completion(model=model, messages=[{"role": "user", "content": prompt}])
        metrics["total_input_tokens"] += response.usage.prompt_tokens
        metrics["total_output_tokens"] += response.usage.completion_tokens

        try:
            result = json.loads(response.choices[0].message.content)
            metrics["success"] = True
            return {"result": result, "metrics": metrics}
        except json.JSONDecodeError:
            if attempt == max_retries:
                metrics["success"] = False
                return {"result": None, "metrics": metrics}

Track retry_rate = (total_attempts - total_requests) / total_requests. A rate above 5% indicates the prompt needs few-shot examples or a tighter output schema. A rate above 15% indicates the wrong model tier — upgrade.

A/B Testing Prompt Changes for Cost Impact

Never deploy a prompt change to 100% of traffic without measuring cost delta. Use LiteLLM’s callback hooks to log per-variant metrics.

import litellm
from litellm.integrations.custom_logger import CustomLogger

class PromptABLogger(CustomLogger):
    def __init__(self, experiment_id: str):
        self.experiment_id = experiment_id
        self.variants = {}

    def log_success_event(self, kwargs, response_obj, start_time, end_time):
        variant = kwargs.get("metadata", {}).get("variant", "control")
        model = kwargs.get("model", "unknown")

        key = f"{self.experiment_id}:{variant}:{model}"
        if key not in self.variants:
            self.variants[key] = {"requests": 0, "input_tokens": 0, "output_tokens": 0, "cost": 0}

        usage = response_obj.usage
        self.variants[key]["requests"] += 1
        self.variants[key]["input_tokens"] += usage.prompt_tokens
        self.variants[key]["output_tokens"] += usage.completion_tokens
        self.variants[key]["cost"] += litellm.completion_cost(completion_response=response_obj)

    def report(self) -> dict:
        """Summarize cost per request per variant."""
        return {
            k: {
                **v,
                "cost_per_request": v["cost"] / max(v["requests"], 1),
                "avg_input_tokens": v["input_tokens"] / max(v["requests"], 1)
            }
            for k, v in self.variants.items()
        }

logger = PromptABLogger("system-prompt-v2-test")
litellm.callbacks = [logger]

# Tag each request with variant
def call_with_variant(prompt: str, variant: str) -> str:
    response = litellm.completion(
        model="bedrock/claude-3-5-sonnet",
        messages=[{"role": "user", "content": prompt}],
        metadata={"variant": variant}
    )
    return response.choices[0].message.content

# Route 10% to treatment, 90% to control
import random
variant = "treatment" if random.random() < 0.10 else "control"
result = call_with_variant(user_prompt, variant)

After 1,000 requests per variant, compare cost_per_request and downstream quality metrics (task success rate, retry rate, human eval scores). Require both to improve before full rollout.

Quick Reference — Cost Reduction Levers by Impact

Lever Complexity Typical Cost Reduction Risk
Model routing (right-sizing) Medium 70–80% on classification tasks Accuracy regression on edge cases
Batch inference (async workloads) Low 50% Latency increase to 24h
Prompt caching (stable system prompts) Low 30–60% on repeated-prefix workloads Hit rate must stay >30%
Context window management Medium 40–80% on multi-turn Loss of distant context
Output minimization + structured output Low 20–40% Parse failures if schema is too strict
LLMLingua compression (RAG context) High 60–90% on retrieval context 3–8% task accuracy loss
Embedding dimensionality reduction Low 50% storage (matryoshka) 5% retrieval quality loss
Batch embedding (OpenAI Batch API) Low 50% embedding cost Async only

Starting point recommendation: Deploy model routing and batch inference first — both have high leverage and low implementation risk. Prompt caching comes second for any deployment with a large, stable system prompt. Context window management is essential for any multi-turn or agentic use case that runs at volume. LLMLingua compression is a later-stage optimization once the others are baseline.

Sources