← Agent Frameworks 🕐 20 min read

Agent Frameworks

Prompt Engineering for Cost Optimization — Token Efficiency Techniques (2026)

1. Token Counting Fundamentals · March 2026

Token spend on LLM APIs is now a first-class engineering concern.

Token spend on LLM APIs is now a first-class engineering concern. Output tokens cost 3–5× more than input tokens; reasoning/thinking tokens are billed at output rates; tool definitions load on every request whether used or not; and non-English content can cost 65–340% more than equivalent English text. This note synthesizes the current state of each lever available to reduce token consumption without degrading quality — with measured compression ratios, before/after examples, and decision rules for when each technique applies.

How Claude and Amazon Nova Tokenize

Both Claude and Amazon Nova use Byte Pair Encoding (BPE) variants, but with meaningfully different vocabulary sizes and training corpora, which produces different token counts for identical inputs.

Claude (Anthropic): Anthropic uses a proprietary BPE tokenizer with a vocabulary of approximately 65,000 tokens. English prose runs at roughly 4–5 characters per token — comparable to OpenAI’s cl100k_base. Claude Opus 4.7 introduced an updated tokenizer that can map the same input to 1.0–1.35× more tokens depending on content type; factor this into any cost projection for that model generation.

Amazon Nova: Nova models (Nova Micro, Nova Lite, Nova Pro) use a vocabulary closer to 100,000 tokens, which in principle offers more compact encoding for rare and compound words. However, because token boundaries differ, a prompt that counts at 1,000 tokens on Claude may count at 800–1,100 on Nova for the same text — making cross-model cost comparisons unreliable without measuring directly.

Practical implications:

Content type	Claude tokens (est.)	Behavioral note
English prose	~250 tokens / 1,000 chars	Baseline
JSON with keys	~350–450 tokens / 1,000 chars	Keys and quotes add 40–80% overhead
Code (Python)	~200–280 tokens / 1,000 chars	Indentation tokenizes efficiently
Korean / Japanese	3–5× English equivalent	CJK languages pay a structural tokenization penalty
Mathematical notation	1.2–1.5× English	LaTeX symbols often split across tokens

Measurement rule: Never estimate token counts from character counts in production. Use anthropic.count_tokens() (Claude API) or bedrock:CountTokens (Bedrock) before committing to a prompt design. Token count varies by model family and version.

2. System Prompt Compression

System prompts are the highest-leverage compression target because they repeat on every call. A 2,000-token system prompt that fires 10,000 times per day costs 20 million input tokens per day — before any user messages.

Compression Techniques

Remove procedural filler. Instructions like “Please make sure to carefully read the following guidelines before responding” add tokens without adding information the model uses for generation.

Collapse enumerated rules into structured directives. Natural language lists tokenize inefficiently compared to tight bullet syntax.

Use examples only when the task has non-obvious format requirements. A single well-chosen example often outperforms three marginal ones (see Section 3).

Replace prose with structured sections. XML-tagged sections (<rules>, <format>, <constraints>) let you strip connecting prose entirely.

Before/After Example

Before (verbose, 187 tokens):

You are a helpful customer service assistant for Acme Corp. Your job is to help 
customers with their questions and concerns. You should always be polite and 
professional. When responding, please make sure to address the customer's specific 
question directly. You should avoid using jargon or technical language that the 
customer might not understand. If you don't know the answer to something, please 
say so rather than making something up. Always end your responses by asking if 
there is anything else you can help with.

After (compressed, 74 tokens — 60% reduction):

<role>Acme Corp customer service</role>
<rules>
- Address the specific question first
- Plain language only; no jargon
- Admit uncertainty rather than guess
- Close: ask if further help needed
</rules>

Token delta: −113 tokens per call. At 10,000 calls/day on Claude Sonnet 4.6 ($3/M input tokens): $1.23/day saved, $450/year from one prompt rewrite.

Compression Ratios Observed in Practice

Technique	Typical token reduction	Quality risk
Remove filler phrases	10–20%	Negligible
Collapse prose rules → bullets	20–35%	Negligible
XML structure with tag compression	25–45%	Low
Semantic summarization of examples	40–60%	Medium — test task accuracy
LLMLingua automated compression	50–80%	Medium — requires evaluation
Soft-prompt compression (500xCompressor)	Up to 480×	High — specialized setup required

3. Few-Shot Example Selection

The Cost Equation

Each few-shot example adds tokens proportional to its length on every call. A 300-token example in a system prompt firing 50,000 times per month costs 15 million tokens. The question is whether the accuracy gain justifies that spend.

Research finding (2025): For most reasoning tasks, 1–3 examples reach the accuracy plateau. In generation tasks (summarization, drafting), adding examples continues to help up to roughly 5–8 before returns diminish. Beyond that, there is documented accuracy degradation from domain mismatch — accuracy dropped from 67.2% to 53.9% when example count increased from 0 to 100 in a domain-shifted setting.

Decision Framework

Task type                   Minimum effective examples
─────────────────────────────────────────────────────
Classification / routing    1–2 (boundary cases only)
Extraction / parsing        1–3 (cover format variations)
Reasoning / math            1 (show step structure once)
Generation (summaries)      2–5 (show length + tone)
Format-critical output      1 per distinct format variant

Selecting Examples for Token Efficiency

Shortest example that demonstrates the boundary. A 40-token example that shows the unusual case beats a 200-token example that shows the easy case.
Vary examples by decision boundary, not by topic. Two examples that look different but map to the same behavior waste tokens.
Measure per-example marginal accuracy gain. Remove examples one at a time and eval. Most systems tolerate removal of 50–80% of examples with <2% accuracy loss.
Dynamic example injection. For embedding-retrieved few-shot (selecting examples per-query), you pay only for the examples relevant to each call rather than a fixed set. This typically cuts few-shot token spend by 40–70%.

4. Output Length Control

Why Output Tokens Matter More

Output tokens are priced at a premium across every major API: 3× on Claude Haiku, 5× on Claude Sonnet, and similarly on Bedrock Nova models. A system that spends 1,000 tokens on input and generates 500 on output spends 71% of its token budget on the output side. Output length optimization is therefore the highest-ROI single lever.

`max_tokens` Tuning

Set max_tokens as a hard ceiling. Most production prompts never require the default ceiling; leaving it at 4,096 or 8,192 invites padding.

Calibration method:

Sample 200 representative requests
Record the 95th-percentile actual token output
Set max_tokens to p95 + 20% buffer
Log truncation events; raise ceiling only if truncation rate > 0.5%

For RAG answer synthesis: p95 is often 150–300 tokens. For document drafting: 600–1,200. Using p95 instead of the default 4,096 typically yields 40–70% output token reduction.

Explicit Length Instructions

Embedding length constraints directly in the prompt reduces output length in practice by 30–50% compared to relying on max_tokens alone:

Instruction	Average output reduction
“Answer in 2–3 sentences.”	~60% vs. unconstrained
“List only the top 3 items.”	~75% vs. full enumeration
“Respond in under 50 words.”	~70% vs. unconstrained
“JSON only, no explanation.”	~80% vs. prose + JSON

Structured vs. Prose Output

Requesting structured output formats prevents explanatory preamble and closing remarks, which commonly add 50–150 tokens per response (“Great question! Here is what I found…” / “I hope this helps!”). JSON-mode or XML-tagged output eliminates this class of verbosity entirely.

5. Context Window Management for Long Conversations

The Cost Curve

In a multi-turn conversation, every turn re-sends the entire prior context unless you actively manage it. A 10-turn conversation where each turn averages 300 tokens generates 1+2+3…+10 × 300 = 16,500 cumulative tokens — 3.3× the token volume of the raw content. At 100 conversations/day, this overhead exceeds the base content cost by 230%.

Sliding Window

Drop the oldest N turns when context exceeds a threshold. Simple to implement, zero additional LLM calls, no quality loss for tasks where recent context dominates.

Use when: Tasks are locally coherent — coding sessions, document editing, customer service where each message is mostly self-contained.

Downside: Loses early context. If turn 1 contained a constraint or preference, it will be forgotten after the window advances past it.

Configuration pattern:

WINDOW_TURNS = 6          # Keep last 6 turns
SYSTEM_ALWAYS_INCLUDED = True

Summarization

Periodically compress older turns into a rolling summary. Requires one LLM call per compression event (typically every 10–15 turns). LLM-based summarization has been shown to reduce API costs by up to 2× with no degradation in agent performance. Mem0-style atomic-fact extraction (extracting discrete facts rather than prose summaries) achieves 80–91% token reduction over full-context approaches while improving response quality by 26%.

Use when: Conversations persist across sessions, early context contains durable preferences or constraints, or conversation history is used for personalization.

Cost model:

Compression call cost = (N_turns_compressed × avg_turn_tokens) × input_price
                        + summary_tokens × output_price

Break-even: 1 compression call pays for itself when the summary is
            reused in ≥ 3 subsequent turns (typical ratio is 10:1 turns per compression)

Hybrid Pattern (Production Default)

Maintain: system prompt (always) + rolling summary of turns 1…N-8 + full text of turns N-7…N. This combines the accuracy of recency bias with the cost savings of compression on the bulk of history.

6. Tool/Function Definition Overhead

The Hidden Cost

Tool definitions are transmitted as part of every API request that includes tools, even if none of the tools are invoked. Real-world measurements have found 55,000–134,000 tokens of tool-definition overhead before any user message or task content — the result of loading all available MCP server tools at startup.

Cost example: 134,000-token tool overhead on Claude Sonnet 4.6 at $3/M input tokens = $0.40 per request. At 1,000 requests/day = $400/day = $146,000/year — solely from unused tool definitions.

Minimization Strategies

1. Dynamic tool loading. Load only the tools relevant to the current task. One production implementation reduced overhead from 134,000 to 8,700 tokens (85% reduction) by implementing a tool-search pattern: a lightweight router selects which tools to include based on the user’s query before constructing the full API call.

2. Compress tool descriptions. Tool description fields are often written for human readability. Trim to the minimum the model needs:

Before (32 tokens):

{
  "description": "This function retrieves the current weather conditions 
    for a given city, including temperature, humidity, and wind speed."
}

After (14 tokens):

{
  "description": "Get current weather (temp, humidity, wind) for a city."
}

3. Compress parameter descriptions. Same principle: "description": "The name of the city for which you want to retrieve weather data" → "description": "City name".

4. Remove optional parameters when not needed. Every parameter definition adds tokens. If a parameter is rarely used, consider a variant tool schema that omits it.

5. Prefer direct commands over MCP tool wrappers where practical. A shell command has zero schema overhead; an equivalent MCP tool pays schema cost on every call.

Tool Schema Token Audit (Example Ratios)

Tool complexity	Tokens per definition	After compression
Simple lookup (1 param)	40–80	15–25
Medium tool (3–5 params)	120–250	50–90
Complex tool (8+ params)	300–600	100–200
Full MCP server (20 tools)	4,000–12,000	1,500–4,000

7. RAG Context Compression

The Token Math in RAG

A typical RAG pipeline retrieves k=5 chunks of 512 tokens each = 2,560 tokens of context per query, before the question itself or any system instructions. At 50,000 queries/day on Sonnet 4.6: 128 million context tokens/day = $384/day = $140,000/year — for retrieved context alone.

Reducing k

Most production RAG deployments use k=5–10 based on convention rather than measurement. Evaluation typically shows accuracy plateaus at k=3 for single-hop QA and k=5 for multi-hop. Reducing k from 5 to 3 cuts retrieved context by 40% with <2% accuracy loss on most tasks.

Chunk Compression at Retrieval Time

Rather than passing raw chunks to the LLM, compress them:

Extractive compression: Select the sentences within each chunk most relevant to the query (using BM25 or a cross-encoder). A 512-token chunk typically compresses to 80–150 tokens retaining the answer-relevant content.

Abstractive compression: Summarize each chunk in 1–2 sentences before injecting. Higher quality than extractive; costs one small-model call per chunk. Net positive when the compression ratio exceeds ~5:1.

AttnComp (2025): Attention-guided adaptive context compression. Demonstrated 17× compression rate while reducing end-to-end RAG latency to 49% of the uncompressed baseline. Requires a finetuned compression model.

Late chunking: Embed at the document level, then extract relevant spans at query time — eliminates the chunk-granularity mismatch problem where an answer straddles chunk boundaries. Reduces false retrievals and the need for large k values.

Compression Ratio vs. Quality Table (RAG Context)

Method	Compression ratio	Accuracy delta	Setup cost
Reduce k from 5→3	1.7×	−1–2%	Zero
Extractive sentence selection	3–5×	−2–5%	Low
Summary-based retrieval	5–10×	−3–8%	Medium
Late chunking	2–3×	+1–3% (improved)	Medium
AttnComp (finetuned)	17×	−1–3%	High

8. Structured Output Format Token Efficiency

Format Comparison

The choice of output format has a measurable but often underappreciated effect on token count:

Benchmark results (same structured data, different formats):

Format	Relative token count	Notes
Markdown (headers + bullets)	1.0× (baseline)	Most token-efficient for hierarchical prose
YAML	1.1×	Slightly more verbose than Markdown
XML	1.3–1.5×	Tag overhead; useful for parse precision
JSON (pretty-printed)	1.8–2.5×	Keys, quotes, and brackets expensive
JSON (minified)	1.2–1.4×	Efficient for deeply nested structures
TSV / CSV	0.7–0.9×	Most efficient for tabular data

Markdown uses 34–38% fewer tokens than JSON and ~10% fewer than YAML for comparable hierarchical content. However, JSON minified becomes the efficiency leader for complex nested structures. For tabular data with uniform columns, TSV is the clear winner.

Important caveat: Forcing JSON output can degrade reasoning quality by 10–15% on tasks where the model’s natural reasoning format is prose. This is a real tradeoff — measure accuracy, not just tokens.

Practical Rules

Tabular data → TSV or CSV. The 30–40% token savings over JSON are reliable and quality-neutral.
Config / hierarchical settings → YAML. More compact than JSON with equivalent machine parseability.
Prose with sections → Markdown. Default for human-readable structured output.
Strict machine parsing where downstream code owns the schema → JSON minified. Accept the token cost for reliability.
Delimiter injection (TOON format, 2025): For uniform tabular data, a new format using special delimiter tokens demonstrated 40% fewer tokens than equivalent JSON with no accuracy loss.

9. Chain-of-Thought: Cost vs. Value

The Core Tradeoff

Explicit chain-of-thought (CoT) prompting inflates token costs 2–5× and adds latency — yet delivers no measurable accuracy gain for most production tasks. For frontier models (Claude 3.5+, GPT-4o, Gemini 2.0), the models already perform internal chain-of-thought reasoning during training. Layering explicit CoT on top is redundant for most tasks.

Where explicit CoT still adds value:

Multi-step arithmetic and symbolic reasoning where the chain is genuinely long and the model is not a reasoning-specialized variant
Tasks requiring explicit audit trails (legal, compliance — where the reasoning is itself part of the deliverable)
Novel problem structures not covered by training distribution
Smaller models (Haiku, Nova Micro) that have not been trained with implicit CoT

Where explicit CoT is pure overhead:

Classification tasks (the label is the answer, not the reasoning path)
Retrieval-augmented QA where the answer is in the context
Summarization
Any task already solved at high accuracy without CoT

Token Impact of CoT

Performance improvements of only 2–5 accuracy points require 4–5× more tokens in typical benchmarks. More specifically, reasoning models gained only 2–3% accuracy from explicit CoT while taking 20–80% longer to respond. TokenSkip (2025) demonstrated that cutting CoT token count roughly in half (313 → 181 tokens on GSM8K) produced negligible accuracy loss — the redundancy in most CoT traces is substantial.

Controlling Reasoning Verbosity (Claude)

Claude’s extended thinking feature (thinking.type: "enabled") is billed at output token rates and uses a minimum budget of 1,024 tokens. The full thinking budget can run to 128K tokens.

Cost controls available:

Disable extended thinking for task categories that don’t require it (classification, retrieval, summarization)
Set budget_tokens to the minimum necessary. Anthropic recommends starting at 1,024 and incrementing based on quality measurement
Use /effort low in Claude Code contexts to lower the thinking ceiling
Route tasks to Claude Haiku 4.5 (no extended thinking) for latency-sensitive, high-volume tasks where reasoning depth is not required

# Minimal extended thinking config
response = client.messages.create(
    model="claude-sonnet-4-6",
    thinking={
        "type": "enabled",
        "budget_tokens": 2000   # vs. default 8000; verify quality holds
    },
    max_tokens=4000,
    messages=[...]
)

10. Claude-Specific: Cache-Friendly Prompt Structure

How Prompt Caching Works

Anthropic’s prompt caching creates a server-side cache keyed on the exact byte sequence of the prompt prefix up to a cache breakpoint. Cache hit tokens cost approximately 10% of full input token price. Cache write tokens cost approximately 125% of full input token price. The TTL is 5 minutes for standard requests and 1 hour for ephemeral cache.

Break-even: A cached prefix pays for itself when hit 2 or more times within the TTL window. In practice, break-even is reached within the same conversation (turn 2 onwards) for any prompt with a stable prefix.

Observed savings in production: 60–90% reduction in input token costs on workloads with a stable system prompt.

The Stable-Prefix Rule

Structure every prompt so that stable content appears before variable content — in order from most stable to least stable:

1. System prompt (static, never changes)
2. Tool definitions (static per deployment)
3. Static few-shot examples (if used)
4. Retrieved context (changes per query, but often reused across a session)
5. Conversation history (changes every turn)
6. Current user message (always new)

A cache hit occurs only if the prefix is byte-for-byte identical through the cache breakpoint. Any change in the prefix — including a timestamp, a UUID, a user ID, or a reordered rule — causes a cache miss and a cache write charge.

Common cache-busting mistakes:

Embedding datetime.now() or a request ID in the system prompt
Prepending the username to static instructions
Randomly shuffling few-shot examples
Appending a session ID to tool descriptions

Cache Breakpoint Placement

Claude allows up to 4 cache breakpoints per request (Claude 3.7+ / Sonnet 4.6+). Optimal placement:

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": LARGE_STABLE_CONTEXT,           # 30K tokens of policy docs
                "cache_control": {"type": "ephemeral"}  # Breakpoint 1
            },
            {
                "type": "text",
                "text": current_user_query              # Dynamic, not cached
            }
        ]
    }
]

One-hour TTL pattern for production: Cache the system prompt and tool definitions. This fits deployments where the prefix is truly static per deployment (not per user). Five-minute TTL fits conversation history caching across turns.

11. Prompt Compression Libraries

LLMLingua Family (Microsoft Research)

LLMLingua (EMNLP 2023): Uses a small auxiliary language model to score token importance and remove low-information tokens. Achieves up to 20× compression with minimal performance loss. Three-stage pipeline: dynamic budget controller, iterative token-level compression, instruction-tuned distribution alignment.

LongLLMLingua (ACL 2024): Optimized for long-context scenarios. Achieves 17.1% performance improvement at 4× compression on long-context benchmarks. Particularly effective for RAG context where irrelevant passages dominate retrieval.

LLMLingua-2: Data-distillation-based, task-agnostic. Reduces end-to-end latency by up to 2.9× at 2–5× compression ratios. Maintains similar performance at up to 14× compression on GSM8K. Faster than LLMLingua-1 because the compressor is smaller and the compression is per-token (binary keep/drop) rather than iterative.

Library	Best compression	Quality at 2–5×	Quality at 10×	Setup
LLMLingua	20×	Minimal loss	Small loss	Medium
LongLLMLingua	17× + performance gain	Can improve	Minimal loss	Medium
LLMLingua-2	14×	Minimal loss	Small loss	Low
EFPC (2025)	5× tested	39.9% better than LLMLingua-2	—	Medium

Selective Context (2023)

Identifies and removes semantically redundant content from prompts using self-information scoring. Achieves 2–5× compression with minimal quality loss on most NLP benchmarks. Simpler setup than LLMLingua; does not require a separate compressor model (uses the target LLM itself for scoring). Best suited for prompts with verbose background context.

500xCompressor (2024)

Soft-prompt compression: converts natural language tokens into a small set of special tokens. Achieves compression ratios up to 480×. Requires fine-tuning a decoder model on compression objectives. Quality at extreme ratios (>50×) degrades significantly on tasks requiring precise fact retrieval. Suited to research contexts and specialized deployments where the compression model can be task-tuned.

EFPC (2025)

Efficient and Flexible Prompt Compression. Outperforms LLMLingua-2 by 39.9% on single-document and 20.8% on multi-document benchmarks at 5× compression. Evaluated against standard NLP benchmarks; production results vary.

12. Evaluation Methodology for Prompt Compression

The Core Problem

Compression degrades are not uniform across task types. A system prompt compressed 50% may lose performance on edge cases while maintaining accuracy on common cases — making aggregate accuracy metrics misleading.

Recommended Evaluation Framework

Step 1: Define task-specific quality metrics before compressing.

Classification: per-class F1, not just accuracy (compressed prompts often drop minority class performance first)
Generation: ROUGE-L, BERTScore, or human evaluation on a fixed 50-question set
Extraction: exact match and partial match on a test set with known ground truth
Reasoning: decompose by problem type; complex multi-step problems degrade before simple ones

Step 2: Build a compression test set covering edge cases. The happy path typically survives aggressive compression. Build your eval set from:

Failure cases from production logs
Adversarial inputs (ambiguous, boundary-crossing, rare formats)
Minority class examples (for classification tasks)

Step 3: Sweep compression ratios.

Compression ratio → quality metric plot:

100% (no compression) ─────────────────────────── 92% accuracy
 80%                  ──────────────────────────── 91%
 60%                  ────────────────────────────  90%
 40%                  ──────────────────────────── 87%  ← inflection point
 20%                  ────────────────────────────  71%  ← cliff

Identify the inflection point — the compression ratio below which quality drops sharply. Set your production compression budget 10–15 percentage points above the inflection.

Step 4: Disaggregate by input category. Run the full eval sweep separately on:

Short inputs (<200 tokens) vs. long inputs (>2,000 tokens)
Simple vs. complex task instances
Domain-matched vs. domain-shifted examples

Step 5: Establish a regression test. Before any prompt compression change ships, run the full eval set. Define a maximum acceptable quality delta (typically 2–3% on the primary metric). Gate deployment on this check.

Quality Metrics Summary

Task type	Primary metric	Secondary metric
Classification	F1 per class	Minority class recall
Extraction	Exact match	Partial match
Summarization	ROUGE-L	Human preference
Code generation	Pass@1	Pass@5
Reasoning / math	Exact answer match	Step-level accuracy
RAG QA	Faithfulness score	Answer completeness

Summary: Cost Reduction Decision Matrix

Technique	Token savings range	Quality risk	Effort	When to apply first
System prompt compression (manual)	30–60% of prompt tokens	Low	2 hours	Always — start here
Prompt caching (stable prefix)	60–90% on cached portion	Zero	4 hours	Any app with stable system prompt
Output `max_tokens` + length instructions	30–70% of output tokens	Low	1 hour	Always — high ROI
Tool schema compression + dynamic loading	40–85% of tool tokens	Low	4–8 hours	Any agent with >5 tools
Reduce RAG k (5→3)	40% of context tokens	1–2% accuracy	2 hours	Any RAG app
Extractive RAG compression	3–5× context reduction	2–5% accuracy	1 day	High-volume RAG
Sliding window (drop old turns)	50–80% on long sessions	Medium	4 hours	Multi-turn agents
Summarization memory	80–91% vs. full history	Low	1–2 days	Persistent agents
Disable CoT on simple tasks	2–5× output reduction	Task-dependent	2 hours	Classification, retrieval
Extended thinking budget reduction	Per thinking tokens saved	Test carefully	2 hours	Claude reasoning models
LLMLingua-2 automated compression	2–14×	1–5% per 5×	1–3 days	High-volume, stable prompts
Few-shot example pruning	20–70% of example tokens	<2% with eval	4 hours	Any prompted system

Recommended sequencing: System prompt compression → prompt caching → output control → tool schema cleanup → RAG k reduction → context window management → automated compression (LLMLingua) → CoT control. Each step should be evaluated independently before proceeding.

13. Batch Processing and Model Routing

Batch API for Non-Latency-Sensitive Work

Anthropic’s Message Batches API (available on Bedrock via async inference) processes requests with a 24-hour window and charges approximately 50% of the standard input/output rate. For any workload where the result is not needed in real time — nightly enrichment, bulk classification, document processing pipelines, eval runs — batch processing is the single cheapest path with zero quality tradeoff.

Eligible workloads: Document ingestion, scheduled report generation, fine-tuning data preparation, bulk entity extraction, eval scoring, A/B prompt comparison runs.

Ineligible workloads: Interactive chat, real-time classification, any task where latency SLA < 5 minutes.

Implementation pattern:

batch = client.messages.batches.create(
    requests=[
        {"custom_id": f"item-{i}", "params": {"model": "claude-sonnet-4-6",
         "max_tokens": 256, "messages": [{"role": "user", "content": doc}]}}
        for i, doc in enumerate(documents)
    ]
)
# Poll for completion; retrieve results as JSONL

A batch of 10,000 document classifications at 500 tokens each = 5 million tokens. At standard Sonnet 4.6 input pricing ($3/M) = $15. With batch pricing (50% discount) = $7.50. Annualized at 5 batches/week = $1,950 savings per year from this single switch.

Model Routing by Task Complexity

Not every task requires Sonnet or Opus. A tiered routing strategy assigns each request to the cheapest model that meets the quality bar:

Tier	Model	Price (input/output, $/M tokens)	Use case
Light	Claude Haiku 4.5	$0.80 / $4.00	Classification, extraction, short QA
Standard	Claude Sonnet 4.6	$3.00 / $15.00	Reasoning, summarization, generation
Heavy	Claude Opus 4	$15.00 / $75.00	Complex reasoning, code architecture, research

Routing implementation: Use a lightweight classifier (Haiku itself, or a fine-tuned small model) to categorize each request by complexity before dispatching. A routing overhead of 50–100 tokens per request pays for itself if even 30% of requests are correctly downrouted from Sonnet to Haiku.

Observed savings in practice: Routing 60% of requests to Haiku and 40% to Sonnet reduces weighted average cost by 55% versus all-Sonnet at equal quality on routable task distributions.

Bedrock-Specific: Nova Model Tier

Amazon Nova Micro, Lite, and Pro offer significantly lower per-token pricing than Claude on Bedrock for tasks that do not require Claude’s specific capabilities. For high-volume extraction, classification, and structured generation tasks where Claude is not a contractual or quality requirement, the Nova tier can reduce token costs by 70–85% vs. Claude Sonnet. Apply the same quality evaluation protocol as with any model swap: measure on a task-representative eval set before committing.

Appendix: Instrumentation Checklist

Before optimizing, you need to be able to measure. This checklist ensures the necessary observability is in place.

Per-request logging (minimum viable):

input_tokens — from the API response usage field (not estimated)
output_tokens — from the API response usage field
cache_read_input_tokens — from the API response usage field (0 if not using caching)
cache_creation_input_tokens — cache write events; should amortize to near-zero over time
model — which model tier handled the request
route or task_type — enables per-task-type cost breakdown
latency_ms — output tokens/second correlates with cost and user experience together
truncated — whether max_tokens was hit; non-zero rate means ceiling is too low

Dashboards to build (in order of priority):

Daily token spend by model × task_type — reveals which routes cost the most
Cache hit rate by system prompt variant — reveals cache-busting issues
Output token length distribution by route — p50/p95/p99 guides max_tokens tuning
Tool token overhead as fraction of total input — identifies tool bloat candidates
Compression ratio trend over time (if using LLMLingua) — detects drift in prompt content that bypasses the compressor

Cost attribution formula:

daily_cost = (input_tokens × input_price)
           + (output_tokens × output_price)
           - (cache_read_input_tokens × input_price × 0.90)   # 90% discount
           + (cache_creation_input_tokens × input_price × 0.25) # 25% surcharge

Running this formula per route per day makes the dollar value of each optimization concrete before any engineering work begins.