← Agent Frameworks 🕐 19 min read
Agent Frameworks

RAG and Vector DB Cost Optimization on AWS — 2026

1. RAG Cost Anatomy · March 2026

A production RAG pipeline has five distinct cost buckets. Most FinOps reviews conflate them, leading to optimization effort aimed at the wrong layer.

Audience: Platform engineers and FinOps leads running production RAG pipelines at Fortune 500 companies.

TL;DR: Most enterprise RAG pipelines overspend by 2–4x. The cost floor is rarely the LLM — it is the vector store idle compute and oversize retrieval context sent upstream. This note maps the full cost anatomy, provides break-even formulas for each AWS vector backend, and details the retrieval and caching techniques that reduce total cost by 40–70% without degrading answer quality.

A production RAG pipeline has five distinct cost buckets. Most FinOps reviews conflate them, leading to optimization effort aimed at the wrong layer.

1.1 The Five Buckets

Bucket Cost Driver Scaling Behavior
Embedding Ingestion (one-time) Embedding API tokens × corpus size Linear with corpus; paid once at ingest, then at incremental sync
Embedding Queries (per-query) Embedding API tokens × query volume Linear with QPS; typically small per-query ($0.00002–$0.00012)
Vector Store: Storage GB stored × months Linear with corpus, grows slowly for fixed knowledge bases
Vector Store: Query Compute OCUs/pods/RCUs × hours OR per-query fee Highly variable; minimum floors dominate low-QPS workloads
LLM Inference (system prompt + retrieved context + user query + output) tokens × model rate Context window dominates; grows with k and chunk size

1.2 How Each Component Scales

Corpus size affects ingestion cost linearly and vector store storage linearly. For a 1M-token corpus at Titan V2 rates ($0.02/1M tokens), ingestion costs $0.02 — negligible. At 1B tokens it is $20. Ingest is rarely the constraint.

Query volume dominates embedding query cost and LLM cost. At 100K queries/day with Titan V2 at average 50-token queries: 5M tokens/day × $0.02/1M = $0.10/day. Embedding queries are cheap; LLM calls on the resulting context are not.

LLM context inflation is the biggest hidden cost. A direct prompt might be 500 tokens. A RAG-augmented prompt adds: system preamble (300 tokens) + k retrieved chunks (k × chunk_size tokens) + user query (50 tokens). With k=5 and 512-token chunks, you add 2,560 tokens per query. At Claude Sonnet 4 pricing ($3/1M input tokens), that is $0.0077/query additional cost — $770/day at 100K queries. Optimizing k from 5→3 saves $462/day on that workload with no other changes.

Formula — LLM input token cost per query:

cost_per_query = (system_tokens + k × chunk_tokens + query_tokens) × input_price_per_token
savings_from_reducing_k = (k_before - k_after) × chunk_tokens × query_volume × input_price_per_token

1.3 Cost Comparison Matrix (100K queries/day, 10M vector corpus)

The following estimates use us-east-1 pricing as of June 2026.

Backend Monthly Vector Store Cost Per-Query Retrieval Cost Notes
Bedrock KB + OpenSearch Serverless Classic $350 minimum + $0.10/GB storage Included in OCU cost OCU floor kills cost at low QPS
Bedrock KB + Aurora pgvector $0.10/ACU-hr (Serverless v2, min 0.5 ACU ≈ $36/mo) ~$0.00 (included in ACU) Best for <5M vectors already on Aurora
Bedrock KB + S3 Vectors $0.06/GB/mo storage + $2.50/MM queries $0.0000025/query New default for KB as of Dec 2025; 90% cheaper than OSS
OpenSearch Serverless NextGen $0/idle (scale-to-zero, May 2026) $0.24/OCU-hr while active New architecture eliminates floor cost
OpenSearch Serverless Classic $350/month minimum (2 OCU always-on) Included in OCU Avoid for new deployments
Pinecone Serverless ~$0/storage (object store backed) ~$0.00000025/query unit Cheapest per-query at low-mid scale; proprietary
pgvector on Aurora PostgreSQL $0.10/ACU-hr, Serverless v2 from $36/mo Included in ACU 9x faster with pgvector 0.8.0 (Aurora June 2026)
pgvector on RDS PostgreSQL $0.15–$0.50/hr depending on instance Included in instance db.r7g.large ≈ $180/mo; practical for ≤5M vectors

2. Bedrock Knowledge Bases Pricing Deep Dive

Bedrock Knowledge Bases (KB) is a managed RAG orchestration service. It connects data sources (S3, Confluence, SharePoint, web crawl) to a vector store and exposes a RetrieveAndGenerate API. The pricing has four layers.

2.1 Embedding Model Costs (Ingestion + Query)

Model Price (input tokens) Dimensions MTEB Score Notes
Amazon Titan Text Embeddings G1 $0.10/1M tokens 1,536 ~56 Legacy; avoid for new deployments
Amazon Titan Text Embeddings V2 $0.02/1M tokens 256/512/1,024 60.37 Default for Bedrock KB; 5× cheaper than Cohere
Cohere Embed v3 (English) $0.10/1M tokens 1,024 64.5 Stronger multilingual, higher recall
Cohere Embed v4 $0.12/1M tokens 1,024 66.3 Highest commercial managed quality

For a 10M-token corpus ingested once, then 100K queries/day at avg 50 tokens/query (5M tokens/day):

  • Titan V2 ingestion: 10M × $0.02/1M = $0.20 one-time
  • Titan V2 query embedding: 5M × $0.02/1M = $0.10/day = $3/month
  • Cohere Embed v4 ingestion: 10M × $0.12/1M = $1.20 one-time
  • Cohere Embed v4 query embedding: 5M × $0.12/1M = $0.60/day = $18/month

Embedding query costs are dominated by query volume, not ingestion. The quality premium for Cohere costs $15/month extra at this volume — easily justified for multilingual or high-precision use cases.

2.2 Vector Store Storage Costs

Bedrock KB supports three vector store backends. Storage costs in us-east-1:

OpenSearch Serverless (Classic, avoid for new KB):

  • Vector storage: $0.10/GB-month (OpenSearch-managed)
  • Minimum compute: 2 OCUs × $0.24/hr × 730hr = $350.40/month regardless of usage
  • At 1,536-dimension float32 vectors: 1M vectors ≈ 6GB → $0.60/mo storage; compute floor dominates

S3 Vectors (default since Dec 2025):

  • Storage: $0.06/GB-month
  • Vector upload (PUT): $0.20/GB (one-time ingest cost)
  • Queries: $2.50/MM queries + data processing charge (tiered: $0.004/TB first 100K vectors, $0.002/TB for 100K–10M)
  • At 1M vectors (768-dim float32 ≈ 3GB): storage $0.18/mo; queries at 100K/day ≈ $0.0025 × 3 = $7.50/month
  • S3 Vectors total (1M vectors, 100K queries/day): ~$7.70/month vs $350+ for OSS Classic

Aurora PostgreSQL with pgvector:

  • Aurora Serverless v2 minimum: 0.5 ACU × $0.12/ACU-hr = $43.80/month
  • Storage: $0.10/GB-month (Aurora I/O-Optimized) or $0.025/GB + $0.20/MM I/O (standard)
  • Best for workloads already on Aurora; pgvector 0.8.0 delivers 9× faster query processing vs 0.7.x

2.3 Sync Job Costs

Bedrock KB sync jobs re-embed changed documents and update the vector index. Costs are embedding API tokens for changed documents only (incremental sync) plus any Data Automation parsing costs:

  • Bedrock Data Automation (PDF/image parsing): $0.010/page
  • For a 100K-page corpus syncing 1% daily: 1,000 pages × $0.01 = $10/day = $300/month — often larger than embedding cost itself

Implication: For document-heavy corpora, parsing cost at sync time can exceed retrieval embedding cost. Pre-parse to plain text at ingest and skip Data Automation where possible.

2.4 RetrieveAndGenerate API — Query Costs

The RetrieveAndGenerate API charges:

  • Underlying LLM input + output tokens at model list price (Claude, Titan, etc.)
  • Embedding model tokens for the query
  • Amazon Rerank 1.0 (optional): $1.00/1,000 reranking queries

Amazon Rerank 1.0 cost at scale: 100K queries/day = 100K reranking calls = $100/day = $3,000/month. At that volume, self-hosted cross-encoder or Cohere Rerank via direct API becomes dramatically cheaper.

2.5 When Does Managed KB Exceed Self-Managed OpenSearch Serverless NextGen?

With OpenSearch Serverless NextGen (launched May 28, 2026) and S3 Vectors as the KB backend, managed Bedrock KB is now cost-competitive at almost all scales. The break-even comparison to watch is Bedrock KB orchestration convenience vs. the operational overhead of running your own retrieval pipeline on LangChain/LlamaIndex with a direct OpenSearch NextGen or Pinecone backend.

Managed KB premium: The RetrieveAndGenerate API charges at full model list price — no Bedrock Marketplace discounts, no provisioned throughput discounts below $100K/month commitment. At >10M queries/month, building a custom retrieval pipeline and calling the LLM directly with Bedrock on-demand pricing or provisioned throughput unlocks 20–40% savings on the LLM portion alone.

3.1 OCU Pricing Architecture

OpenSearch Serverless has two compute planes that bill independently:

Compute Plane Function Classic Minimum NextGen Minimum
Indexing OCUs Document ingestion, index writes 2 OCUs (0.5 primary + 0.5 standby per collection) 0 (scale-to-zero)
Search OCUs Query execution 2 OCUs (0.5 primary + 0.5 replica per collection) 0 (scale-to-zero)

Classic billing: $0.24/OCU-hr. 4-OCU floor (2 indexing + 2 search) = $0.96/hr = $700.80/month at full month. Some sources cite 2-OCU floor ($350/month) if you share indexing and search — this depends on collection count and configuration. The practical floor for a production Knowledge Base collection is $350–$700/month.

NextGen billing (post May 28, 2026): Scale-to-zero when idle. You pay only for active OCU-hours. For intermittent workloads (batch processing, dev/staging, internal tools), this eliminates the idle cost entirely. For 24/7 production workloads with steady QPS, the difference is minimal.

3.2 Vector-Specific Memory Sizing

For vector (k-NN) workloads, the entire HNSW graph must fit in memory, not just the working set. Each OCU provides 6GB of RAM (not 120GB disk). Sizing formula:

RAM_required_GB = vectors × dimensions × bytes_per_float / 1_000_000_000
# float32 = 4 bytes; int8 quantized = 1 byte; binary = 1 bit

# Example: 10M vectors, 1,024 dims, float32
RAM_GB = 10,000,000 × 1,024 × 4 / 1e9 = 40.96 GB → 7 OCUs required (6GB each)

# With scalar quantization (int8):
RAM_GB = 10,000,000 × 1,024 × 1 / 1e9 = 10.24 GB → 2 OCUs sufficient
Corpus Size Dims Encoding RAM Needed Min Search OCUs Monthly Cost (Classic)
1M vectors 1,024 float32 4.1 GB 1 (but 2 min) $350
10M vectors 1,024 float32 41 GB 7 $1,225
100M vectors 1,024 float32 410 GB 69 $12,110
10M vectors 1,024 int8 10.2 GB 2 $350
100M vectors 768 int8 76.8 GB 13 $2,278

Scalar quantization (int8) is the most impactful single cost lever on OpenSearch Serverless. It reduces memory 4×, potentially collapsing 7 OCUs to 2 for a 10M-vector index — saving $875/month at no measurable recall loss for most enterprise corpora.

3.3 HNSW vs IVF — Cost vs Recall Tradeoff

OpenSearch k-NN supports two index types with different cost profiles:

HNSW (Hierarchical Navigable Small World):

  • Highest recall at equal ef_search (default: 512)
  • Higher memory usage: graph overhead is ~1.5× raw vector size
  • Recommended for <50M vectors where recall matters

IVF (Inverted File Index):

  • Lower memory: only active clusters loaded during query
  • Recall drops at high k unless nprobes is tuned
  • Use for >50M vectors where memory is the binding constraint

ef_search tuning: Reducing ef_search from 512 to 64 cuts query latency 3–5× and reduces CPU per query proportionally, but drops recall@10 by ~2–4% depending on data distribution. For enterprise knowledge bases where exact recall is less critical than throughput cost, ef_search=128 is a practical default:

{
  "method": {
    "name": "hnsw",
    "parameters": {
      "ef_construction": 256,
      "m": 16,
      "ef_search": 128
    }
  }
}

3.4 Managed Cluster vs Serverless at Scale

OpenSearch Service (managed cluster) becomes cheaper than Serverless above roughly 50M vectors with 24/7 steady-state QPS. A r7g.2xlarge.search (8 vCPU, 64GB RAM) at $0.528/hr = $386/month per node. A 3-node cluster with 192GB RAM handles ~47M float32 1,024-dim vectors and provides 3× throughput of 8 Serverless OCUs at similar monthly cost.

Rule of thumb: Below 20M vectors or with bursty traffic → Serverless NextGen. Above 50M vectors at sustained QPS → managed cluster with UltraWarm for cold segments. Between 20–50M vectors → model it per workload.

4. Retrieval Optimization for Cost

Every token of retrieved context sent to the LLM costs money. Retrieval optimization is about maximizing relevance density (useful signal per token) while minimizing context length.

4.1 Reducing k — The Simplest Lever

The most direct cost reduction is passing fewer chunks to the LLM. Most production systems default to k=5 or k=10 without justification.

k Value Retrieved Tokens (512-chunk) Added Cost/Query (Claude Sonnet @ $3/1M) Marginal Benefit
k=3 1,536 $0.0046 Baseline for focused corpora
k=5 2,560 $0.0077 Default; $0.0031 premium over k=3
k=10 5,120 $0.0154 $0.0077 premium over k=5; rarely justified

At 100K queries/day, the difference between k=5 and k=3 is $310/day = $9,300/month on Claude Sonnet. Validating that k=3 achieves equivalent answer quality is worth the engineering investment.

4.2 Hybrid Search (BM25 + Semantic)

Semantic-only search misses exact keyword matches that are trivially retrievable by BM25. The consequence is higher k needed to guarantee relevant documents appear, inflating context. Hybrid search (reciprocal rank fusion of BM25 + semantic scores) achieves equivalent or better recall at lower k.

OpenSearch Serverless supports hybrid search natively via the neural_sparse and hybrid query types. Enabling it requires no additional OCUs and typically allows reducing k from 5 to 3 while maintaining recall@3 equivalent to k=5 semantic-only.

Implementation note: Use alpha weighting to blend scores. For technical documentation, alpha=0.3 (30% semantic, 70% BM25) often outperforms pure semantic. For conversational knowledge bases, alpha=0.7 performs better. A/B test with your corpus before setting permanently.

4.3 Reranking — Higher Precision, Lower k

A two-stage retrieval pipeline retrieves k_initial=20–50 candidates cheaply, then reranks with a cross-encoder to surface the top k_final=3–5. The LLM only sees k_final chunks. The cost tradeoff:

Stage 1 cost: Vector search for k=50 (near-zero additional cost over k=5 in most vector DBs) Stage 2 cost (reranker): Cross-encoder over 50 candidates per query

Reranker Option Price Latency Added Notes
Cohere Rerank 3.5 $2.00/1,000 searches 80–150ms p50 Best default; English + 100 languages
Cohere Rerank 4 Pro $0.0025/search = $2.50/1,000 Similar Marginal quality gain over 3.5
Amazon Rerank 1.0 (via Bedrock) $1.00/1,000 searches ~100ms Convenient for KB; expensive at scale
Self-hosted cross-encoder (e.g. ms-marco-MiniLM-L-6-v2 on g4dn.xlarge) $0.526/hr ÷ throughput 20–80ms Break-even at ~200K queries/day

Cohere Rerank 3.5 monthly cost at 100K queries/day: 100,000 × $0.002 = $200/day = $6,000/month

This is only justified if reranking enables k_final=3 instead of k_final=5 (saving $9,300/month on LLM context at Claude Sonnet prices) — which it typically does, netting a $3,300/month saving after reranker cost.

Self-hosted break-even: A g4dn.xlarge at $0.526/hr ($389/month on-demand) can handle ~500 rerank calls/second with MiniLM-L-6. At 100K queries/day (1.16 queries/second), utilization is <1% — self-hosting is not justified. At 300K queries/day, break-even arrives.

4.4 Chunk Size Optimization

Chunk size is a three-way tradeoff between retrieval precision, LLM context size, and index size.

Chunk Size Index Size Impact Retrieval Precision LLM Context per k=3 Best For
128 tokens 4× more chunks than 512 High (specific claims) 384 tokens Factual Q&A, citations
256 tokens 2× more chunks Good 768 tokens Technical docs
512 tokens Baseline Medium 1,536 tokens General knowledge base
1,024 tokens 0.5× chunks Low (diluted) 3,072 tokens Narrative/policy docs

Smaller chunks reduce LLM context (lower cost) and improve retrieval specificity, but increase the chance of missing context that spans chunk boundaries. Parent-child chunking resolves this:

  • Index at 128-token “child” chunks for precision retrieval
  • On match, return the parent 512-token chunk to the LLM for context completeness
  • Net effect: precision of 128-token retrieval, context of 512-token augmentation
  • LLM tokens at k=3: 3 × 512 = 1,536 vs 3 × 1,024 = 3,072 for large-chunk baseline → 50% reduction

4.5 Late Chunking (Jina AI)

Late chunking (Jina Embeddings v2, 2024; widely adopted in 2025–2026) processes the full document through the transformer before splitting, so each chunk’s embedding carries cross-document context. Benefits:

  • Fewer index chunks for equivalent semantic coverage
  • Better recall on queries that reference context spread across paragraph boundaries
  • Supported natively in Jina’s long-context model (8,192 token context window)

The cost implication: fewer chunks at the same semantic quality means smaller index (lower storage cost), lower vector store query cost, and — if paired with parent-child — lower LLM context cost. For document-heavy enterprise knowledge bases (>100K documents), late chunking is worth the pipeline engineering.

5. Embedding Model Cost and Quality Tradeoff

5.1 Pricing and Quality Matrix (June 2026)

Model Provider Price/1M tokens MTEB Score Dimensions Max Tokens Self-Host Option
text-embedding-3-small OpenAI $0.02/1M 62.3 1,536 (reducible) 8,191 No (API only)
text-embedding-3-large OpenAI $0.13/1M 64.6 3,072 (reducible) 8,191 No
Titan Embeddings V2 AWS $0.02/1M 60.37 256/512/1,024 8,192 No (Bedrock)
Cohere Embed v3 (English) Cohere $0.10/1M 64.5 1,024 512 No
Cohere Embed v4 Cohere $0.12/1M 66.3 1,024 128K No
BGE-M3 BAAI (HF) Free (self-host) 65.4 1,024 8,192 Yes; $0.001–$0.019/1M on GPU
Voyage 3.5 Voyage AI $0.06/1M 67.1 1,024 32K No
jina-embeddings-v3 Jina AI $0.02/1M 65.7 1,024 8,192 Yes

5.2 Titan V2 Dimensionality Options

Titan V2 is the only AWS-native model offering reduced dimensionality at no price premium:

  • 256 dims: 75% storage reduction vs 1,024. MTEB drops ~3 points. Use for high-volume, lower-precision workloads.
  • 512 dims: 50% storage reduction. MTEB drops ~1.5 points. Recommended default for cost-sensitive deployments.
  • 1,024 dims: Full quality. Use for precision-critical or multilingual queries.

Storage impact at 10M vectors:

  • float32, 1,024 dims: 40.96 GB
  • float32, 512 dims: 20.48 GB → halves OpenSearch OCU requirement and S3 storage cost
  • int8 quantized, 512 dims: 5.12 GB → reduces to single OCU territory

5.3 Self-Hosted Embeddings Break-Even

Self-hosting embeddings on SageMaker or EC2 makes sense only above a volume threshold where managed API cost exceeds fixed infrastructure cost.

SageMaker real-time endpoint (BGE-M3, ml.g4dn.xlarge):

  • Instance cost: $0.526/hr on-demand = $378/month
  • Throughput: ~60,000 tokens/second sustained = ~216M tokens/hour at full utilization
  • Break-even vs Titan V2 ($0.02/1M): $378/month ÷ $0.02/1M = 18.9B tokens/month needed to break even at 100% utilization
  • At 50% utilization: 9.45B tokens/month break-even
  • At 25% utilization: 4.7B tokens/month break-even

Practical interpretation: For an enterprise RAG pipeline doing 100K queries/day at 50 tokens/query = 5M query tokens/day = 150M tokens/month, self-hosting is not justified on API embedding cost alone. Self-hosting BGE-M3 becomes economically rational above ~1B tokens/month with a dedicated endpoint — roughly 30M queries/day or large nightly batch re-ingestion workflows.

Spot pricing (A100 on EC2 p4d equivalent, ~$1.04/hr spot): Break-even drops to ~720M tokens/month at 50% utilization. Spot is viable for batch ingestion pipelines with interruption tolerance, not for synchronous query embedding.

5.4 Dimensionality Reduction via Matryoshka Embeddings

text-embedding-3-small and text-embedding-3-large support Matryoshka Representation Learning (MRL), allowing the embedding to be truncated to lower dimensions without re-embedding. This is available on OpenAI models via the dimensions parameter.

At 256 dims from text-embedding-3-small, MTEB drops from 62.3 to ~60.5 — roughly equal to Titan V2 at 1,024 dims, but stored at 75% less memory. For OpenSearch Serverless, this can halve OCU costs on large corpora.

6. Caching Strategies for RAG

6.1 Semantic Query Cache Architecture

A semantic cache intercepts queries before they reach the retrieval pipeline. The cache key is the query embedding; the lookup is a vector similarity search against previously cached (embedding, result) pairs. If the new query is within a similarity threshold (cosine distance < 0.05–0.10), the cached result is returned.

Query → Embed → Cache Lookup (vector similarity) → [HIT] → return cached LLM response
                                                   → [MISS] → Retrieve → LLM → store result → return

Cache storage: Redis with RediSearch (ElastiCache) or a small in-process Qdrant instance. For enterprise Knowledge Bases, ElastiCache Serverless with vector search support ($0.125/GB-hr) is the managed path.

6.2 Expected Hit Rates by Workload Type

Workload Type Expected Cache Hit Rate Rationale
Internal HR/policy FAQ 40–60% Small question space, repeated queries
IT helpdesk/support 30–50% High repetition on common issues
Customer-facing product FAQ 20–40% Moderate diversity
Financial/legal document Q&A 10–20% Higher query diversity, lower repetition
Research/exploration queries 5–10% High novelty, low repeat patterns
Mixed enterprise knowledge base 15–25% Weighted average of above

GPTCache production benchmarks (2024): 61.6–68.8% hit rates with 97%+ accuracy on FAQ-heavy workloads; 20–35% on diverse enterprise corpora.

6.3 Cost Impact Formula

# Daily cost without cache
cost_baseline = queries_per_day × cost_per_query

# Daily cost with cache
cost_cached = queries_per_day × (1 - hit_rate) × cost_per_query
            + queries_per_day × hit_rate × cache_cost_per_hit

# Where cache_cost_per_hit ≈ $0.0000002 (Redis lookup, ~10x cheaper than one embedding call)
# Savings = queries_per_day × hit_rate × (cost_per_query - cache_cost_per_hit)

Worked example: 100K queries/day, $0.013 fully-loaded cost per query (embedding + retrieval + LLM), 30% cache hit rate:

Baseline daily cost: 100,000 × $0.013 = $1,300/day
Cached daily cost:   100,000 × 0.70 × $0.013 + 100,000 × 0.30 × $0.0000002
                   = $910 + $0.006 ≈ $910/day
Daily savings: $390/day = $11,700/month

At $1/month for ElastiCache Serverless at this cache size, the ROI is ~11,700:1 monthly. Semantic caching is the highest-ROI optimization for repetitive query workloads.

6.4 Cache Layer Design

Similarity threshold calibration: A threshold that is too loose (< 0.03 cosine distance) misses semantically equivalent queries. Too strict (> 0.15) returns results for semantically different queries, degrading answer quality. Recommended starting point: 0.05–0.08 for factual Q&A, 0.03–0.05 for precise technical queries.

Cache invalidation: Knowledge base updates require cache invalidation on affected document namespaces. Use Redis keyspace tagging by document source or topic cluster. On KB sync, purge cache keys tagged to updated sources.

Cache warming: For predictable query patterns (e.g., monthly report generation, recurring compliance queries), pre-populate the cache by running the expected query set before the scheduled workload arrives.

Implementation options ranked by operational overhead:

  1. LangChain SemanticSimilarityExactMatchCache — simplest; in-process; no dedicated infra
  2. GPTCache — MIT licensed; supports LangChain/LlamaIndex; pluggable vector store and backend
  3. Redis LangCache (RedisVL) — production-grade; supports ElastiCache; 73% cost reduction reported on high-rep workloads
  4. Custom ElastiCache + OpenSearch thin cache — most control; highest ops burden

7. Agentic RAG Cost Patterns

7.1 Cost Multiplication in Agentic Loops

Classic RAG: 1 retrieve → 1 LLM call → response. Token multiplier: 1×.

Agentic RAG: N retrieval steps, each triggered by the agent’s reasoning, with accumulated context. Token multiplier: 3–10× depending on loop depth and context carryforward strategy.

Pattern Retrieval Calls Avg Token Multiplier When Justified
Single-hop RAG 1 1× Factual lookups, FAQ
Multi-hop RAG 2–4 2–4× Cross-document synthesis
Iterative refinement RAG 3–6 4–8× Ambiguous or complex queries
Unguarded agentic loop Unbounded Up to 20× observed Never — must be capped

Context carryforward is the main cost driver in loops. Each loop iteration appends previous retrieval results to the prompt, compounding token cost. A 3-loop agent carrying forward 1,500 tokens per hop spends 4,500 tokens on accumulated context before the final LLM call.

7.2 Adaptive RAG — Route to Cost-Appropriate Architecture

The most cost-effective agentic RAG architecture uses a lightweight classifier to route queries to the cheapest sufficient retrieval strategy:

Query → Complexity Classifier (T5-small or similar, <$0.0001/call) →
  [Simple]  → Single-hop RAG → LLM (1× tokens)
  [Medium]  → Hybrid search k=3 + reranker → LLM (1.5× tokens)
  [Complex] → Agentic multi-hop → LLM (4–8× tokens)

Empirical distribution for enterprise knowledge bases: 70–85% of queries are “simple” (single-hop sufficient), 10–20% medium, 5–10% complex. Routing correctly reduces average cost by 30–50% vs routing everything through agentic.

7.3 Self-RAG Pattern

Self-RAG augments the LLM with a retrieval decision token: the model emits a [RETRIEVE] or [NO_RETRIEVE] token before generating, deciding whether external context is needed. Cost benefit: simple generation queries skip retrieval entirely, eliminating embedding + vector DB + context inflation costs on 20–40% of queries for mixed workloads.

Self-RAG requires fine-tuned models with the retrieval decision token in vocabulary. Llama-3 fine-tunes for Self-RAG are available on HuggingFace. At enterprise scale, the inference cost of a slightly larger fine-tuned model is typically offset by the retrieval savings on skipped queries.

7.4 Speculative RAG

Speculative RAG runs retrieval and generation in parallel on a draft retrieval set, then verifies and corrects. Cost is similar to standard RAG but latency is lower (parallel execution). Not a cost optimization — primarily a latency optimization. Only relevant where RAG latency exceeds SLA.

7.5 Guardrails to Prevent Runaway Loops

Every agentic RAG deployment must implement hard limits:

# Required guardrails for agentic RAG in production
MAX_RETRIEVAL_HOPS = 4          # Hard cap on loop iterations
MAX_CONTEXT_TOKENS = 8000       # Truncate accumulated context before LLM call
MAX_RETRIEVAL_TIME_MS = 5000    # Timeout per retrieval step
DEDUPLICATE_RESULTS = True      # Do not re-retrieve documents already in context
LOOP_DETECTION_WINDOW = 3       # Halt if same document retrieved 3× in same session

Without DEDUPLICATE_RESULTS, agentic loops on small knowledge bases will retrieve the same top-5 documents repeatedly, spending LLM tokens on redundant context. This is the most common cause of 10–20× token overruns relative to expected cost.

Circuit breaker pattern: Implement a per-session token budget counter. When accumulated retrieval + context tokens exceed the budget (e.g., 50% of the model’s context window), force the agent to generate a final answer with available context rather than retrieving further.

7.6 Query Decomposition Overhead

Multi-hop queries are often decomposed by an LLM planner before retrieval. Each decomposition call costs:

  • 1 LLM call at full model rate (planning call, typically 200–500 input + 100–200 output tokens)
  • Then 2–4 retrieval sub-queries at standard retrieval cost

Decomposition overhead: $0.002–$0.005/query at Claude Sonnet pricing for the planning step. Amortize against the benefit of more precise sub-queries enabling k=3 instead of k=10. The planning overhead is justified for complex queries; it is pure waste for queries that could be answered with single-hop retrieval — reinforcing the need for adaptive routing.

8. Cost Optimization Playbook — Priority Order

8.1 Quick Wins (Days, No Architecture Change)

  1. Enable int8 scalar quantization on OpenSearch Serverless → 4× memory reduction → halve OCU count on most corpora → saves $175–$875/month immediately
  2. Reduce k from default (5–10) to minimum viable (3) → 40–60% reduction in LLM context tokens → model the savings using the formula in Section 1.2
  3. Switch Bedrock KB to S3 Vectors backend → eliminate $350–$700/month OpenSearch minimum floor if on Classic
  4. Enable Titan V2 512-dim instead of 1,024-dim → halve storage and OCU requirements at <2% quality loss for most corpora

8.2 Medium Effort (Weeks, Pipeline Changes)

  1. Implement semantic query cache → 15–40% hit rate for most enterprise workloads → $11,700/month savings at 100K queries/day (30% hit rate, worked example above)
  2. Add hybrid search (BM25 + semantic) → enables k reduction with maintained recall → native on OpenSearch Serverless, free
  3. Parent-child chunking → 50% LLM context reduction with equivalent answer quality → requires re-indexing
  4. Add adaptive routing classifier → routes 70–85% of queries to cheaper single-hop path → 30–50% average cost reduction on mixed workloads

8.3 Larger Investment (Months, Architecture Decisions)

  1. Self-hosted cross-encoder reranker → justified at >300K queries/day; g4dn.xlarge at $389/month vs $6,000/month Cohere API
  2. Self-hosted embeddings (BGE-M3 on SageMaker) → justified at >1B tokens/month; otherwise managed API is cheaper
  3. Migrate to OpenSearch managed cluster → justified at >50M vectors with sustained 24/7 QPS; saves vs Serverless above that threshold
  4. Fine-tune domain-specific embedding model → typically 3–5 MTEB points improvement on domain queries; justifiable for >10M queries/month where quality directly drives revenue (e.g., search-as-product)

9. Reference Cost Model — 100K Queries/Day, 10M Vector Corpus

Component Unoptimized Optimized Savings
Vector store (OpenSearch Serverless Classic, float32, 1,024-dim) $2,278/mo — —
Vector store (OpenSearch Serverless NextGen + int8 + 512-dim) — $350/mo $1,928/mo
Embedding queries (Cohere Embed v3, k=5) $18/mo $18/mo (Titan V2) → $3/mo $15/mo
LLM context (k=5, 512-chunk, Claude Sonnet) $23,250/mo $9,300/mo (k=3) $13,950/mo
Reranker (none → Cohere Rerank 3.5) $0 $6,000/mo (enables k=3) Net -$7,950/mo after reranker cost
Semantic cache (30% hit rate on above) $0 -$11,700/mo savings $11,700/mo
Total ~$25,546/mo ~$7,671/mo ~$17,875/mo (70% reduction)

These numbers are directional. Actual costs depend on model choice, query distribution, and cache hit rate. Instrument your pipeline to measure each bucket independently before committing to optimization priorities.

10. Monitoring and Attribution Framework

To make cost optimization decisions with data rather than intuition, every production RAG pipeline should emit the following metrics:

# Per-query telemetry (emit to CloudWatch, Datadog, or equivalent)
rag.query.embedding_tokens          # query embedding input tokens
rag.query.retrieved_chunks          # k value actually used
rag.query.retrieved_tokens          # total context tokens sent to LLM
rag.query.llm_input_tokens          # full LLM input (system + context + query)
rag.query.llm_output_tokens         # LLM output tokens
rag.query.cache_hit                 # boolean; enables hit rate calculation
rag.query.retrieval_hops            # for agentic RAG: number of retrieve-reason cycles
rag.query.total_cost_usd            # attributed cost at list prices

# Aggregate metrics (daily)
rag.daily.embedding_cost_usd
rag.daily.vector_store_cost_usd     # attributed from OCU/query/storage bills
rag.daily.llm_cost_usd
rag.daily.cache_hit_rate
rag.daily.avg_context_tokens

Without per-query cost attribution, FinOps reviews are working from aggregate invoice lines with no actionable levers. The engineering investment to add these metrics pays back in the first optimization sprint.

Research note compiled June 2026. Pricing reflects AWS, Cohere, OpenAI, and Pinecone public list prices in us-east-1 at time of writing. OpenSearch Serverless NextGen (launched May 28, 2026) changes the Classic OCU economics significantly; validate current pricing at AWS OpenSearch pricing before budgeting. Bedrock Knowledge Bases pricing at AWS Bedrock pricing.