Streaming via InvokeModelWithResponseStream or ConverseStream does not change what you pay for model inference: billing is driven by token count alone, not delivery mode. What streaming does change is the infrastructure cost envelope surrounding inference — connection duration, compute tier selection, load balancer configuration, and FinOps attribution complexity. Getting streaming right in production requires understanding four cost planes simultaneously: model cost (token-based, unchanged by streaming), infrastructure cost (Lambda, API Gateway, ALB, NAT Gateway), observability cost (CloudWatch metrics, logging), and operational cost (engineering time spent on timeout and abandonment edge cases). This note maps all four planes and the interactions between them.
1. Streaming vs. Non-Streaming: The Token Cost Parity Rule
The invariant
AWS Bedrock bills for inference based on actual tokens processed and generated, not the delivery mechanism. A Claude Sonnet 4.5 request that generates 500 output tokens costs the same whether those tokens arrive as a single JSON blob via InvokeModel / Converse or as a stream of server-sent events via InvokeModelWithResponseStream / ConverseStream.
As of June 2026, representative on-demand rates for Claude 3.5 Sonnet (us-east-1):
| Token type | Standard rate |
|---|---|
| Input | $6.00 / 1M tokens |
| Output | $30.00 / 1M tokens |
| Cache write | $7.50 / 1M tokens |
| Cache read | $0.60 / 1M tokens |
These rates apply identically to streaming and non-streaming API calls. There is no streaming surcharge on the Bedrock model invoice line.
What actually differs operationally
The token-cost parity rule masks meaningful infrastructure cost differences:
Connection duration. A non-streaming call occupies an upstream connection for the full prefill + decode cycle, but the connection is synchronous — the caller blocks until completion. A streaming call holds an open TCP connection from first byte to last token. For a 500-token response at 40 tokens/second, that is ~12.5 seconds of open connection. At 1,000 concurrent streaming users, that is 12,500 connection-seconds per second of load — a fundamentally different connection-management problem than 1,000 synchronous calls completing in 12 seconds aggregate.
Lambda billing for long generations. Lambda charges in 1ms increments from invocation start to function return. For non-streaming Lambda-backed inference, the function blocks until the model returns, then immediately responds and bills end. For streaming Lambda, the function remains active while streaming chunks to the client — billed the entire duration. A 15-second streaming response billed at 1,769 MB of memory costs $0.0000264792 per GB-second × (15 seconds × 1.769 GB) = $0.000703 per invocation, versus a synchronous call that bills only model processing time without the response-transmission tail.
Lambda response streaming surcharge. AWS Lambda introduced a network transfer charge for response streaming: you are billed for bytes streamed beyond the first 6 MB per invocation. For LLM text responses, this threshold is rarely hit (500 output tokens ≈ 2–3 KB), but streaming large multimodal outputs (image descriptions, code generations) can breach it. Check the AWS Lambda line in Cost Explorer if streaming costs appear elevated.
API Gateway REST streaming (GA November 2025). API Gateway REST APIs now natively support response streaming, eliminating the previous need to use HTTP APIs or WebSocket APIs purely to avoid the 29-second integration timeout. Key cost implication: previously, teams routed LLM traffic through HTTP APIs (which support payload streaming natively) and accepted the 5-minute idle timeout there; the November 2025 REST streaming GA means you can stay on REST and get chunked transfer without switching API types mid-migration.
2. Time-to-First-Token Economics
The TTFT metric (March 2026)
Amazon Bedrock published two production-critical CloudWatch metrics in March 2026: TimeToFirstToken and EstimatedTPMQuotaUsage. Both are emitted at no additional cost for every successfully completed streaming request (ConverseStream and InvokeModelWithResponseStream only — not available for non-streaming APIs).
TimeToFirstToken measures the server-side latency in milliseconds from when Bedrock receives a streaming request to when the service generates the first response token. It is pure server-side signal — it excludes client-to-AWS network transit and AWS-to-client network transit. This distinction matters: a degraded TimeToFirstToken points to model-side prefill latency, not network conditions.
Metric dimensions:
ModelId— always presentServiceTier— Standard, Priority, Flex, or ReservedResolvedServiceTier— the tier that actually served the request (useful when cross-region inference profiles reroute traffic)
Published with 1-minute aggregation, ~1–2 minute appearance lag in CloudWatch.
TTFT and cost: no relationship
TimeToFirstToken has zero bearing on model cost. A 200ms TTFT and a 2,000ms TTFT on identical requests produce identical invoices. TTFT is a user experience metric — it governs perceived responsiveness, not billing.
The engineering trap is spending money to reduce TTFT when the marginal cost of doing so exceeds the revenue impact of perceived latency. Specific cost interventions to reduce TTFT:
| Intervention | Monthly cost delta | TTFT improvement |
|---|---|---|
| Reserved Throughput (1K TPM, 1 month) | +$X (contact AWS) | 15–25% at peak (no throttling) |
| Priority tier premium over Standard | +25–40% on model tokens | Up to 25% better output tokens/second |
| Cross-region inference profile (us-east + us-west) | None (same token price) | Reduces TTFT variance under regional load |
| ECS Fargate always-on container vs Lambda | +~$45–90/month (1 vCPU/2GB always on) | Eliminates Lambda cold-start latency (300–500ms) |
Cross-region inference profiles are the highest-ROI TTFT improvement: same token cost, lower P95/P99 by spreading load across regions. Enable with a system inference profile ARN.
Output tokens-per-second (OTPS): the cost-relevant throughput metric
TTFT tells you how long prefill took. OTPS tells you how fast the model is decoding. OTPS is the metric that matters for cost-per-unit-time in streaming workloads.
The relationship between Bedrock runtime metrics:
InvocationLatency (ms) = TimeToFirstToken (ms) + (OutputTokenCount / OTPS) × 1000
Solving for OTPS:
OTPS = OutputTokenCount / (InvocationLatency − TimeToFirstToken) × 1000
Note: TimeToFirstToken is only published for streaming APIs, so OTPS can only be computed for streaming traffic. This means non-streaming workloads have no server-side decode throughput signal — another reason to prefer streaming APIs for production observability, not just user experience.
CloudWatch metric math expression for OTPS:
# m1 = InvocationLatency (p50), m2 = OutputTokenCount (p50), m3 = TimeToFirstToken (p50)
OTPS = m2 / (m1 - m3) * 1000
A stable OTPS under load confirms the model is decoding at expected throughput. A drop in OTPS with stable TTFT signals model-side decode degradation — the signal to alarm on. An elevated TTFT with stable OTPS signals a prefill bottleneck — usually a long input prompt or cross-region routing latency.
Alarm threshold: set the OTPS static threshold at ~80% of your established baseline. For Claude Sonnet at normal load, OTPS typically runs 40–60 tokens/second. Set alarm at 35 tokens/second to detect meaningful degradation before users notice.
3. Streaming in Production Architectures
Three viable streaming architectures exist on AWS for LLM inference. Each carries a distinct cost profile. The choice is not purely technical — it is financial.
Architecture A: Lambda Function URLs + Response Streaming
Client
│ HTTP/1.1 chunked or HTTP/2 DATA frames
▼
Lambda Function URL (invoke mode: RESPONSE_STREAM)
│ aws_sdk.BedrockRuntime.converse_stream()
▼
Amazon Bedrock (ConverseStream API)
How it works. Lambda response streaming (GA since April 2023, now available in all commercial regions as of April 2026) lets a Lambda function write tokens to the response stream as they arrive from Bedrock, without buffering the full response. The client receives chunks with low TTFB (tens of milliseconds from first Bedrock token).
Cost structure.
- Lambda execution: billed per ms from invocation start to stream close (client-side close, not function return).
- Lambda streaming surcharge: $0 for first 6 MB per invocation; charged beyond that.
- No API Gateway charges — Lambda Function URLs bypass API Gateway entirely.
- No ALB data processing fee.
- Data transfer: standard Lambda → Internet egress rates apply ($0.09/GB for first 10 TB/month).
Timeout considerations. Lambda maximum execution duration is 15 minutes. Streaming a 4,000-token response at 40 tokens/second takes ~100 seconds. Lambda Function URLs accommodate this comfortably. However, if the client disconnects early, the Lambda function must detect the write failure or it will continue billing while generating tokens the client will never receive.
Cold start penalty. First invocation after idle: 300–500ms added to TTFT. For a production streaming endpoint that must meet P99 TTFT SLAs, Lambda cold starts are a significant concern. Mitigations: provisioned concurrency (increases cost by ~$0.000004646 per GB-second of provisioned but unused concurrency), or moving to ECS Fargate for always-warm containers.
Best for: Infrequent or bursty streaming workloads where you can tolerate occasional cold-start TTFT spikes, and where response sizes stay under ~5 MB.
Architecture B: API Gateway HTTP/REST + Lambda Streaming
Client
│ HTTPS
▼
API Gateway (HTTP API or REST API [Nov 2025 GA streaming])
│ Lambda proxy integration
▼
Lambda (response streaming mode)
│
▼
Amazon Bedrock
Cost structure.
- API Gateway HTTP API: $1.00 per 1M requests + $0.0035/million connection minutes (streaming connections held open).
- API Gateway REST API (streaming): same per-request pricing; verify connection-minute billing applies.
- Lambda: same as Architecture A.
- ALB not in path, so no ALB idle timeout issue.
Key limit: API Gateway idle timeout. Regional and private API Gateway endpoints enforce a 5-minute (300-second) idle timeout on streaming connections — the connection closes if no data is sent for 5 minutes. For LLM inference, idle periods within a stream are brief (tokens flow continuously once generation starts), so this is not a practical issue. Edge-optimized endpoints have a much shorter 30-second timeout — do not use edge-optimized endpoints for streaming LLM responses.
29-second integration timeout (legacy issue). Prior to the November 2025 REST API streaming GA, REST API integrations had a hard 29-second limit, which broke any generation taking longer. HTTP APIs had a 29-second integration timeout as well, but payload streaming bypassed this by keeping the connection open between chunks. With the 2025 REST API streaming GA, this limitation is resolved for REST APIs. The 29-second timeout still applies to non-streaming REST API integrations.
Best for: Teams already using API Gateway for auth, rate limiting, or WAF, where adding a separate Lambda Function URL introduces a second endpoint to secure and monitor.
Architecture C: API Gateway WebSocket + Lambda
Client
│ WSS (WebSocket Secure)
▼
API Gateway WebSocket
│ $connect / $disconnect routes
▼
Connection Manager Lambda
│ DynamoDB (connectionId → session mapping)
▼
Inference Lambda
│ Bedrock ConverseStream → PostToConnection API
▼
Client (pushed via connectionId)
Cost structure.
- API Gateway WebSocket: $1.00 per 1M messages + $0.25 per million connection minutes.
- DynamoDB: reads/writes for connection management (low volume, often negligible).
- Lambda: $connect, $disconnect, and inference function invocations.
PostToConnectionAPI calls: $1.00 per 1M messages (each token chunk is one message).
Connection-minute billing reality. A 30-second streaming response on a WebSocket connection costs: 0.5 connection-minutes × $0.25/1M connection-minutes = $0.000000125 — trivial per connection. At 100K concurrent streaming sessions of 30 seconds each: 50,000 connection-minutes × $0.25/1M = $0.0125 per 30-second window. Not a meaningful cost driver.
PostToConnection message cost reality. A 500-token response streamed as 500 individual PostToConnection calls: 500 × $1/1M = $0.0000005. Also trivial. Batching tokens into groups of 5–10 per PostToConnection call reduces API call volume without meaningfully harming perceived streaming quality.
The 29-second integration timeout problem. WebSocket Lambda handlers have a 29-second integration timeout on the $default route. For long generations, the pattern requires the Inference Lambda to be invoked asynchronously (not as a synchronous WebSocket handler) — it runs standalone, streams tokens via PostToConnection, and returns 200 immediately to the WebSocket routing layer. This adds architectural complexity that HTTP streaming avoids.
When to use. Bidirectional applications where the client needs to interrupt generation, inject new instructions mid-stream, or receive multiple concurrent streams on the same connection. For pure unidirectional token delivery, HTTP streaming (Architecture A or B) is simpler and cheaper.
Architecture D: ECS Fargate Persistent Streaming
Client
│ HTTPS
▼
ALB (Application Load Balancer)
│ Target group → ECS service
▼
ECS Fargate container (FastAPI / Express running streaming endpoint)
│ Persistent HTTP/2 connection to Bedrock
▼
Amazon Bedrock
Cost structure.
- ECS Fargate: 1 vCPU + 2 GB RAM = $0.04048/vCPU-hour + $0.004445/GB-hour = ~$47.30/month always-on. Scale to zero is not supported — minimum task count for availability is 1.
- ALB: $0.008/LCU-hour + $0.018/hour (fixed) = ~$13/month base + LCU charges.
- ALB idle timeout: default 60 seconds — the single most common production failure mode for streaming LLM endpoints. Must be increased.
- Data processing: ALB charges $0.008/LCU, where 1 LCU = 1 GB/hour of data processed. For text LLM responses, this is negligible.
The ALB 60-second idle timeout trap. ALB’s default idle connection timeout is 60 seconds. If a generation takes longer — a 1,500-token response at 25 tokens/second takes 60 seconds — the ALB closes the connection exactly at the timeout boundary, producing a 504 to the client and a partially-streamed response. The fix is mandatory:
# AWS CLI: set ALB idle timeout to 300 seconds
aws elbv2 modify-load-balancer-attributes \
--load-balancer-arn arn:aws:elasticloadbalancing:... \
--attributes Key=idle_timeout.timeout_seconds,Value=300
For very long-form generations (reports, code generation agents), set to 600 seconds or higher. The cost impact of this change is zero — ALB idle timeout affects connection behavior, not billing.
When to use. Sustained high-concurrency streaming workloads (>100 concurrent streams continuously) where Lambda cold starts are unacceptable and the $47–60/month always-on container cost is justified by avoided throttling and cold start remediation costs.
Architecture comparison
TTFT Cold Start Monthly Base Cost Max Concurrency Complexity
Lambda URLs 300-500ms ~$0 (pay per use) 1000s (auto) Low
API GW + Lambda 300-500ms ~$0 + $2 API GW 1000s (auto) Medium
WebSocket + Lambda 300-500ms ~$0 + msg cost 10K connections High
ECS Fargate <50ms ~$60/container Configured limit Medium
4. Cost Attribution Challenges with Streaming
The mid-stream budget enforcement problem
Token count for a streaming request is only fully known at stream completion — the usage block in ConverseStream arrives in the final messageStop event. This creates a window during which a request is consuming quota and generating tokens, but your budget enforcement layer has no confirmed cost signal.
The practical consequence: you cannot hard-stop a streaming request mid-flight based on cost without either:
- Estimating cost from the max_tokens parameter upfront (conservative, may block requests that would have been cheap)
- Tracking the stream in real time and calling
StopStreamor closing the connection when a token count threshold is crossed
Approach 2 requires counting token chunks as they arrive. In ConverseStream, each contentBlockDelta event carries a text delta; you can count characters and apply a rough tokens ≈ characters/4 heuristic to estimate running token count. When this running estimate exceeds your budget threshold, close the upstream connection to Bedrock. Note: you will be billed for all tokens generated up to the point of closure, including the current chunk being processed.
Quota reservation vs. billing. Bedrock reserves quota (Tokens Per Minute) based on max_tokens at the start of a request, but bills only for actual tokens generated. The EstimatedTPMQuotaUsage CloudWatch metric reflects actual post-completion consumption, not the upfront reservation. For budget enforcement, this means:
- Your TPM quota fills faster than your dollar spend (reserved but unconsummated tokens consume quota)
- Your dollar spend is lower than the quota-based projection
- Design budget alerts around CUR line items, not TPM quota consumption
LiteLLM streaming cost attribution
LiteLLM buffers streaming chunks via CustomStreamWrapper and fires cost callbacks when the stream is exhausted — specifically in the success_callback after the final chunk is consumed. The callback receives the complete usage object including input_tokens, output_tokens, cache_read_input_tokens, and cache_write_input_tokens.
A known bug (LiteLLM issue #11789, active as of early 2026): when streaming is enabled and prompt caching is active, cache_read_input_tokens may be counted as regular input tokens for cost calculation, inflating the reported cost by up to 90% (cache read rate is 10% of regular input rate). Workaround: use LiteLLM’s on_success callback to extract usage.cache_read_input_tokens directly and compute cost manually:
def cost_from_usage(usage, model="claude-3-5-sonnet-20241022"):
input_cost = (usage.input_tokens - usage.cache_read_input_tokens) * 6.00 / 1e6
cache_read_cost = usage.cache_read_input_tokens * 0.60 / 1e6
cache_write_cost = usage.cache_write_input_tokens * 7.50 / 1e6
output_cost = usage.output_tokens * 30.00 / 1e6
return input_cost + cache_read_cost + cache_write_cost + output_cost
For the Bedrock passthrough routes (/invoke and /converse), LiteLLM PR #12123 added streaming cost calculation support via event stream decoding on passthrough calls. This is the path used when LiteLLM proxies native Bedrock API calls rather than translating to OpenAI format.
CloudWatch OutputTokenCount for streaming: aggregated, not per-request
OutputTokenCount in the AWS/Bedrock namespace is an aggregated CloudWatch metric — it does not provide per-request token counts. For per-request token attribution, you have two paths:
-
Response headers (InvokeModel / InvokeModelWithResponseStream): Anthropic Claude on Bedrock returns
x-amzn-bedrock-input-token-countandx-amzn-bedrock-output-token-countin HTTP response headers. For streaming, these headers are present on the final HTTP response after the stream closes. -
ConverseStream usage event: The
ConverseStreamAPI includes ametadataevent at stream end containing the fullusageobject withinputTokens,outputTokens,cacheReadInputTokens, andcacheWriteInputTokens. This is the authoritative per-request token count for Converse API callers. -
Application Inference Profiles + CUR 2.0: If you route all inference through an Application Inference Profile ARN (tagged with team, project, or cost center identifiers), those tags flow through to AWS Cost and Usage Reports 2.0, enabling per-tag daily spend aggregation. This is not per-request, but it is the finest granularity available in AWS billing data. As of April 2026, IAM principal-based cost allocation is also available in CUR 2.0, letting you attribute spend by the IAM role that made the API call.
Partial response billing: what you owe when a stream is interrupted
If a streaming connection is closed before the model finishes generating — either by client disconnect, ALB timeout, Lambda execution timeout, or explicit cancellation — billing applies to all tokens generated up to the point of interruption. AWS does not issue credits for partial streams.
This has a non-obvious implication: stream abandonment is pure cost with zero value delivered. A user who closes the browser tab 2 seconds into a 15-second generation pays for ~80 tokens (2s × 40 tokens/s) that were generated and discarded.
The practical mitigation is detecting client disconnect on the server side and cancelling the upstream Bedrock request promptly. In Lambda streaming, a write failure to the response stream surface indicates client disconnect. In ECS/Fargate, the framework (FastAPI, Express) surfaces connection close events. In WebSocket, the $disconnect route fires. In all cases, once disconnect is detected, await bedrock_stream.close() or equivalent should be called immediately.
5. Streaming and Prompt Caching Interaction
Cache compatibility with streaming APIs
Prompt caching works with both ConverseStream and InvokeModelWithResponseStream. The cache is model-side, not API-side — the streaming vs. non-streaming distinction is invisible to the caching layer. A cached prefix is read at the same 10%-of-regular-input-rate whether the response is streamed or returned synchronously.
To enable caching on ConverseStream, include cachePoint blocks in the system prompt or message list at positions that should be cached:
response = bedrock_client.converse_stream(
modelId="anthropic.claude-3-5-sonnet-20241022-v2:0",
system=[
{"text": long_system_prompt},
{"cachePoint": {"type": "default"}} # cache everything above this point
],
messages=[{"role": "user", "content": [{"text": user_message}]}]
)
Cache read in streaming: the CUR representation
In CUR 2.0, a streaming request with a cache hit appears as two line items (same pattern as non-streaming):
AmazonBedrock:CacheReadInputTokens— tokens served from cache at 10% rateAmazonBedrock:OutputTokens— generated output at full rate
The cacheReadInputTokens field in the stream’s final metadata event is the authoritative count of cache-served tokens for that request. Monitor this: a high cacheReadInputTokens / inputTokens ratio (>80%) on a chat application indicates the cache is working as intended and your effective input cost is ~10% of the sticker rate.
Cache TTL and streaming context windows
The default cache TTL for Amazon Nova and Claude models on Bedrock is 5 minutes. For streaming workloads with multi-turn conversations, the 5-minute TTL means the system prompt cache is valid across turns in an active session but may expire between sessions. TTL-aware prefetching (warming the cache with a no-op synthetic request before a known session) is an advanced optimization with a concrete cost: one cache write per TTL window per active session.
Cache write premium: $7.50/1M vs $6.00/1M regular input = 25% premium. For a 10,000-token system prompt, one cache write costs $0.075. If that system prompt is reused across 10+ turns per session, the cache read savings per turn = $0.0594 (10,000 tokens × ($6.00 − $0.60) / 1M = $0.054/turn). Break-even on cache write premium at turn 2; profitable from turn 3 onward.
6. Infrastructure Cost of Streaming at Scale
API Gateway connection limits
API Gateway WebSocket APIs have a default limit of 10,000 concurrent connections per API, per region. This is a soft limit — request increases via Service Quotas console. For HTTP streaming (non-WebSocket), the equivalent limit is the concurrent execution limit of the backing Lambda (default 1,000 unreserved concurrency per region, also soft). ECS Fargate streaming endpoints scale based on task count × connections-per-task; no hard API Gateway connection limit applies.
At 10K concurrent streaming sessions averaging 30 seconds each, your steady-state connection count is:
Throughput required = (sessions/hour × avg_duration_seconds) / 3600
Example: 50,000 sessions/hour × 30s = 1,500,000 connection-seconds/hour = 417 concurrent connections
For most production LLM applications, the 10K WebSocket connection limit is not the binding constraint. Lambda concurrency typically throttles first.
ALB idle timeout: the $0 configuration that prevents $X in lost revenue
ALB default idle timeout: 60 seconds. Recommended for LLM streaming workloads: 300–600 seconds.
Cost of not changing it: any generation exceeding 60 seconds produces a 504 error. At P95 output lengths of 1,500 tokens and 25 tokens/second decode rate, P95 generation time is 60 seconds — meaning 5% of all requests fail by default. The configuration change costs nothing and prevents silent streaming failures.
For Nginx-fronted ECS deployments, also set:
proxy_read_timeout 300s;
proxy_send_timeout 300s;
# Disable response buffering for SSE/streaming:
proxy_buffering off;
X-Accel-Buffering: no;
NAT Gateway data transfer for streaming
NAT Gateway charges $0.045/GB of data processed. For LLM text responses, token volume is modest: 1M output tokens ≈ ~4M characters ≈ ~4 MB. At $0.045/GB, 1 billion output tokens ≈ 4 TB ≈ $180 in NAT Gateway data processing.
The hidden cost multiplier: cross-AZ traffic. If your ECS inference container runs in us-east-1a and your NAT Gateway is in us-east-1b, you pay $0.01/GB in each direction for the AZ crossing, plus the $0.045/GB NAT processing fee — effectively $0.065/GB for AWS-internal traffic before any internet egress. Use VPC endpoints for Bedrock API calls to eliminate NAT Gateway entirely:
# Create VPC endpoint for Bedrock runtime (eliminates NAT GW for Bedrock API calls)
aws ec2 create-vpc-endpoint \
--vpc-id vpc-xxx \
--service-name com.amazonaws.us-east-1.bedrock-runtime \
--vpc-endpoint-type Interface \
--subnet-ids subnet-xxx \
--security-group-ids sg-xxx
VPC endpoints cost $0.01/GB processed — 78% cheaper than NAT Gateway for Bedrock API traffic. For streaming workloads making continuous Bedrock API calls, this is a meaningful FinOps win.
LiteLLM proxy: buffering vs. pass-through mode
LiteLLM proxy can operate in two streaming modes:
Pass-through mode (default for streaming). Chunks from Bedrock are forwarded to the client as they arrive, with minimal buffering. Cost: low proxy compute overhead (~5–10ms additional latency per chunk), no memory accumulation.
Buffered mode. LiteLLM accumulates the full response before forwarding. Used when post-processing (content filtering, cost attribution) requires the complete response. Cost: higher proxy memory usage (full response in RAM), longer TTFB for clients, additional Lambda memory charges if proxy runs on Lambda.
For streaming workloads, configure LiteLLM to pass through streaming directly and use the success_callback for cost attribution after stream completion — do not buffer streaming responses to extract token counts mid-stream.
7. Monitoring Streaming Workloads
Key metrics dashboard
A production streaming endpoint requires four metric families monitored in CloudWatch:
1. Latency metrics (user experience)
| Metric | Namespace | Statistic | Alert threshold |
|---|---|---|---|
TimeToFirstToken |
AWS/Bedrock | P50, P95, P99 | P99 > 3× P50 baseline |
InvocationLatency |
AWS/Bedrock | P50, P95, P99 | P95 > 2× P50 |
ALB TargetResponseTime |
AWS/ApplicationELB | P95 | P95 > generation_time_budget |
2. Throughput metrics (capacity health)
| Metric | Source | Formula | Alert threshold |
|---|---|---|---|
OTPS (output tokens/second) |
CloudWatch metric math | OutputTokenCount / (InvocationLatency - TimeToFirstToken) * 1000 |
< 80% of baseline |
EstimatedTPMQuotaUsage |
AWS/Bedrock | Average | > 70% of quota limit |
3. Error and abandonment metrics (waste signal)
| Metric | Source | What it measures |
|---|---|---|
4XXError / 5XXError |
API Gateway | Client errors, throttles (429), timeouts (504) |
ALB HTTP_Code_ELB_504_Count |
AWS/ApplicationELB | Connections killed by ALB timeout |
Lambda Errors + Duration P99 |
AWS/Lambda | Functions hitting timeout before stream completes |
| Stream abandonment rate (custom) | Application metric | Client disconnects before stream end / total streams |
Stream abandonment rate requires application-level instrumentation — publish it as a custom CloudWatch metric:
cloudwatch.put_metric_data(
Namespace="YourApp/Streaming",
MetricData=[{
"MetricName": "StreamAbandonment",
"Value": 1 if client_disconnected_early else 0,
"Unit": "Count",
"Dimensions": [{"Name": "ModelId", "Value": model_id}]
}]
)
A stream abandonment rate above 10% signals either UX problems (responses are too slow to initiate) or TTFT problems (users give up before first token). It also directly quantifies wasted spend: (abandoned_tokens / total_tokens) × model_cost = tokens billed but never seen.
4. Cost metrics (FinOps)
| Metric | Source | Cadence |
|---|---|---|
InputTokens, OutputTokens |
AWS/Bedrock CloudWatch | Per minute (aggregated) |
CacheReadInputTokens, CacheWriteInputTokens |
AWS/Bedrock CloudWatch | Per minute (aggregated) |
| Per-tag spend | CUR 2.0 + Application Inference Profiles | Daily |
| Bedrock line item in Cost Explorer | AWS Billing | Daily |
For real-time per-request cost tracking, the only path is application-level instrumentation — extract token counts from the ConverseStream metadata event and emit them as custom CloudWatch metrics or write to a cost tracking database.
P50/P95/P99 TTFT as SLA metric
Define streaming SLAs in terms of TTFT percentiles, not mean:
Tier-1 (interactive chat): P50 TTFT < 800ms, P95 < 2,000ms
Tier-2 (async workbench): P50 TTFT < 2,000ms, P95 < 5,000ms
Tier-3 (batch with streaming): No TTFT SLA; total InvocationLatency P95 only
The gap between P50 and P95 TTFT is a load indicator: large gaps indicate queue buildup at peak. Small gaps indicate consistent headroom. If P95/P50 TTFT ratio exceeds 3×, investigate either Reserved Throughput capacity or Priority tier.
8. Cost Optimization for Streaming-Heavy Workloads
Reserved Service Tier for predictable streaming load
Reserved Throughput (also called Provisioned Throughput) purchases a committed capacity of N tokens-per-minute for 1 or 3 months. On-demand pricing applies per token; Reserved pricing is a fixed monthly fee for the provisioned TPM block, with per-token rates that are typically lower for sustained workloads.
Benefits for streaming specifically:
- No 429 ThrottlingException during generation (quota guaranteed up to reserved limit)
- Lower TTFT variance — no queueing behind other on-demand tenants
ServiceTierandResolvedServiceTierCloudWatch dimensions let you confirm traffic is hitting reserved capacity
Reserved is not appropriate for streaming workloads with high variance or low utilization. If your peak-to-trough ratio exceeds 3:1, on-demand with Priority tier is usually cheaper.
Flex tier and streaming
Flex tier supports all Bedrock inference APIs including ConverseStream and InvokeModelWithResponseStream. The tradeoff: Flex requests are processed after Standard requests during peak periods, which manifests as elevated TimeToFirstToken and higher queue-induced TTFT variance. For streaming workloads where TTFT directly affects user experience, Flex is appropriate only for:
- Background agent pipelines where the user is not waiting for first token
- Batch-with-streaming patterns where generation is pre-triggered before the user requests it
- Development and testing traffic
Never route interactive user-facing streaming requests to Flex tier.
Pre-generation for known content
For content that can be predicted before user request — report generation triggered by a scheduled event, daily briefings, known conversational branches — generate and cache the full response before the user initiates streaming. When the user requests the stream, serve from cache character-by-character at synthetic streaming speed.
Cost impact: model inference cost is unchanged (still billed per token at generation time), but TTFT perceived by the user drops to near-zero and stream abandonment drops to near-zero (users see content immediately). Infrastructure cost: a small cache (Redis / DynamoDB) to store pre-generated text.
This pattern is particularly valuable for AI applications where the initial system message or context is expensive to process. Pre-generate the first response turn, stream it from cache, then switch to live streaming for subsequent turns.
max_tokens as streaming cost guardrail
Setting max_tokens explicitly on every streaming request is the primary streaming cost guardrail. Without it, models default to their maximum context window output length, which can be 4,096–8,192 tokens for a request that should produce 200 tokens. Each unnecessary output token costs 5× the input token rate.
response = bedrock_client.converse_stream(
modelId="anthropic.claude-3-5-sonnet-20241022-v2:0",
inferenceConfig={
"maxTokens": 512, # hard cap; adjust per use case
"temperature": 0.7,
"topP": 0.9
},
messages=[...]
)
max_tokens also affects quota consumption: Bedrock’s quota system reserves bandwidth equal to max_tokens × burndown_multiplier at request start. Setting max_tokens accurately reduces quota pressure, which reduces 429 frequency at high concurrency. A 4,096 max_tokens on a request that generates 200 tokens consumes 20× the intended quota slot. For Claude models with a 5× output token burndown multiplier:
Effective TPM consumed = InputTokens + CacheWriteTokens + (OutputTokens × 5)
With max_tokens=4096, quota reservation = InputTokens + 4096 × 5 = InputTokens + 20,480
With max_tokens=512, quota reservation = InputTokens + 512 × 5 = InputTokens + 2,560
The quota efficiency gain from right-sizing max_tokens is 8× in this example — directly translating to 8× more concurrent streaming requests before throttling occurs.
Cross-region inference profiles: free TTFT improvement
Cross-region inference profiles route requests across us-east-1, us-west-2, and other participating regions, selecting the lowest-latency available endpoint at request time. The token price is identical to single-region on-demand. The benefit is absorbed in two ways:
- Lower P95/P99 TTFT (regional capacity imbalances are smoothed)
- Reduced 429 rate during regional congestion events
Enable by using a system inference profile ARN (e.g., us.anthropic.claude-3-5-sonnet-20241022-v2:0) instead of a region-specific model ARN. The ResolvedServiceTier and ModelId dimensions in CloudWatch will show which region served each request.
Appendix A: Cost Calculation Reference
Streaming request cost decomposition (Claude 3.5 Sonnet, Standard tier)
Scenario: 1,000 token system prompt (cached, already written), 200 token user message, 500 token response.
Without caching:
Input: (1000 + 200) tokens × $6.00/1M = $0.0072
Output: 500 tokens × $30.00/1M = $0.015
Total: = $0.0222 per request
With cache hit on system prompt:
Cache read: 1000 tokens × $0.60/1M = $0.0006
Regular input: 200 tokens × $6.00/1M = $0.0012
Output: 500 tokens × $30.00/1M = $0.015
Total: = $0.0168 per request
Savings vs uncached: = 24.3%
These costs apply identically whether the response is streamed or returned synchronously.
Stream abandonment cost (example)
Scenario: 500 output tokens requested, client disconnects after 100 tokens generated.
Tokens billed: 100 output tokens × $30.00/1M = $0.003
Value delivered: 100/500 = 20% of intended response
Cost per useful token: $0.003/100 = $0.00003 (same as full response)
Wasted spend: 400 tokens × $30.00/1M = $0.012 that produced output user never saw
At scale: if 15% of streaming requests are abandoned at 20% completion, and average output is 500 tokens:
Expected output tokens per request: 500
Avg tokens generated before abandon: 0.85 × 500 + 0.15 × (0.20 × 500) = 440 tokens
Effective cost vs theoretical minimum: 440/500 = 88% efficiency
Wasted cost: 12% of output token spend
NAT Gateway vs VPC endpoint for Bedrock: annual savings
Scenario: 1 billion output tokens/year ≈ 4 TB of Bedrock API response data processed.
NAT Gateway: 4 TB × $0.045/GB = $180/year (data processing only)
VPC Endpoint: 4 TB × $0.01/GB = $40/year
Annual saving: $140/year per 1B output tokens
For most production LLM applications (≤100M tokens/month), NAT Gateway optimization saves <$15/month — low priority unless combined with other AWS service traffic.
Appendix B: Monitoring Code Snippets
OTPS CloudWatch alarm (Boto3)
import boto3
cw = boto3.client("cloudwatch", region_name="us-east-1")
MODEL_ID = "us.anthropic.claude-3-5-sonnet-20241022-v1:0"
cw.put_metric_alarm(
AlarmName="Bedrock-OTPS-Degraded",
AlarmDescription="Model decode throughput dropped below 80% of baseline",
Metrics=[
{
"Id": "m1",
"MetricStat": {
"Metric": {
"Namespace": "AWS/Bedrock",
"MetricName": "InvocationLatency",
"Dimensions": [{"Name": "ModelId", "Value": MODEL_ID}],
},
"Period": 300,
"Stat": "p50",
},
"ReturnData": False,
},
{
"Id": "m2",
"MetricStat": {
"Metric": {
"Namespace": "AWS/Bedrock",
"MetricName": "OutputTokenCount",
"Dimensions": [{"Name": "ModelId", "Value": MODEL_ID}],
},
"Period": 300,
"Stat": "p50",
},
"ReturnData": False,
},
{
"Id": "m3",
"MetricStat": {
"Metric": {
"Namespace": "AWS/Bedrock",
"MetricName": "TimeToFirstToken",
"Dimensions": [{"Name": "ModelId", "Value": MODEL_ID}],
},
"Period": 300,
"Stat": "p50",
},
"ReturnData": False,
},
{
"Id": "otps",
"Expression": "m2 / (m1 - m3) * 1000",
"Label": "OTPS (tokens/s)",
"ReturnData": True,
},
],
ComparisonOperator="LessThanThreshold",
Threshold=35, # 80% of 44 tokens/s baseline
EvaluationPeriods=5,
DatapointsToAlarm=3,
TreatMissingData="ignore",
)
Stream abandonment custom metric (Python)
import boto3
import time
cw = boto3.client("cloudwatch")
async def stream_with_abandonment_tracking(bedrock_client, request_params):
tokens_generated = 0
completed = False
start_time = time.time()
try:
response = bedrock_client.converse_stream(**request_params)
for event in response["stream"]:
if "contentBlockDelta" in event:
tokens_generated += 1 # approximate; use metadata event for exact count
yield event["contentBlockDelta"]["delta"]["text"]
elif "messageStop" in event:
completed = True
except (BrokenPipeError, ConnectionResetError):
pass # client disconnected
finally:
cw.put_metric_data(
Namespace="YourApp/Streaming",
MetricData=[
{
"MetricName": "StreamCompleted",
"Value": 1 if completed else 0,
"Unit": "Count",
},
{
"MetricName": "StreamAbandoned",
"Value": 0 if completed else 1,
"Unit": "Count",
},
{
"MetricName": "TokensGeneratedBeforeAbandon",
"Value": tokens_generated if not completed else 0,
"Unit": "Count",
},
]
)
Key Findings Summary
-
Token cost parity is absolute. Streaming and non-streaming Bedrock API calls are billed identically. The streaming surcharge is zero at the model layer.
-
TTFT does not affect cost. It affects user experience and may justify infrastructure spend (Reserved tier, Priority tier, cross-region profiles), but optimizing TTFT is a revenue protection decision, not a cost reduction decision.
-
The 60-second ALB idle timeout is the most common $0-fix production failure. Set it to 300s or higher for any LLM streaming endpoint behind an ALB. The default is silently wrong for long-form generation.
-
Stream abandonment is measurable wasted spend. Instrument it explicitly. A 10%+ abandonment rate indicates a TTFT or UX problem and directly quantifies money spent on tokens no user received.
-
Prompt caching works with streaming.
CacheReadInputTokensin the ConverseStream metadata event confirms cache hits. The LiteLLM streaming + caching cost calculation bug (issue #11789) requires a manual cost computation workaround. -
Per-request cost visibility requires application-level instrumentation. CloudWatch
OutputTokenCountis aggregated, not per-request. Use ConverseStream’s finalmetadataevent for exact per-request token counts. -
max_tokensright-sizing is the highest-leverage streaming cost optimization. Oversizedmax_tokenswastes quota 5× faster than the corresponding token cost, causing premature 429 throttling that forces Reserved Throughput purchases that could be avoided. -
VPC endpoints for Bedrock are the correct NAT Gateway avoidance play. At $0.01/GB vs $0.045/GB, VPC endpoints save 78% on Bedrock API data processing costs and eliminate cross-AZ transfer charges for ECS/Fargate architectures.
Sources: AWS Bedrock CloudWatch metrics documentation (March 2026), AWS Lambda response streaming GA announcement (April 2026), API Gateway REST streaming GA (November 2025), LiteLLM GitHub issues #11789 and PR #12123, AWS Bedrock service tiers documentation, AWS pricing pages for Bedrock, Lambda, API Gateway, and NAT Gateway.