← Agent Frameworks 🕐 19 min read

Agent Frameworks

Bedrock Streaming Cost Patterns — InvokeModelWithResponseStream and ConverseStream (2026)

Executive Summary · March 2026

Streaming in AWS Bedrock does not change what you are billed — you pay the same per-token price whether you use `InvokeModelWithResponseStream`, `ConverseStream`, or their synchronous counterparts.

Streaming in AWS Bedrock does not change what you are billed — you pay the same per-token price whether you use InvokeModelWithResponseStream, ConverseStream, or their synchronous counterparts. Streaming is purely a delivery mechanism that changes how tokens arrive at the client, not how AWS accounts for them. The cost implications of streaming are therefore indirect: streaming affects throughput quota consumption through the burndown mechanism, changes how Provisioned Throughput capacity is held over time, determines which CloudWatch metrics are emitted, and creates partial-billing exposure when clients disconnect before generation completes. Engineering teams optimizing Bedrock spend must understand these second-order effects, not a streaming surcharge that does not exist.

1. Streaming vs Non-Streaming Billing: The Equivalence Principle

AWS does not charge a premium or apply a discount for streaming delivery. Both ConverseStream and InvokeModelWithResponseStream bill at identical per-token rates as their synchronous counterparts (Converse and InvokeModel). The four token types and their CUR usage type patterns are:

Token type	CUR usage type pattern	Relative cost
Input tokens	`-input-tokens` or `-mantle-input-tokens-*`	Base rate
Output tokens	`-output-tokens` or `-mantle-output-tokens-*`	5x input for Claude 3.7+
Cache read tokens	`*-cache-read-input-token-count`	~10% of input rate
Cache write tokens	`*-cache-write-input-token-count`	125–200% of input rate

Source: Understanding your Amazon Bedrock Cost and Usage Report data

The CUR line_item_usage_type field does not encode whether a request was streaming or synchronous. A ConverseStream call generating 500 output tokens produces identical line items to a Converse call generating the same 500 tokens. There is no streaming flag in billing data.

The indirect cost implication: Streaming connections hold an open HTTP connection for the duration of generation — typically 2–30 seconds for a typical LLM response. This means simultaneous streaming connections consume more concurrent connection slots against API Gateway or VPC endpoint limits than the same number of completed synchronous calls would. For workloads with hundreds of simultaneous users, connection concurrency limits (not token costs) become the operational constraint.

2. TimeToFirstToken: What It Is, What Drives It, Cost Implications

The Metric

TimeToFirstToken (TTFT) was added to the AWS/Bedrock CloudWatch namespace in March 2026. It measures, in milliseconds, the latency from when Bedrock receives a streaming request to when the service generates the first response token. TTFT is emitted only for streaming APIs — ConverseStream and InvokeModelWithResponseStream. Synchronous Converse and InvokeModel calls do not populate this metric.

Source: Amazon Bedrock now supports observability of First Token Latency and Quota Consumption

TTFT is automatically emitted at no additional cost for every successful streaming inference request. No API changes, SDK updates, or opt-in configuration are required.

The Prefill-Decode Model

An inference request passes through two sequential compute stages:

Prefill — The model processes the entire input prompt in a single forward pass and produces the first output token. Duration scales primarily with input token count. This is the main driver of TimeToFirstToken.
Decode — The model generates each subsequent output token sequentially, one token per forward pass. Total decode time scales with output token count. Per-token decode time is relatively stable for a given model under consistent load.

These stages produce the following relationship between CloudWatch metrics:

InvocationLatency (ms) = TimeToFirstToken (ms) + (OutputTokenCount / OTPS) * 1000

where OTPS = output tokens per second. Solving for OTPS:

OTPS = OutputTokenCount / (InvocationLatency - TimeToFirstToken) * 1000

Source: Diagnose InvocationLatency increases using output tokens per second (OTPS)

How TTFT Differs from InvocationLatency

InvocationLatency is the wall-clock time of the entire request — from receipt to last token generated. A rising InvocationLatency alone cannot tell you whether:

The model is generating tokens more slowly (service-side throughput degradation), or
The model is generating more tokens per request (workload change — longer prompts, longer responses)

TimeToFirstToken isolates the prefill phase. If InvocationLatency rises but TimeToFirstToken is stable and OTPS drops, the slowdown is in the decode phase (throughput degradation). If TimeToFirstToken rises proportionally to input token count, the slowdown is workload-driven (prompts are getting longer).

Cost Implications of TTFT

TTFT has no direct billing effect — you are not billed per millisecond of TTFT. The cost implications are operational:

Interactive workloads (chatbots, copilots): High TTFT degrades perceived quality even if total generation time is acceptable. Users abandon sessions that feel slow before the first word appears. Abandoned sessions may still incur partial billing (see Section 7).
Batch workloads: TTFT is largely irrelevant. What matters is InvocationLatency (total time) and throughput (requests per minute against quota).
Quota utilization: At request start, Bedrock deducts InputTokenCount + max_tokens from your TPM/TPD quota. Long prefill phases (high TTFT) mean quota is held against a pending large deduction before any tokens are returned, which can throttle concurrent request admission under high load.

Prompt Caching and TTFT

Prompt caching on Bedrock dramatically reduces TTFT by skipping prefill computation for cached prefix tokens. On cache hit, the model reads from cache and skips forward-pass computation for those tokens.

Observed TTFT reductions with cache hits:

Up to 85% TTFT reduction vs uncached (AWS documentation)
Nova Micro: 407ms baseline TTFT → 334ms on cache hit in benchmarks
TTL options: 5-minute sliding window (1.25x cache write rate) or 1-hour (2.0x cache write rate)

Source: Amazon Bedrock Prompt Caching with the Amazon Nova Model

Important: Caches do not replicate across AWS regions. Cross-region inference that routes a request to a different region than where the cache was written will experience a cache miss, with full prefill latency. This creates TTFT variance in cross-region inference profiles that appears as inconsistent UX rather than a billing issue but masks cache hit rate degradation.

3. Streaming and Provisioned Throughput: Capacity Consumption Mechanics

How PT Is Priced and What It Buys

Provisioned Throughput (PT) is billed hourly at a fixed rate regardless of actual utilization. You purchase Model Units (MUs), where each MU specifies:

A fixed number of input tokens it can process across all requests within a span of one minute
A fixed number of output tokens it can generate across all requests within a span of one minute

Source: Increase model invocation capacity with Provisioned Throughput in Amazon Bedrock

The Streaming Capacity Hold Problem

With synchronous (non-streaming) inference, a request occupies PT capacity from submission until the response is fully generated and returned. The request completes atomically from the PT capacity manager’s perspective.

With streaming, a request occupies PT capacity for the full duration of generation, which for a long output at typical decode rates can be 10–30 seconds. During this time, the connection is open, the model is actively generating, and PT capacity is consumed. If you have 1 MU capable of handling 1,000 output tokens per minute and a streaming request generates 500 tokens over 15 seconds, that MU is occupied for the full 15 seconds.

The practical implication: Streaming requests serialize poorly against a single MU. If interactive users submit long-generation requests (reports, summaries, code) via streaming, they hold MU capacity longer than fast requests would. For PT workloads with variable output length, the effective concurrency under streaming is lower than the MU specification suggests, because slow long-response requests block fast short-response requests from using that capacity.

Mitigation: Use max_tokens to cap output length on PT requests. Setting max_tokens to a value close to expected average output (rather than the model maximum) both reduces average capacity hold time and improves quota utilization for on-demand requests (see Section 2 on burndown).

max_tokens and TPM Quota Under Streaming

On on-demand inference, quota is deducted at request start as InputTokenCount + max_tokens. If a streaming request has max_tokens=32000 but generates only 400 tokens, the initial deduction is large (potentially throttling other concurrent requests) and is only corrected at request completion. For streaming workloads with high concurrency, this transient over-deduction can cause spurious throttling.

Recommended practice: Set max_tokens to approximately 125% of your p90 observed output token count. This minimizes the gap between initial deduction and final adjusted deduction while providing headroom for outlier responses.

4. ConverseStream vs InvokeModelWithResponseStream: CUR Appearance and Tool Use

API Design Differences

Dimension	ConverseStream	InvokeModelWithResponseStream
Payload format	Unified across all models	Model-specific JSON schema
Payload size limit	~3.75 MB	~5 MB
Tool use support	Native `toolConfig` parameter	Model-specific implementation
Multi-turn context	Native `messages` array	Manual context assembly
IAM permission required	`bedrock:InvokeModelWithResponseStream`	`bedrock:InvokeModelWithResponseStream`
CUR line item distinction	None — same usage types	None — same usage types

Source: Bedrock Converse API vs InvokeModel

Critical finding: The CUR does not distinguish between ConverseStream and InvokeModelWithResponseStream. Both appear as the same *-input-tokens and *-output-tokens line items. If you need to attribute cost to a specific API operation, you must join CUR data to model invocation logs at the requestId grain, not read it from CUR directly.

CUR does not contain per-request line items. Both CUR formats aggregate cost by usage type, operation, and pricing resource over an hour or a day; per-requestId attribution requires model invocation logs.

Tool Use Cost with Streaming

When using ConverseStream with tool use, the streaming response delivers tool invocation blocks as a sequence of events rather than a single complete object:

ContentBlockStart — contains tool_use_id and tool name
ContentBlockDelta — delivers toolInput chunks as partial JSON
ContentBlockStop — signals end of tool input block
MessageStop — includes stopReason: "tool_use"
metadata event — includes final usage with inputTokens, outputTokens, totalTokens

Token counting for tool use in streaming: tool definitions passed in toolConfig count as input tokens. Tool results returned in subsequent turns count as input tokens in those turns. The model’s tool invocation request counts as output tokens. The billing for a tool-augmented streaming response is therefore spread across multiple turns, with each turn billed separately at the model’s standard token rates.

Source: AWS Bedrock Converse API with Rust: Tool Use + Streaming

Token usage in streaming metadata events from ConverseStream:

{
  "metadata": {
    "usage": {
      "inputTokens": 450,
      "outputTokens": 127,
      "totalTokens": 577,
      "cacheReadInputTokens": 380,
      "cacheWriteInputTokens": 0
    },
    "metrics": {
      "latencyMs": 3240
    }
  }
}

The usage block is delivered in the final metadata event, not incrementally. Streaming callers cannot access token counts until the stream closes.

5. Streaming and Prompt Caching: Cache Hit Rate Monitoring

How Cache Hits Affect Streaming Behavior

Prompt caching reduces TTFT for streaming requests by skipping prefill computation for cached prefix tokens. When a cache hit occurs, the model can skip the forward pass for cached tokens and jump directly to generating output tokens. This is visible in streaming as a reduced delay before the first ContentBlockDelta event arrives.

Cache hits do not affect token pricing directly — you pay cache-read-input-token-count at the reduced rate (~10% of standard input rate) rather than input-tokens at the full rate. In CUR, these appear as separate line items.

Cache Miss Rate Monitoring for Streaming Workloads

CloudWatch metrics for cache monitoring in the AWS/Bedrock namespace:

Metric	Description
`CacheReadInputTokens`	Tokens read from cache per request; 0 if no cache hit
`CacheWriteInputTokens`	Tokens written to cache per request; 0 if not a cache write
`InputTokenCount`	Non-cached input tokens processed by the model

Cache hit rate is not a published metric. Compute it via metric math:

CacheHitRate = CacheReadInputTokens / (InputTokenCount + CacheReadInputTokens)

A meaningful cache hit rate requires consistent prompt prefix structure across requests. Common failure modes that drive cache miss rates up in streaming workloads:

System prompt variation — any change to the system prompt at the start of a cached block invalidates the cache entry. Dynamic timestamps, session IDs, or per-user personalization in system prompts destroy cache locality.
Cross-region routing — cache entries are region-local. Cross-region inference profiles that route requests across regions will produce cache misses whenever requests land in a cold region.
TTL expiration — the 5-minute sliding-window TTL resets on each hit, but low-frequency users (one request per 6+ minutes) never benefit from caching. For agent loops making rapid sequential calls, the 5-minute TTL is usually sufficient; for async user workflows, the 1-hour TTL option is more appropriate.
Streaming and cache write rate — cache writes cost 1.25x input rate for 5-minute TTL and 2.0x for 1-hour TTL. For high-frequency streaming workloads with stable system prompts, the write cost is amortized quickly. For low-frequency workloads, cache writes may exceed the read savings; instrument both CacheWriteInputTokens and CacheReadInputTokens and compute the net cost delta before committing to a TTL tier.

6. Connection Cost: VPC Endpoints and Long-Lived Streaming Connections

VPC Interface Endpoint Pricing

AWS Bedrock is accessible via VPC Interface Endpoints (AWS PrivateLink). Pricing applies at two levels:

Hourly ENI cost: Each interface endpoint creates ENIs in each availability zone. A typical Bedrock workload requiring Bedrock Runtime + Bedrock Agent Runtime endpoints across three AZs creates approximately 6 ENIs, costing roughly $0.01/hr per ENI — approximately $43–$65/month for the endpoint infrastructure alone.
Data processing cost: $0.01 per GB of data processed through the endpoint. For streaming, every token delivered over the endpoint consumes data volume. A 1,000-token response at ~4 bytes per token is ~4 KB. At millions of responses per month, endpoint data processing charges become material.

Source: VPC-connected Bedrock AgentCore Runtime-hosted agents: beware of NAT Gateway costs!

Streaming vs NAT Gateway

Without VPC endpoints, Bedrock traffic from a VPC must egress through NAT Gateway or Internet Gateway. NAT Gateway charges $0.045 per GB of data processed. For streaming workloads delivering large outputs, this cost exceeds VPC endpoint data processing costs by roughly 4.5x. The break-even point for VPC endpoints vs NAT Gateway depends on traffic volume and endpoint count, but for any workload generating more than ~30 GB/month of streaming output, VPC endpoints reduce total network cost.

Long-Lived Connections and API Gateway

If you proxy Bedrock through API Gateway (common for adding auth layers), note that API Gateway HTTP APIs support streaming responses, but REST APIs do not. API Gateway HTTP APIs charge per connection minute for WebSocket routes; REST API integrations require a different pattern (Lambda response streaming). For Bedrock streaming proxied through API Gateway REST APIs, the pattern of holding a synchronous integration open for 10–30 seconds increases API Gateway timeout risks and connection costs. Lambda response streaming + HTTP API is the recommended architecture for Bedrock streaming behind API Gateway.

7. Streaming Abort Handling: Partial Response Billing

What Happens When a Client Disconnects Mid-Stream

AWS Bedrock bills for tokens that the model generates, not tokens the client receives. When a client disconnects during a streaming response, the model continues generating until either the generation completes naturally or the server-side detects the dropped connection. The billing behavior:

Tokens generated before the disconnect: Billed at standard output rates.
Tokens generated after the disconnect but before server detects the drop: Also billed. The model does not stop generating the moment the TCP connection drops.
Tokens that were never generated: Not billed.

This is confirmed by the behavior documented for guardrail blocks: mid-stream refusals (generation blocked after partial output) are billed for tokens generated before the block fires.

Source: AWS Bedrock unexpected charges analysis

Practical Implications

For interactive streaming applications where users frequently abandon long-running responses (common in agent-loop UX where users interrupt generation):

Implement client-side abort signals and propagate them to the Bedrock SDK call. Use the AbortController pattern in JavaScript/TypeScript or cancellation tokens in Python to terminate the streaming iterator immediately on user cancel.
Set max_tokens conservatively for interactive sessions. This caps the maximum exposure on an abandoned stream.
Monitor abandoned streams via the gap between OutputTokenCount (CloudWatch) and tokens your application actually consumed (application-side counter). A large divergence indicates users are abandoning streams — wasted spend.

8. LiteLLM Streaming Configuration for Bedrock

Basic Streaming Setup

import litellm

response = litellm.completion(
    model="bedrock/anthropic.claude-sonnet-4-5-20250929-v1:0",
    messages=[{"role": "user", "content": "Summarize the following document..."}],
    stream=True,
    max_tokens=1500,
)

for chunk in response:
    # chunk.choices[0].delta.content contains streamed text
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Streaming Cost Attribution via Callbacks

LiteLLM accumulates token counts across streaming chunks and fires cost callbacks on stream completion, not on each chunk. The response_cost key is available in the kwargs dict of a success_callback:

import litellm

def track_streaming_cost(**kwargs):
    response_cost = kwargs.get("response_cost", 0)
    model = kwargs.get("model")
    usage = kwargs.get("usage")
    # Log to your cost attribution system
    print(f"Model: {model} | Cost: ${response_cost:.6f} | Tokens: {usage}")

litellm.success_callback = [track_streaming_cost]

Source: LiteLLM Spend Tracking

Bedrock Passthrough Streaming Cost Tracking

LiteLLM’s passthrough routes (/bedrock/invoke, /bedrock/converse) added streaming cost tracking in PR #12123. The implementation decodes the event stream from Bedrock’s passthrough response, reconstructs the complete response via _build_complete_streaming_response(), and applies litellm.completion_cost() at stream end. This enables cost attribution for workloads that use the Bedrock SDK directly through LiteLLM as a proxy.

Source: Bedrock Passthrough cost tracking PR

Per-User/Team Cost Attribution

For multi-tenant streaming workloads, use Bedrock Application Inference Profiles combined with LiteLLM’s metadata:

response = litellm.completion(
    model="bedrock/arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/myapp-prod",
    messages=messages,
    stream=True,
    metadata={"user_id": user_id, "team": "engineering"},
)

Application Inference Profiles tag CUR line items for cost allocation by project/team without requiring separate AWS accounts.

Streaming Timeout Configuration

LiteLLM’s streaming iterator includes REPEATED_STREAMING_CHUNK_LIMIT to prevent infinite loops caused by misbehaving model endpoints. For Bedrock-specific timeout handling:

response = litellm.completion(
    model="bedrock/anthropic.claude-sonnet-4-5-20250929-v1:0",
    messages=messages,
    stream=True,
    timeout=60,         # wall-clock timeout for entire stream
    stream_timeout=30,  # timeout for individual chunk delivery
)

Timeouts on streaming calls trigger an asyncio.TimeoutError (async) or ReadTimeout (sync). Unlike a client disconnect, a timeout may not stop server-side generation before partial billing accrues. Set max_tokens defensively in addition to timeout configuration.

9. Real-Time Cost Estimation During Streaming

The Core Problem

Token counts for a streaming request are only available in the final metadata event (ConverseStream) or the last chunk of InvokeModelWithResponseStream. There is no mid-stream token counter published by AWS. For budget enforcement on interactive sessions, you must estimate cost incrementally from character counts and enforce limits before submitting requests.

Pre-Request Budget Check Pattern

import anthropic  # for token counting; works with Bedrock models
import litellm

COST_PER_INPUT_TOKEN = 3.00 / 1_000_000   # Claude Sonnet 4.x on-demand
COST_PER_OUTPUT_TOKEN = 15.00 / 1_000_000
MAX_SESSION_BUDGET_USD = 0.50

def estimate_max_response_cost(messages: list, max_tokens: int) -> float:
    """Upper-bound cost estimate before invoking the model."""
    # Count input tokens via Bedrock CountTokens API or tiktoken approximation
    input_count = sum(len(m["content"].split()) * 1.33 for m in messages)  # rough approximation
    max_output_cost = max_tokens * COST_PER_OUTPUT_TOKEN
    input_cost = input_count * COST_PER_INPUT_TOKEN
    return input_cost + max_output_cost

def guarded_streaming_call(messages, session_spend_so_far, session_budget):
    estimated = estimate_max_response_cost(messages, max_tokens=1000)
    if session_spend_so_far + estimated > session_budget:
        raise BudgetExceededError(f"Projected cost ${estimated:.4f} would exceed session budget")
    
    response = litellm.completion(
        model="bedrock/anthropic.claude-sonnet-4-5-20250929-v1:0",
        messages=messages,
        stream=True,
        max_tokens=1000,
    )
    return response

Bedrock CountTokens API

For precise pre-request token counting, use the CountTokens API:

import boto3

bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")

token_response = bedrock.count_tokens(
    modelId="anthropic.claude-sonnet-4-5-20250929-v1:0",
    messages=messages,
    system=[{"text": system_prompt}],
)
input_token_count = token_response["inputTokenCount"]

This is a synchronous API call that does not generate output — it returns only the token count for the given request body. It adds a round-trip before each streaming call but enables budget enforcement based on actual (not estimated) input token counts.

Mid-Stream Token Estimation

If you must track token consumption during a long streaming response without waiting for the final metadata event, use character-count heuristics:

English prose: approximately 4 characters per token
Code: approximately 3 characters per token
Whitespace-dense content: approximately 5 characters per token

Track cumulative character output per stream. At 4 chars/token average and $15.00/million output tokens (Claude Sonnet 4.x), each 4,000 characters represents approximately $0.015. For a $0.10 per-session budget with 50% reserved for output, enforce a 26,667-character streaming limit.

10. Streaming for Agent Loops: When to Stream vs Not Stream

The Fundamental Tradeoff

Agent loops consist of two types of inference calls:

Reasoning steps — the model generates thought, selects tools, formulates tool calls. The output is a structured JSON block that the orchestrator consumes programmatically.
Final response — the model generates human-readable text delivered to the end user.

For reasoning steps, streaming provides no user-facing benefit and creates overhead. The orchestrator cannot act on a partial tool call — it must wait for the complete ContentBlockStop event before dispatching the tool. Streaming these calls adds connection overhead and makes token counting more complex without improving latency.

For final response generation, streaming is essential for interactive UX. Users tolerate 3–5 seconds for TTFT on a final response; they do not tolerate 15–30 seconds of silence while waiting for a complete response.

Recommended Pattern: Hybrid Streaming

async def agent_step(messages, is_final_response: bool):
    model_id = "bedrock/anthropic.claude-sonnet-4-5-20250929-v1:0"
    
    if is_final_response:
        # Stream to user — latency-sensitive
        response = await litellm.acompletion(
            model=model_id,
            messages=messages,
            stream=True,
            max_tokens=2000,
        )
        return response  # caller iterates and sends chunks to frontend
    else:
        # Synchronous for tool routing — completeness required before dispatch
        response = await litellm.acompletion(
            model=model_id,
            messages=messages,
            stream=False,
            max_tokens=500,  # tool calls are compact
        )
        return response.choices[0].message.content

Cost Amplification in Agent Loops

Multi-turn agent architectures amplify token costs because each turn re-sends the full conversation history as input tokens. For an agent loop with N turns, input token cost scales as O(N²) if full context is preserved. Streaming does not change this — the economics of multi-turn context accumulation apply regardless of delivery mode.

Empirical observation from production agent workloads: total inference cost is 5–8x higher than a single-turn baseline due to context amplification. Reducing max loop iterations from 10 to 6 improved p95 latency by 35% while maintaining 95%+ task completion — the cost savings were proportional. Source: Building AgentoCore: Enterprise-Grade Autonomous AI Agents with AWS Bedrock

Streaming and Prompt Caching in Agent Loops

Agent loops with stable system prompts are ideal candidates for prompt caching. In a 6-turn loop with a 2,000-token system prompt:

Without caching: 2,000 × 6 = 12,000 input tokens for system prompt alone
With caching (5-minute TTL, all hits): 2,000 cache write tokens on turn 1, then 2,000 cache read tokens × 5 = 10,000 cache read tokens

At Claude Sonnet 4.x pricing ($3.00/M input, $0.30/M cache read), this reduces system prompt input cost by ~90% over the loop duration. The cache hit rate is high because loop turns execute sequentially within seconds — well within the 5-minute sliding window.

11. CloudWatch Metrics for Streaming

Complete Metrics Reference

All metrics below are in the AWS/Bedrock namespace, available with ModelId and Region dimensions.

Metric	Type	Streaming only	Description
`InvocationLatency`	Milliseconds	No	Wall-clock time from request receipt to last token generated
`TimeToFirstToken`	Milliseconds	Yes	Time from request receipt to first response token (streaming APIs only)
`OutputTokenCount`	Count	No	Output tokens generated per request
`InputTokenCount`	Count	No	Non-cached input tokens per request
`CacheReadInputTokens`	Count	No	Input tokens read from prompt cache
`CacheWriteInputTokens`	Count	No	Input tokens written to prompt cache
`EstimatedTPMQuotaUsage`	Count	No	Burndown-adjusted quota consumption estimate
`InvocationClientErrors`	Count	No	4xx errors (includes streaming validation errors)
`InvocationServerErrors`	Count	No	5xx errors (includes `modelStreamErrorException` HTTP 424)
`InvocationThrottles`	Count	No	429 throttle events (includes streaming throttles)

Source: Improve operational visibility for inference workloads on Amazon Bedrock with new CloudWatch metrics for TTFT and Estimated Quota Consumption

OTPS Alarm (Model Throughput Degradation)

import boto3

cw = boto3.client("cloudwatch", region_name="us-east-1")

MODEL_ID = "us.anthropic.claude-sonnet-4-5-20250929-v1:0"
OTPS_THRESHOLD = 44  # ~80% of expected ~55 tokens/second baseline

cw.put_metric_alarm(
    AlarmName="Bedrock-OTPS-Low",
    AlarmDescription="Fires when model-side throughput drops, isolating service degradation from workload changes.",
    Metrics=[
        {
            "Id": "m1",
            "MetricStat": {
                "Metric": {
                    "Namespace": "AWS/Bedrock",
                    "MetricName": "InvocationLatency",
                    "Dimensions": [{"Name": "ModelId", "Value": MODEL_ID}],
                },
                "Period": 300,
                "Stat": "p50",
            },
            "ReturnData": False,
        },
        {
            "Id": "m2",
            "MetricStat": {
                "Metric": {
                    "Namespace": "AWS/Bedrock",
                    "MetricName": "OutputTokenCount",
                    "Dimensions": [{"Name": "ModelId", "Value": MODEL_ID}],
                },
                "Period": 300,
                "Stat": "p50",
            },
            "ReturnData": False,
        },
        {
            "Id": "m3",
            "MetricStat": {
                "Metric": {
                    "Namespace": "AWS/Bedrock",
                    "MetricName": "TimeToFirstToken",
                    "Dimensions": [{"Name": "ModelId", "Value": MODEL_ID}],
                },
                "Period": 300,
                "Stat": "p50",
            },
            "ReturnData": False,
        },
        {
            "Id": "otps",
            "Expression": "m2 / (m1 - m3) * 1000",
            "Label": "OTPS",
            "ReturnData": True,
        },
    ],
    ComparisonOperator="LessThanThreshold",
    Threshold=OTPS_THRESHOLD,
    EvaluationPeriods=5,
    DatapointsToAlarm=3,
    TreatMissingData="ignore",
    AlarmActions=[],  # add SNS ARN
)

Source: Diagnose InvocationLatency increases using output tokens per second (OTPS)

TTFT Percentile Distribution Alarm

cw.put_metric_alarm(
    AlarmName="Bedrock-TTFT-p95-High",
    AlarmDescription="Fires when p95 TTFT exceeds interactive threshold, indicating prefill latency degradation.",
    Metrics=[
        {
            "Id": "m1",
            "MetricStat": {
                "Metric": {
                    "Namespace": "AWS/Bedrock",
                    "MetricName": "TimeToFirstToken",
                    "Dimensions": [{"Name": "ModelId", "Value": MODEL_ID}],
                },
                "Period": 300,
                "Stat": "p95",
            },
            "ReturnData": True,
        }
    ],
    ComparisonOperator="GreaterThanThreshold",
    Threshold=3000,  # ms — tune to your SLA
    EvaluationPeriods=3,
    DatapointsToAlarm=2,
    TreatMissingData="notBreaching",
    AlarmActions=[],
)

Streaming Error Rate vs Synchronous

Streaming-specific error codes to monitor:

modelStreamErrorException (HTTP 424) — stream error mid-generation; appears in InvocationServerErrors. Requires client-side retry logic since the connection was already established and the HTTP 200 header was already sent before the error occurred.
ModelTimeoutException (HTTP 408) — generation exceeded model timeout; more common in streaming (long outputs at low OTPS) than synchronous calls.
ThrottlingException (HTTP 429) — quota exceeded; same mechanism for streaming and synchronous, but streaming requests with high max_tokens are more susceptible due to the upfront quota deduction mechanic.

12. Bedrock Converse API Streaming with Tool Use: Event Sequence and Token Counting

The Streaming Tool Use Event Sequence

When a ConverseStream request generates a tool use response, the stream delivers events in this order:

messageStart       { role: "assistant" }
contentBlockStart  { contentBlockIndex: 0, start: { toolUse: { toolUseId: "call_abc123", name: "get_weather" } } }
contentBlockDelta  { contentBlockIndex: 0, delta: { toolUse: { input: '{"loca' } } }
contentBlockDelta  { contentBlockIndex: 0, delta: { toolUse: { input: 'tion": "NYC"}' } } }
contentBlockStop   { contentBlockIndex: 0 }
messageStop        { stopReason: "tool_use" }
metadata           { usage: { inputTokens: 450, outputTokens: 89, totalTokens: 539 }, metrics: { latencyMs: 2100 } }

The orchestrator must buffer ContentBlockDelta events to reconstruct the complete tool input JSON before dispatching the tool call. Partial JSON fragments are not valid for dispatch. The ContentBlockStop event signals that the tool input buffer is complete.

Token Counting for Streaming Tool Calls

Token counts for tool calls in ConverseStream are reported in the final metadata event and include:

inputTokens: Full conversation history + system prompt + tool definitions
outputTokens: The tool invocation block content (tool name + arguments as JSON)

Tool definitions in toolConfig are expensive input. A schema with 5 tool definitions can add 200–400 input tokens per request. For agent loops that do not need all tools at every step, dynamically filtering the toolConfig to only include relevant tools reduces input token cost and TTFT (smaller prefill).

Multi-Block Streaming Responses

When the model generates both text and tool calls in a single response (common in chain-of-thought-before-tool-call patterns), the stream delivers multiple numbered content blocks:

contentBlockStart  { contentBlockIndex: 0, start: { text: {} } }
contentBlockDelta  { contentBlockIndex: 0, delta: { text: "I need to check the weather..." } }
contentBlockStop   { contentBlockIndex: 0 }
contentBlockStart  { contentBlockIndex: 1, start: { toolUse: { toolUseId: "...", name: "get_weather" } } }
contentBlockDelta  { contentBlockIndex: 1, delta: { toolUse: { input: '{"location": "NYC"}' } } }
contentBlockStop   { contentBlockIndex: 1 }
messageStop        { stopReason: "tool_use" }
metadata           { usage: { inputTokens: 520, outputTokens: 134 } }

For streaming UI display, route contentBlockDelta events where the preceding contentBlockStart had start.text to the user-facing stream. Buffer contentBlockDelta events where contentBlockStart had start.toolUse for tool dispatch — do not display raw tool input JSON to users.

Key Decision Matrix

Question	Answer
Does streaming cost more than synchronous?	No — identical per-token billing
Does streaming affect TPM quota consumption?	Indirectly — via max_tokens upfront deduction
Is TTFT billed?	No — latency metric only, no time-based billing
Are tokens billed before client disconnect?	Yes — tokens generated before server detects drop are billed
Does ConverseStream appear differently in CUR vs InvokeModelWithResponseStream?	No — same usage type patterns
Does prompt caching reduce TTFT for streaming?	Yes — up to 85% TTFT reduction on cache hit
Should agent loop tool calls use streaming?	No — synchronous preferred; streaming tool calls require buffering before dispatch
Should agent loop final responses use streaming?	Yes — user-facing latency requires streaming delivery

1. Streaming vs Non-Streaming Billing: The Equivalence Principle

2. TimeToFirstToken: What It Is, What Drives It, Cost Implications

The Metric

The Prefill-Decode Model

How TTFT Differs from InvocationLatency

Cost Implications of TTFT

Prompt Caching and TTFT

3. Streaming and Provisioned Throughput: Capacity Consumption Mechanics

How PT Is Priced and What It Buys

The Streaming Capacity Hold Problem

max_tokens and TPM Quota Under Streaming

4. ConverseStream vs InvokeModelWithResponseStream: CUR Appearance and Tool Use

API Design Differences

Tool Use Cost with Streaming

5. Streaming and Prompt Caching: Cache Hit Rate Monitoring

How Cache Hits Affect Streaming Behavior

Cache Miss Rate Monitoring for Streaming Workloads

6. Connection Cost: VPC Endpoints and Long-Lived Streaming Connections

VPC Interface Endpoint Pricing

Streaming vs NAT Gateway

Long-Lived Connections and API Gateway

7. Streaming Abort Handling: Partial Response Billing

What Happens When a Client Disconnects Mid-Stream

Practical Implications

8. LiteLLM Streaming Configuration for Bedrock

Basic Streaming Setup

Streaming Cost Attribution via Callbacks

Bedrock Passthrough Streaming Cost Tracking

Per-User/Team Cost Attribution

Streaming Timeout Configuration

9. Real-Time Cost Estimation During Streaming

The Core Problem

Pre-Request Budget Check Pattern

Bedrock CountTokens API

Mid-Stream Token Estimation

10. Streaming for Agent Loops: When to Stream vs Not Stream

The Fundamental Tradeoff

Recommended Pattern: Hybrid Streaming

Cost Amplification in Agent Loops

Streaming and Prompt Caching in Agent Loops

11. CloudWatch Metrics for Streaming

Complete Metrics Reference

OTPS Alarm (Model Throughput Degradation)

TTFT Percentile Distribution Alarm

Streaming Error Rate vs Synchronous

12. Bedrock Converse API Streaming with Tool Use: Event Sequence and Token Counting

The Streaming Tool Use Event Sequence

Token Counting for Streaming Tool Calls

Multi-Block Streaming Responses

Key Decision Matrix

Sources