← Agent Frameworks 🕐 16 min read

Agent Frameworks

Agentic AI Cost Explosion — Failure Modes, Hard Controls, and Cost-Aware Agent Design (2026)

1. How Agents Compound Costs · March 2026

LLM APIs are stateless. Every turn in a multi-turn agent loop resends the entire conversation history to the model.

Audience: Platform engineers and FinOps leads managing Bedrock, LiteLLM, or DIY agent infrastructure.

The core problem in one sentence: A single LLM call costs cents; agentic loops that accumulate context across steps, retry failed tools, and re-invoke the model on every turn routinely compound to dollars or more per task — a 30–150× cost multiplier that doesn’t show up until production traffic arrives.

The Quadratic Context Problem

LLM APIs are stateless. Every turn in a multi-turn agent loop resends the entire conversation history to the model. This is not a bug — it is how transformer attention works — but the billing consequence is severe.

For an N-step agent with roughly constant per-step tokens T:

Total input tokens ≈ T × (1 + 2 + 3 + ... + N) = T × N(N+1)/2

A 10-step agent with 2 K tokens added per step accumulates ~110 K input tokens (quadratic growth), not 20 K (linear assumption). Researchers at Stanford’s Digital Economy Lab confirmed that “even when running the same agent on the same task, costs varied by up to 30×” because agents cannot predict ahead of time how much context they will accumulate.

ReAct Token Budget Analysis

The ReAct pattern (Reason → Act → Observe loop) is the default architecture for tool-using agents. Real production numbers:

Agent type	Steps	Tokens/step	Total input tokens	Cost at Claude Sonnet 4 pricing ($3/M input)
Direct single-turn query	1	2 K	2 K	$0.006
Minimal ReAct (5 steps)	5	2–3 K	~20 K	$0.06
Typical ReAct (10 steps)	10	3–5 K	~80 K	$0.24
Heavy ReAct (20 steps)	20	4–6 K	~500 K	$1.50
ReAct + reflection loops	20 + 10 critique	4–6 K	~900 K	$2.70

Gartner’s March 2026 analysis puts agentic models at 5–30× more tokens per task than a standard chatbot. LeanOps benchmarks in production show 50× token consumption versus chat. At the top of that range, a workload that costs $0.006 per call becomes $0.30–$0.90 per agent task — and $1.20 per interaction is now a documented 2026 production figure.

Tool Call JSON Overhead

Every model invocation that includes tool definitions pays to re-serialize those definitions as input tokens. A typical action group with 10 tools, each with a JSON schema, adds 1–3 K tokens per step. For a 20-step agent, that is an extra 20–60 K tokens of pure schema overhead per task.

Tool definitions are not free passengers. They ride the full context window on every turn.

Parallel vs Sequential Tool Calls

Modern APIs (Anthropic, OpenAI) support parallel tool calls in a single model response. The tradeoff:

Sequential: Fewer tokens per call (no parallel overhead), but N round-trips × context growth = higher cumulative cost.
Parallel: One model call can dispatch several tools simultaneously, collapsing N steps to N/k steps where k is the parallelism factor. For independent sub-tasks, parallel tool use reduces total tokens by cutting the number of context-accumulating turns.

When tasks allow it, parallel tool calls are the single highest-leverage structural change for cost reduction.

Reflection and Self-Critique Loops

Agents that include a self-critique or verification step after each action — patterns seen in AutoGen, CrewAI, and custom multi-agent pipelines — effectively double token usage for those steps. A 10-step agent with per-step critique:

Step output: 3 K tokens
Critique call: full context (growing) + step output again → additional 4–8 K tokens per step
Net effect: ~1.8× total tokens vs no critique, but the value of catching errors early can justify this if bounded

The danger is unbounded critique loops: critique → revise → critique → revise. Enforce a maximum critique depth (typically 1–2 rounds).

2. Bedrock Agents Cost Anatomy

What Bedrock Agents Adds to the Bill

The Bedrock Agents orchestration layer itself carries no separate fee beyond standard model token rates. But it adds several sources of token inflation that a DIY agent on the raw Bedrock InvokeModel API would not incur:

Orchestration system prompt: Bedrock Agents injects a multi-thousand-token orchestration prompt on every model call. This prompt instructs the model how to format ReAct-style Thought/Action/Observation turns, defines the agent’s persona, and embeds action group routing logic. Observed overhead: 1.5–4 K tokens per invocation.
Action group definitions: All configured action group schemas (Lambda function signatures, API gateway schemas, or inline function definitions) are serialized and injected as tool definitions on every call. A well-populated agent with 5 action groups each exposing 5 operations adds 3–8 K tokens of schema overhead per turn.
Session memory context: When session memory is enabled, Bedrock Agents prepends a memory summary to each call. This is bounded and compressed, but adds 500–2 K tokens depending on memory configuration.
Trace overhead: Bedrock Agents returns a detailed trace object in InlineAgentResponse that includes per-step reasoning, tool inputs/outputs, and model latency. The trace itself is not billed — it is metadata in the API response — but it is invaluable for cost attribution.

Total orchestration overhead vs raw API: 3–12 K additional input tokens per step vs a DIY agent using InvokeModel directly. For a 10-step task, that is 30–120 K extra input tokens — a $0.09–$0.36 premium at Sonnet pricing that scales linearly with agent complexity.

Extracting Per-Step Costs from Bedrock Agents Trace

CloudWatch emits InputTokenCount and OutputTokenCount at the invocation level (the full agent task), not per agent step. This creates an attribution gap: you know what a task cost, but not which step burned the tokens.

The InlineAgentResponse (or InvokeAgentResponse) trace object fills this gap. Each step in the trace includes a modelInvocationInput and modelInvocationOutput field with token counts:

import boto3
import json

bedrock_runtime = boto3.client("bedrock-agent-runtime", region_name="us-east-1")

response = bedrock_runtime.invoke_agent(
    agentId="YOUR_AGENT_ID",
    agentAliasId="YOUR_ALIAS_ID",
    sessionId="session-123",
    inputText="Analyze Q1 revenue by region and flag outliers",
    enableTrace=True,  # required for per-step visibility
)

step_costs = []
for event in response["completion"]:
    if "trace" in event:
        trace = event["trace"]["trace"]
        if "orchestrationTrace" in trace:
            orch = trace["orchestrationTrace"]
            if "modelInvocationInput" in orch:
                step_input = orch["modelInvocationInput"]
                # token counts live in the usage field
                usage = step_input.get("inferenceConfiguration", {})
                # log step context size
                print(f"Step input: {json.dumps(step_input, indent=2)[:500]}")
            if "modelInvocationOutput" in orch:
                output = orch["modelInvocationOutput"]
                metadata = output.get("metadata", {})
                usage = metadata.get("usage", {})
                step_costs.append({
                    "input_tokens": usage.get("inputTokens", 0),
                    "output_tokens": usage.get("outputTokens", 0),
                })

total_input = sum(s["input_tokens"] for s in step_costs)
total_output = sum(s["output_tokens"] for s in step_costs)
print(f"Per-step breakdown: {step_costs}")
print(f"Total — Input: {total_input}, Output: {total_output}")

This per-step trace gives you the context growth curve per task. Step 1 might consume 4 K input tokens; by step 10, that same agent call consumes 18 K input tokens. Plotting this curve per agent type reveals exactly where context bloat concentrates.

AgentCore / Bedrock AgentCore (2026)

Amazon Bedrock AgentCore (GA’d in 2025, broadly deployed in 2026) adds runtime-level observability via OpenTelemetry-compatible telemetry routed to CloudWatch. Per-runtime cost attribution is now possible by correlating each runtime’s IAM role ARN with Bedrock model invocation logs:

# List AgentCore runtimes and their associated costs
import boto3

logs = boto3.client("logs")
cloudwatch = boto3.client("cloudwatch")

# CloudWatch metrics require BOTH Resource (runtime ARN) AND Service dimensions
response = cloudwatch.get_metric_statistics(
    Namespace="AWS/Bedrock/AgentCore",
    MetricName="InputTokenCount",
    Dimensions=[
        {"Name": "Resource", "Value": "arn:aws:bedrock:us-east-1:123456789:agentcore-runtime/my-runtime"},
        {"Name": "Service", "Value": "AgentCore.Runtime"},
    ],
    StartTime="2026-06-01T00:00:00Z",
    EndTime="2026-06-17T00:00:00Z",
    Period=86400,  # daily
    Statistics=["Sum"],
)

3. Cost Explosion Failure Modes

Failure Mode 1: Infinite Retry Loops

What happens: Agent calls a tool (Lambda, API, database query). The tool returns an error. The orchestrator retries without a circuit breaker. The model, unable to distinguish a transient failure from a permanent one, retries indefinitely.

Cost impact: A 3-second Lambda timeout × 100 retries × full context re-serialization per retry = the same agent that costs $0.50 in the happy path costs $50 in the retry-storm path. Uber’s infrastructure team documented this failure mode causing entire monthly API budgets to be consumed before mid-April.

Signature in logs: ClientErrors CloudWatch metric spikes; InvocationLatency grows linearly (not tapering); per-session token count exceeds 10× the task’s expected maximum.

Failure Mode 2: Context Accumulation Without Summarization

What happens: Agent naively passes the full conversation history (Thought + Action + Observation × N) to the model on every step. No pruning, no summarization. By step 15, the context window is carrying 8 prior tool outputs — most of which are no longer needed.

Cost impact: Measured production data shows a 10-step workflow that accumulates context without pruning reaches 15,000+ tokens before the final step, versus 3,000–5,000 tokens with selective injection — a 3–5× input token premium. For a 20-step financial analysis agent running 100 times/day, this is the difference between $30/day and $120/day in input token costs alone.

Real-world measurement: MindStudio benchmarks show selective context injection reduces token usage by 60–70% on multi-step agents.

Failure Mode 3: Over-Broad Tool Definitions

What happens: Agent is configured with all available tools for all possible tasks. A financial analysis agent also has access to email tools, CRM update tools, and calendar management tools — none of which it will use in the current task.

Cost impact: Each irrelevant tool schema adds 200–500 tokens per tool per step. An agent with 20 tools when it needs 3 pays for 17 dead tools on every model call. At 500 tokens per irrelevant tool × 17 tools × 15 steps = 127,500 extra input tokens per task. At Claude Sonnet pricing, that is $0.38 of pure waste per task — before the agent does any real work.

Failure Mode 4: Redundant Retrieval

What happens: RAG retrieval is called at every step because the agent does not track what is already in context. Step 3 retrieves the Q1 revenue report. Step 7 retrieves it again because the retrieval trigger fires on every financial question.

Cost impact: Each RAG retrieval call costs both the embedding compute and the retrieved chunk tokens injected into context. Duplicate retrieved documents inflate context without adding information. A document that is 3 K tokens, retrieved 5 times across a task, adds 15 K tokens of redundant context — $0.045 in direct token waste, plus the upstream cost of 5 embedding queries.

Failure Mode 5: Hallucinated Tool Calls

What happens: Model generates a tool call with invented or malformed parameters — a non-existent field name, a UUID where a slug is expected, a date in the wrong format. The tool call fails. The orchestrator passes the error back to the model. The model retries with another hallucinated parameter. The loop continues.

Cost impact: Each failed tool call round-trip burns: (1) the full context re-serialization to generate the bad call, (2) the tool execution (often a Lambda cold start), (3) the error response injected back into context, growing the history. Three failed attempts before the agent gives up: 3× the per-step cost plus growing error traces in context. For complex tool schemas with many optional fields, hallucination rates on initial calls can reach 15–25% without few-shot examples.

4. Hard Cost Controls

Hard controls enforce limits at the infrastructure layer. The agent cannot bypass them. They trade task completion rate for cost predictability.

Control 1: Step Budget (Max N Steps Per Task)

The most direct control. Set a maximum number of model invocations per agent session. When exceeded, abort and surface to the caller.

# Custom agent loop with step budget
MAX_STEPS = 15

class BudgetedAgentExecutor:
    def __init__(self, max_steps=MAX_STEPS):
        self.max_steps = max_steps
        self.step_count = 0
    
    def should_continue(self) -> bool:
        self.step_count += 1
        if self.step_count > self.max_steps:
            raise StepBudgetExceeded(
                f"Agent exceeded {self.max_steps} steps. "
                f"Task aborted. Human review required."
            )
        return True

For Bedrock Agents, use the maxLength parameter in the agent’s orchestration configuration (maps to maximum orchestration steps). AWS documents this as a session-level setting; combine it with a wrapper that catches the agent’s sessionExpiredException and logs it as a cost-control event, not an error.

Recommended defaults by task type:

Simple lookup / single-tool tasks: max 5 steps
Multi-tool research tasks: max 15 steps
Complex multi-document analysis: max 25 steps
Never allow unbounded: set max_steps=50 as an absolute floor even for complex tasks

Control 2: Token Budget Per Session

A token budget checks cumulative input + output tokens against a threshold and kills the session when exceeded. Unlike a step budget, this is sensitive to context size — a 3-step agent with huge tool outputs hits the token budget before a 10-step agent with small outputs.

# LiteLLM virtual key with per-agent token budget
import litellm

# Create a virtual key scoped to a specific agent identity
# max_budget in USD; budget_duration resets the counter
response = litellm.proxy.generate_key(
    max_budget=0.50,          # $0.50 per session
    budget_duration="1d",     # resets daily; use "1h" for tighter control
    metadata={"agent_id": "financial-analysis-agent-v2"},
    tags=["agent", "finance"],
)
agent_key = response["key"]

# Agent uses this key for all its LLM calls
# LiteLLM proxy enforces the budget before forwarding
# When exceeded: BudgetExceededError raised, not forwarded to model provider

For Bedrock-native deployments without LiteLLM, implement a pre-call Lambda hook that queries a DynamoDB session-cost table, rejects calls over threshold, and logs the rejection as a CloudWatch custom metric.

Control 3: Timeout Guards

Wall-clock time is a cheap proxy for runaway loops. A task that should complete in 30 seconds and is still running after 5 minutes is almost certainly stuck.

import asyncio
from contextlib import asynccontextmanager

@asynccontextmanager
async def agent_timeout(seconds: int, task_id: str):
    try:
        async with asyncio.timeout(seconds):
            yield
    except asyncio.TimeoutError:
        # Emit cost alert before raising
        emit_metric("agent.timeout", tags={"task_id": task_id})
        raise AgentTimeoutError(
            f"Task {task_id} exceeded {seconds}s wall-clock limit. "
            f"Last checkpoint saved. Escalating to human queue."
        )

# Usage
async with agent_timeout(120, task_id="fin-analysis-20260617"):
    result = await run_agent(task)

For Bedrock Agents, set idleSessionTTLInSeconds to prevent orphaned sessions from accumulating context charges. Default is 1800s; set to 300–600s for automated tasks.

Control 4: Circuit Breaker on Tool Failures

If the same tool call fails 3 consecutive times in the same session, abort the task and escalate. Do not allow the agent to retry indefinitely.

from collections import defaultdict

class ToolCircuitBreaker:
    def __init__(self, max_failures: int = 3):
        self.max_failures = max_failures
        self.failure_counts: dict[str, int] = defaultdict(int)
        self.open_circuits: set[str] = set()
    
    def record_failure(self, tool_name: str, error: Exception):
        self.failure_counts[tool_name] += 1
        if self.failure_counts[tool_name] >= self.max_failures:
            self.open_circuits.add(tool_name)
            raise CircuitOpenError(
                f"Tool '{tool_name}' failed {self.max_failures} times. "
                f"Circuit open. Task aborted. Error: {error}"
            )
    
    def record_success(self, tool_name: str):
        self.failure_counts[tool_name] = 0
        self.open_circuits.discard(tool_name)
    
    def is_open(self, tool_name: str) -> bool:
        return tool_name in self.open_circuits
    
    def check(self, tool_name: str):
        if self.is_open(tool_name):
            raise CircuitOpenError(
                f"Tool '{tool_name}' circuit is open. Refusing call."
            )

The five-layer gateway framework documented by AI Security Gateway (2026) formalizes circuit breakers as a pattern that trips on: cost velocity, repeated identical prompts, error rate spike, and context size growth rate. All five triggers belong in a production circuit breaker.

Control 5: LiteLLM `max_budget` Per Virtual Key Scoped to Agent Identity

The key design principle: budget enforcement must live outside the agent code. An agent checking its own budget is a guardrail the agent can walk around on a bad code path. A gateway that enforces budget before forwarding the request is uncircumventable by the agent.

# litellm_config.yaml — proxy configuration
model_list:
  - model_name: claude-sonnet-4
    litellm_params:
      model: bedrock/anthropic.claude-sonnet-4-5
      aws_region_name: us-east-1

general_settings:
  master_key: sk-master-xxxx
  database_url: postgresql://user:pass@localhost/litellm

# Budget enforcement happens at proxy; agent only sees BudgetExceededError

# Create per-agent virtual key via LiteLLM proxy API
curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-master-xxxx" \
  -H "Content-Type: application/json" \
  -d '{
    "max_budget": 2.00,
    "budget_duration": "1d",
    "metadata": {
      "agent_id": "finance-analysis-agent",
      "team": "finops",
      "cost_center": "CC-4421"
    },
    "tags": ["agent", "finance", "prod"]
  }'

LiteLLM also supports session-level budgets via max_budget_per_session on the agent’s litellm_params. Session spend counters expire after 1 hour by default (configurable via LITELLM_MAX_BUDGET_PER_SESSION_TTL). This catches runaway sessions that burn budget within a single task rather than across many tasks.

5. Soft Cost Controls

Soft controls reduce costs without aborting tasks. They trade some accuracy or latency for token efficiency.

Control 1: Context Compression Between Steps

Every K steps, summarize the conversation history using a cheap model (Haiku, Llama 3 8B) and replace the raw history with the summary. The full history is archived but not passed to the model.

import anthropic

client = anthropic.Anthropic()
COMPRESSION_INTERVAL = 5  # compress every 5 steps

def compress_history(history: list[dict], step: int) -> list[dict]:
    if step % COMPRESSION_INTERVAL != 0 or step == 0:
        return history
    
    # Use cheapest model for compression
    raw_history_text = "\n".join(
        f"{msg['role']}: {msg['content']}" for msg in history[:-2]
    )
    
    summary_response = client.messages.create(
        model="claude-haiku-4-5",  # $0.25/M input — 12x cheaper than Sonnet
        max_tokens=500,
        messages=[{
            "role": "user",
            "content": (
                f"Summarize this agent conversation history concisely, "
                f"preserving all key facts and decisions made:\n\n{raw_history_text}"
            )
        }]
    )
    
    summary = summary_response.content[0].text
    
    # Keep: compressed summary + last 2 turns (recency matters)
    return [
        {"role": "system", "content": f"[Compressed history]: {summary}"},
        *history[-2:]
    ]

Measured impact: context compression at every 5 steps reduces total input tokens by 55–70% on 15+ step tasks (MindStudio, 2026). The compression call itself costs ~0.1–0.5 K input tokens at Haiku pricing — negligible compared to the tokens saved.

Control 2: Dynamic Tool Selection

Instead of injecting all tool definitions on every call, select only the tools relevant to the current step’s intent. Use a lightweight classifier (or a fast embedding similarity check) to select the top-K tools from the full tool registry.

from sentence_transformers import SentenceTransformer
import numpy as np

class DynamicToolSelector:
    def __init__(self, all_tools: list[dict], top_k: int = 5):
        self.all_tools = all_tools
        self.top_k = top_k
        self.encoder = SentenceTransformer("all-MiniLM-L6-v2")
        
        # Pre-encode tool descriptions
        self.tool_embeddings = self.encoder.encode(
            [f"{t['name']}: {t['description']}" for t in all_tools]
        )
    
    def select_tools(self, step_context: str) -> list[dict]:
        query_embedding = self.encoder.encode([step_context])
        similarities = np.dot(query_embedding, self.tool_embeddings.T)[0]
        top_indices = np.argsort(similarities)[-self.top_k:][::-1]
        return [self.all_tools[i] for i in top_indices]

An agent with 20 tools that dynamically selects 5 per step reduces tool-definition overhead by 75% — from 3–8 K tokens to 750 K–2 K tokens per turn. For a 15-step task, that recovers 33–90 K tokens of pure overhead.

Control 3: Retrieval Deduplication

Track which document chunks have already been retrieved in the current session. Before executing a RAG call, check whether the requested context is already in the active window.

from hashlib import sha256

class DeduplicatingRetriever:
    def __init__(self, base_retriever):
        self.base_retriever = base_retriever
        self.retrieved_hashes: set[str] = set()
        self.cached_chunks: dict[str, str] = {}
    
    def retrieve(self, query: str, session_id: str) -> list[str]:
        results = self.base_retriever.retrieve(query)
        new_chunks = []
        
        for chunk in results:
            chunk_hash = sha256(chunk.encode()).hexdigest()
            if chunk_hash not in self.retrieved_hashes:
                self.retrieved_hashes.add(chunk_hash)
                self.cached_chunks[chunk_hash] = chunk
                new_chunks.append(chunk)
            # Already in context — skip injection
        
        return new_chunks  # Only return chunks not already seen
    
    def clear_session(self):
        self.retrieved_hashes.clear()
        self.cached_chunks.clear()

In financial analysis workflows where multiple agent steps query the same quarterly reports, deduplication reduces RAG-sourced context tokens by 40–60%.

Control 4: Confidence Threshold for Tool Calls

Agents frequently call tools defensively — retrieving data they already have, confirming facts already established. Add a self-assessment gate before tool dispatch: only proceed if the model’s stated confidence in needing the tool exceeds a threshold.

TOOL_CALL_CONFIDENCE_PROMPT = """
Before calling a tool, assess whether you actually need it.
Rate your confidence that this tool call will provide NEW information
not already present in the conversation history.

Confidence: [0.0 to 1.0]
Reasoning: [one sentence]
Proceed: [yes/no]

Only call the tool if confidence > 0.7 AND proceed = yes.
"""

This is a soft heuristic — not a hard block — but it reduces redundant tool calls by 20–35% in observed production workflows.

Control 5: Speculative Step Prefetching

For agents with predictable step sequences (e.g., financial analysis always retrieves data → normalizes → computes → formats), prefetch the likely next step’s tool result in parallel with the current step’s model call. This saves a full round-trip latency and — when combined with caching — reuses the prefetched result if the prediction was correct.

6. Monitoring and Alerting for Runaway Agents

CloudWatch Alarms for Bedrock Agents

import boto3

cloudwatch = boto3.client("cloudwatch", region_name="us-east-1")

# Alarm 1: Invocation latency > 5 minutes (proxy for runaway loop)
cloudwatch.put_metric_alarm(
    AlarmName="BedrockAgents-RunawayLoop-Latency",
    MetricName="InvocationLatency",
    Namespace="AWS/Bedrock",
    Dimensions=[{"Name": "AgentId", "Value": "YOUR_AGENT_ID"}],
    Statistic="Maximum",
    Period=300,
    EvaluationPeriods=1,
    Threshold=300000,  # 5 minutes in ms
    ComparisonOperator="GreaterThanThreshold",
    AlarmActions=["arn:aws:sns:us-east-1:123456789:agent-cost-alerts"],
    TreatMissingData="notBreaching",
)

# Alarm 2: Error rate spike (tool failures triggering retry loops)
cloudwatch.put_metric_alarm(
    AlarmName="BedrockAgents-HighErrorRate",
    MetricName="ClientErrors",
    Namespace="AWS/Bedrock",
    Dimensions=[{"Name": "AgentId", "Value": "YOUR_AGENT_ID"}],
    Statistic="Sum",
    Period=300,
    EvaluationPeriods=2,
    Threshold=10,
    ComparisonOperator="GreaterThanThreshold",
    AlarmActions=["arn:aws:sns:us-east-1:123456789:agent-cost-alerts"],
)

# Alarm 3: Throttling errors (sign of token velocity spike)
cloudwatch.put_metric_alarm(
    AlarmName="BedrockAgents-ThrottlingSpike",
    MetricName="ThrottlingErrors",
    Namespace="AWS/Bedrock",
    Dimensions=[{"Name": "AgentId", "Value": "YOUR_AGENT_ID"}],
    Statistic="Sum",
    Period=60,
    EvaluationPeriods=3,
    Threshold=5,
    ComparisonOperator="GreaterThanThreshold",
    AlarmActions=["arn:aws:sns:us-east-1:123456789:agent-cost-alerts"],
)

The full Bedrock Agents CloudWatch metric set (available since May 2025): NumberOfInvocations, InvocationLatency, ClientErrors, ThrottlingErrors, InputTokenCount, OutputTokenCount. For AgentCore runtimes, metrics require the Resource (runtime ARN) AND Service (set to AgentCore.Runtime) dimension pair — omitting either causes silent metric gaps.

LiteLLM SpendLogs for Per-Session Attribution

# Query LiteLLM SpendLogs via proxy API for agent cost dashboards
import httpx

async def get_agent_spend(agent_id: str, start_date: str) -> dict:
    async with httpx.AsyncClient() as client:
        response = await client.get(
            "http://localhost:4000/spend/logs",
            headers={"Authorization": f"Bearer {MASTER_KEY}"},
            params={
                "start_date": start_date,
                "metadata": json.dumps({"agent_id": agent_id}),
            }
        )
    logs = response.json()
    
    total_cost = sum(entry["spend"] for entry in logs)
    total_input_tokens = sum(entry["prompt_tokens"] for entry in logs)
    total_output_tokens = sum(entry["completion_tokens"] for entry in logs)
    session_costs = {}
    
    for entry in logs:
        session_id = entry.get("metadata", {}).get("session_id", "unknown")
        session_costs[session_id] = session_costs.get(session_id, 0) + entry["spend"]
    
    return {
        "agent_id": agent_id,
        "total_cost_usd": total_cost,
        "total_input_tokens": total_input_tokens,
        "total_output_tokens": total_output_tokens,
        "session_count": len(session_costs),
        "cost_per_session": total_cost / len(session_costs) if session_costs else 0,
        "most_expensive_session": max(session_costs, key=session_costs.get),
    }

Cost-Per-Completed-Task Regression Alert

Track cost-per-task as a rolling metric. Alert when it regresses beyond 2× the 7-day trailing average — this catches gradual cost creep from context accumulation, new tool additions, or prompt length changes.

# Psuedocode — implement as Lambda on EventBridge 15-min schedule
def check_cost_regression(agent_id: str):
    today_cost_per_task = get_rolling_average(agent_id, hours=1)
    baseline_cost_per_task = get_rolling_average(agent_id, days=7)
    
    if baseline_cost_per_task > 0:
        regression_ratio = today_cost_per_task / baseline_cost_per_task
        if regression_ratio > 2.0:
            alert(
                f"Cost regression detected for {agent_id}: "
                f"${today_cost_per_task:.4f}/task vs "
                f"${baseline_cost_per_task:.4f}/task baseline "
                f"({regression_ratio:.1f}× increase)"
            )

Dead Man’s Switch Pattern

For long-running autonomous agent tasks, implement a heartbeat that an external watchdog expects. If the heartbeat stops arriving — because the agent is stuck in a loop that never completes — the watchdog terminates the session and fires an alert.

import threading
import time

class AgentHeartbeat:
    def __init__(self, timeout_seconds: int = 30, on_timeout=None):
        self.timeout = timeout_seconds
        self.on_timeout = on_timeout
        self._last_beat = time.time()
        self._running = True
        self._thread = threading.Thread(target=self._watchdog, daemon=True)
        self._thread.start()
    
    def beat(self):
        """Agent calls this after each step to signal liveness."""
        self._last_beat = time.time()
    
    def _watchdog(self):
        while self._running:
            elapsed = time.time() - self._last_beat
            if elapsed > self.timeout:
                if self.on_timeout:
                    self.on_timeout(f"Agent stalled for {elapsed:.0f}s")
                self._running = False
            time.sleep(5)
    
    def stop(self):
        self._running = False

7. Cost-Aware Agent Design Patterns

Pattern 1: Cheapest Sufficient Model Per Step Type

Not all agent steps require the same model capability. Use a routing layer that selects the cheapest model that can handle the current step type.

Step type	Recommended model	Approximate cost (Bedrock, per 1M tokens in/out)	Rationale
Task planning / decomposition	Claude Sonnet 4	$3 / $15	Requires strong reasoning
Tool call generation	Claude Haiku 4	$0.25 / $1.25	Structured output, low creativity needed
Retrieval synthesis	Claude Haiku 4	$0.25 / $1.25	Summarization, not reasoning
Multi-doc analysis / judgment	Claude Sonnet 4	$3 / $15	Requires cross-document reasoning
Final synthesis / report	Claude Sonnet 4 or Opus 4	$3–15 / $15–75	Quality matters for output
History compression	Claude Haiku 4	$0.25 / $1.25	Simple summarization

STEP_TO_MODEL = {
    "plan": "bedrock/anthropic.claude-sonnet-4-5",
    "tool_call": "bedrock/anthropic.claude-haiku-4-5",
    "retrieve_synthesize": "bedrock/anthropic.claude-haiku-4-5",
    "analyze": "bedrock/anthropic.claude-sonnet-4-5",
    "compress_history": "bedrock/anthropic.claude-haiku-4-5",
    "final_synthesis": "bedrock/anthropic.claude-sonnet-4-5",
}

def route_step(step_type: str, context: dict) -> str:
    return STEP_TO_MODEL.get(step_type, "bedrock/anthropic.claude-sonnet-4-5")

A financial analysis agent that uses Haiku for tool calls and history compression, reserving Sonnet for planning and final synthesis, reduces blended per-task cost by 40–60% with negligible quality loss on structured subtasks.

Pattern 2: Lazy Retrieval

Do not retrieve proactively. Retrieve only when the model explicitly signals it lacks a required fact. Implement retrieval as a tool the model calls when needed, not as a pre-processing step that runs before every model call.

Before (eager retrieval):

Task → Retrieve all potentially relevant docs → Build large context → Model call

After (lazy retrieval):

Task → Model call with minimal context → Model calls retrieve() tool if needed

Lazy retrieval reduces average retrieval overhead by 50–70% because many agent steps can be completed with information already in context — the model is not offered the option to retrieve things it already has.

Pattern 3: Agent Result Caching

For deterministic or near-deterministic tasks (same input → same expected output), cache completed task results. Before running an agent, check whether a sufficiently similar task was completed recently.

from functools import lru_cache
import hashlib

class AgentResultCache:
    def __init__(self, ttl_seconds: int = 3600):
        self.cache: dict[str, tuple[float, dict]] = {}
        self.ttl = ttl_seconds
    
    def _key(self, task_input: str) -> str:
        return hashlib.sha256(task_input.strip().lower().encode()).hexdigest()
    
    def get(self, task_input: str) -> dict | None:
        key = self._key(task_input)
        if key in self.cache:
            timestamp, result = self.cache[key]
            if time.time() - timestamp < self.ttl:
                return result
        return None
    
    def set(self, task_input: str, result: dict):
        key = self._key(task_input)
        self.cache[key] = (time.time(), result)

For near-duplicate detection (semantically similar but not identical tasks), use a vector similarity threshold (cosine similarity > 0.95) against cached task embeddings. Production deployments that cache agent results for repeated analytical queries — such as daily summary reports or recurring compliance checks — report 60–80% cache hit rates, eliminating those tasks’ LLM costs entirely.

Pattern 4: Human-in-the-Loop at Cost Thresholds

Before the agent enters an expensive phase (e.g., about to spawn 5 parallel sub-agents, or context has already exceeded 50 K tokens), pause and request human confirmation.

HUMAN_APPROVAL_THRESHOLD_USD = 0.25  # pause before spending > $0.25 more

def check_cost_gate(session_cost_so_far: float, projected_next_step_cost: float):
    if (session_cost_so_far + projected_next_step_cost) > HUMAN_APPROVAL_THRESHOLD_USD:
        return HumanApprovalRequired(
            message=(
                f"This task has spent ${session_cost_so_far:.3f} so far. "
                f"Next step is projected to cost ${projected_next_step_cost:.3f}. "
                f"Approve to continue, or abort and review."
            ),
            context=current_agent_state(),
        )
    return None

Human-in-the-loop checkpoints also serve as a natural debugging surface. An agent that asks for approval at $0.25 forces the platform team to see what the agent is doing before it becomes a $5 runaway task.

8. Summary: Cost Control Priority Stack

Apply these in order. The highest-leverage controls come first.

Priority	Control	Mechanism	Expected savings
1	Step budget	Hard kill at N steps	Prevents worst-case runaway
2	Circuit breaker on tool failures	Abort at 3 consecutive failures	Prevents retry-loop cost explosion
3	Token budget per session (LiteLLM gateway)	Gateway-enforced, agent-opaque	Catches overruns before they become unbounded
4	Context compression every K steps	Replace history with Haiku summary	55–70% input token reduction
5	Dynamic tool selection	Inject only relevant tools per step	60–75% tool-definition overhead reduction
6	Cheapest sufficient model routing	Haiku for structured steps, Sonnet for reasoning	40–60% blended cost reduction
7	Lazy retrieval	Retrieve only on explicit model request	50–70% retrieval token reduction
8	Agent result caching	Cache near-identical completed tasks	60–80% cache hit rates in recurring workloads
9	CloudWatch alarms + cost regression alert	Latency proxy + cost-per-task delta	Operational visibility, not prevention
10	Human-in-the-loop at thresholds	Approval gate before expensive phases	Audit surface, prevents authorized runaway

Expected cumulative impact of controls 1–8 applied together: an agent task that naively costs $2.70 (20-step ReAct + reflection) is reducible to $0.30–$0.60 with full control stack applied — a 5–9× cost reduction while maintaining task completion rates above 90% for well-specified tasks.