← Agent Frameworks 🕐 13 min read
Agent Frameworks

LLM Cost-Per-Outcome Instrumentation

Joining Gateway Telemetry to Business Results (2026) · March 2026

Most Fortune 500 platform teams can generate a Bedrock cost report in 10 minutes and cannot answer "what did that spend produce?" in 10 weeks. This note closes that gap.

The Problem in One Sentence

Most Fortune 500 platform teams can generate a Bedrock cost report in 10 minutes and cannot answer “what did that spend produce?” in 10 weeks. This note closes that gap.

1. The Instrumentation Gap

Why Cost-Per-Token Is Necessary but Not Sufficient

Token-level cost tracking — through LiteLLM SpendLogs, Bedrock invocation logs, or OpenTelemetry gen_ai.usage.input_tokens metrics — tells you what the inference engine consumed. It cannot tell you whether the inference produced anything valuable. These are different questions answered by different systems that have never been designed to talk to each other.

The canonical enterprise situation: a gateway (LiteLLM Proxy, Bedrock, Azure APIM) emits spend events with model, input_tokens, output_tokens, cost_usd, and a request_id. An application layer (LangChain agent, custom API, Bedrock Agent) receives the LLM response and determines whether a document was classified, a ticket was resolved, or a suggestion was accepted. These two systems share exactly one field — the request_id — and the join almost never happens.

LeanOps (2026) describes the pattern: after an initial instrumentation sprint that typically drops costs 40–60% through caching and routing, costs creep back up within six months. The reason is that without a cost attribution graph connecting token spend to business outcomes, optimization decisions are made on cost alone, not on cost-per-unit-of-value. Teams optimize away tokens that were actually producing outcomes.

The FinOps Foundation Crawl/Walk/Run Model

The FinOps Foundation’s AI cost maturity framework (State of FinOps 2025, published 2025) maps to three stages:

Stage What you can answer What you cannot
Crawl Total spend by model, by team Which workloads are worth it
Walk Spend by use case, by user cohort Cost per unit of work produced
Run Cost per business outcome; regression detection when cost/outcome degrades —

The State of FinOps 2025 report identified “Getting to unit economics” as one of the five highest year-over-year priority increases (+5 places). The same report found that the majority of enterprise teams are still in Crawl or early Walk. The specific step that is most frequently skipped: joining gateway spend data to application-layer outcome events. Teams build the gateway. They build the application. They never build the join.

What “Run” Maturity Requires

“Run” maturity requires three capabilities, all of which exist as off-the-shelf components but require intentional wiring:

  1. A stable correlation key threading from inference call to business outcome event (request ID / OTel trace ID)
  2. An outcome schema defining what constitutes a measurable unit of value for each workload type
  3. A join surface — a time-windowed store (S3 + Athena, DynamoDB, ClickHouse) where both spend events and outcome events land, queryable together

None of these are hard. All three are skipped by default.

2. Outcome Taxonomy for LLM Workloads

The correct outcome metric is the one closest to a business decision. Below are standard workload types with their primary and secondary outcome metrics.

Code Generation

Metric Definition Instrumentation point
Cost per accepted suggestion Total inference cost ÷ suggestions accepted by developer IDE plugin accept/reject event → outcome store
Cost per merged PR Total inference cost for a feature branch ÷ PRs merged GitHub webhook on pull_request.closed with merged: true
Cost per defect-free commit Total inference cost ÷ commits that pass CI without regression CI pipeline result event → outcome store

Secondary: cost per test generated, cost per code review comment acted on.

Proxy when primary metrics are unavailable: acceptance rate × session cost (acceptance rate alone has a 0.6–0.8 correlation with merge rate in GitHub Copilot internal studies).

Document Processing

Metric Definition Instrumentation point
Cost per document classified Inference cost ÷ documents where label confidence ≥ threshold Classification callback with document_id, label, confidence
Cost per document summarized Inference cost ÷ summaries reviewed and marked “used” User action event in downstream app
Cost per field extracted Inference cost ÷ extraction fields passing validation Validation hook result event

For high-volume pipelines (>10K docs/day), instrument at the batch level: total batch cost ÷ documents successfully processed. Per-document instrumentation at scale requires sampling (see Section 7).

Customer Service

Metric Definition Instrumentation point
Cost per resolved ticket Inference cost for ticket session ÷ tickets closed without reopen within 48h Ticketing webhook on ticket state transition
Cost per deflected call Inference cost for chat session ÷ sessions that did not escalate to phone Escalation absence event after session close
Cost per satisfied query Inference cost ÷ queries receiving thumbs-up or CSAT ≥ 4 Feedback widget event

Industry benchmarks for comparison (Fin AI, 2026): vendor-priced per-resolution cost ranges from $0.50–$2.00 for AI-handled tickets. Your internal cost per resolved ticket — inference cost only, excluding orchestration overhead — should land well below $0.50 for well-optimized flows. If it does not, the prompt is doing too much work per turn, context windows are too large, or the wrong model tier is being used for the workload.

Warning on definition drift: “resolved” can mean (a) ticket closed by agent, (b) customer did not reopen within N hours, © LLM-as-judge confirmed resolution, or (d) CSAT ≥ threshold. Pick one definition per workload and encode it in the schema. Mixing definitions across quarters makes the metric untrustworthy for trend analysis.

Research and Analysis

Metric Definition Instrumentation point
Cost per validated insight Inference cost for analysis session ÷ insights marked “actionable” by analyst Annotation event in research tool
Cost per analyst-hour saved Inference cost ÷ (analyst time estimate for manual equivalent) Time-saved field in outcome event, self-reported at session end

Cost per analyst-hour saved is harder to instrument precisely but has the highest executive legibility. The self-report bias is acceptable if the question is asked consistently: “Without AI assistance, how long would this analysis have taken? [<30min / 30min–2h / 2–8h / >8h]”.

Internal Q&A and RAG

Metric Definition Instrumentation point
Cost per answered question Inference cost ÷ questions receiving thumbs-up or no follow-up Feedback event or session-end absence of follow-up query
Cost per session Total inference cost for a user session Session boundary events (session_start, session_end)
Cost per satisfied query Inference cost ÷ queries with positive signal Inline feedback widget

RAG cost decomposition: the total cost per answer has three components — (1) retrieval (embedding call), (2) context assembly (chunk selection), and (3) generation (completion call). Instrument all three with the same session_id to isolate which component is driving cost increases when cost/outcome degrades.

3. Request ID Propagation Patterns

This is the plumbing section. Everything in Section 2 depends on getting this right first.

The Correlation Key Problem

Three systems need to share the same identifier:

  1. Gateway (LiteLLM, Bedrock, APIM) — emits spend event, owns request_id
  2. Application (LangChain, custom API, agent framework) — processes LLM response, owns the business outcome
  3. Outcome store — where both events land for joining

The correlation key must be established before the inference call, passed through the LLM gateway, and injected into the outcome event by the application layer.

LiteLLM: metadata Field and x-litellm-spend-logs-metadata Header

LiteLLM propagates custom context through the metadata field in completions requests and through the x-litellm-spend-logs-metadata HTTP header when calling the proxy. Both paths land in the SpendLog record.

Python — SDK usage:

import litellm
import uuid

# Generate correlation ID before the call
correlation_id = str(uuid.uuid4())
session_id = "session_abc123"
workload_type = "customer_service_deflection"

response = litellm.completion(
    model="gpt-4o",
    messages=[{"role": "user", "content": user_message}],
    metadata={
        "correlation_id": correlation_id,
        "session_id": session_id,
        "workload_type": workload_type,
        "ticket_id": ticket_id,
        "user_id": user_id,
    }
)

# Store correlation_id → use in outcome event later
request_store.set(correlation_id, {
    "ticket_id": ticket_id,
    "session_id": session_id,
    "created_at": time.time()
})

Python — LiteLLM Proxy via HTTP:

import httpx, json, uuid

correlation_id = str(uuid.uuid4())

resp = httpx.post(
    "http://litellm-proxy:4000/v1/chat/completions",
    headers={
        "Authorization": f"Bearer {LITELLM_API_KEY}",
        "x-litellm-spend-logs-metadata": json.dumps({
            "correlation_id": correlation_id,
            "workload_type": "doc_classification",
            "document_id": doc_id,
        }),
        "x-litellm-tags": "doc-processing,prod",
    },
    json={
        "model": "claude-3-5-sonnet",
        "messages": messages,
    }
)

Known issue (as of LiteLLM ~v1.57): On the /v1/messages (Anthropic-format) endpoint, user-supplied values inside litellm_metadata are silently overwritten by proxy-injected metadata. Use the /v1/chat/completions endpoint for reliable metadata propagation, or pass via the HTTP header approach above.

TypeScript — LiteLLM Proxy:

import OpenAI from "openai";
import { v4 as uuidv4 } from "uuid";

const client = new OpenAI({
  baseURL: "http://litellm-proxy:4000/v1",
  apiKey: process.env.LITELLM_API_KEY,
  defaultHeaders: {
    "x-litellm-tags": "customer-service,prod",
  },
});

const correlationId = uuidv4();

const response = await client.chat.completions.create({
  model: "gpt-4o",
  messages,
  // @ts-ignore — LiteLLM extends the OpenAI schema
  metadata: {
    correlation_id: correlationId,
    session_id: sessionId,
    workload_type: "ticket_deflection",
    ticket_id: ticketId,
  },
});

// Attach correlation_id to the outcome event when ticket resolves

OpenTelemetry Trace ID as the Canonical Correlation Key

When your application already uses OTel tracing, the trace_id from the active span is the cleanest correlation key — it is globally unique, already present in infrastructure traces, and supported natively by LiteLLM’s OTel callback.

from opentelemetry import trace
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
import litellm

tracer = trace.get_tracer("my_app")

with tracer.start_as_current_span("handle_ticket") as span:
    otel_trace_id = format(span.get_span_context().trace_id, "032x")
    
    # Pass OTel trace ID through LiteLLM metadata
    response = litellm.completion(
        model="gpt-4o",
        messages=messages,
        metadata={
            "otel_trace_id": otel_trace_id,
            "session_id": session_id,
            "ticket_id": ticket_id,
        }
    )
    
    # The same trace_id appears in:
    # - LiteLLM SpendLog (via metadata)
    # - OTel backend (natively)
    # - Your outcome event (injected below)
    
    outcome_event = {
        "otel_trace_id": otel_trace_id,
        "ticket_id": ticket_id,
        "resolved": True,
        "resolution_time_ms": resolution_time,
        "ts": time.time(),
    }
    emit_outcome(outcome_event)

Configure LiteLLM’s OTel callback in litellm_settings (proxy config):

litellm_settings:
  success_callback: ["otel"]
  otel_exporter: otlp_http
  otel_endpoint: "http://otel-collector:4318"

The resulting span will carry gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and your custom metadata attributes, joinable on trace_id in your observability backend.

Amazon Bedrock: requestMetadata Tagging

Bedrock’s Converse and ConverseStream APIs accept a requestMetadata dict (up to 16 key-value pairs, 256 chars each) that surfaces in CloudWatch invocation logs.

import boto3
import uuid

bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
correlation_id = str(uuid.uuid4())

response = bedrock.converse(
    modelId="anthropic.claude-3-5-sonnet-20241022-v2:0",
    messages=[{"role": "user", "content": [{"text": user_message}]}],
    requestMetadata={
        "correlation_id": correlation_id,
        "workload_type": "doc_classification",
        "team": "finance-ops",
        "environment": "prod",
        "document_id": doc_id[:200],  # stay under 256-char limit
    }
)

# CloudWatch invocation log entry will contain requestMetadata
# Join to outcome events on correlation_id

Cost attribution note: Bedrock requestMetadata does NOT flow into AWS Cost Explorer or Cost and Usage Reports as cost allocation tags. To attribute costs from invocation logs: pull CloudWatch log events, multiply inputTokenCount × input token rate + outputTokenCount × output token rate, group by requestMetadata.team or requestMetadata.workload_type. The AWS blog post “Cost tracking multi-tenant model inference on Amazon Bedrock” (2024) has the canonical Athena query for this.

For Application Inference Profiles (Bedrock’s cost tag mechanism): use profiles to tag at the workload level (e.g., one profile per use case). requestMetadata handles the request-level correlation; inference profiles handle the billing-level tag.

Bedrock Agents: Extracting Token Counts from Trace Objects

Bedrock Agents return a trace object in each response chunk. The token counts live inside the orchestration trace:

import boto3, json

bedrock_agent = boto3.client("bedrock-agent-runtime", region_name="us-east-1")

total_input_tokens = 0
total_output_tokens = 0
correlation_id = str(uuid.uuid4())

response = bedrock_agent.invoke_agent(
    agentId=AGENT_ID,
    agentAliasId=AGENT_ALIAS_ID,
    sessionId=session_id,
    inputText=user_input,
    sessionState={
        "sessionAttributes": {
            "correlation_id": correlation_id,
            "workload_type": "customer_service",
        }
    }
)

for event in response["completion"]:
    if "trace" in event:
        trace = event["trace"]["trace"]
        # Orchestration trace contains token usage per LLM call
        if "orchestrationTrace" in trace:
            orch = trace["orchestrationTrace"]
            if "modelInvocationOutput" in orch:
                usage = orch["modelInvocationOutput"].get("usage", {})
                total_input_tokens += usage.get("inputTokens", 0)
                total_output_tokens += usage.get("outputTokens", 0)
    elif "chunk" in event:
        final_response = event["chunk"]["bytes"].decode()

# Emit cost event with token counts and correlation_id
emit_spend_event({
    "correlation_id": correlation_id,
    "session_id": session_id,
    "input_tokens": total_input_tokens,
    "output_tokens": total_output_tokens,
    "model": "claude-3-5-sonnet",
    "ts": time.time(),
})

4. Data Pipeline Architecture

The Join Problem: Temporal Mismatch

Spend events and outcome events are not co-temporal. The spend event fires at inference completion — typically within 100–500ms of the call. The outcome event fires when the user resolves a ticket, accepts a code suggestion, or a document passes validation — which may be seconds, minutes, or hours later. In some workloads (overnight batch processing, multi-day customer service threads) the gap is measured in days.

The pipeline architecture must handle this temporal mismatch explicitly.

Reference Architecture: S3 + Athena

┌──────────────────────────────────────────────────────────────────────┐
│                         SPEND SIDE                                   │
│                                                                      │
│  LiteLLM Proxy ──► S3 spend events         ─┐                       │
│  Bedrock CW Logs ──► Lambda ──► S3          ─┤──► s3://ai-economics/ │
│  OTel Collector ──► S3 (via exporter)      ─┘    /spend/year=.../   │
│                                                   /dt=.../           │
└──────────────────────────────────────────────────────────────────────┘
                                    │
                                    │  Athena JOIN on correlation_id
                                    │  with late-arrival window
                                    ▼
┌──────────────────────────────────────────────────────────────────────┐
│                        OUTCOME SIDE                                  │
│                                                                      │
│  App layer ──► outcome event ──► Kinesis Firehose ──► S3            │
│  Ticketing webhooks ──► Lambda ──► S3                                │
│  CI/CD webhooks ──► Lambda ──► S3                  /outcomes/year=.../│
│  IDE events ──► S3                                 /dt=.../          │
│                                                                      │
└──────────────────────────────────────────────────────────────────────┘

Spend Event Schema

{
  "event_type": "llm_spend",
  "request_id": "litellm-req-uuid",
  "correlation_id": "app-level-uuid",
  "otel_trace_id": "32-hex-trace-id",
  "session_id": "session-uuid",
  "workload_type": "customer_service_deflection",
  "team": "cx-platform",
  "model": "gpt-4o-2024-11-20",
  "provider": "openai",
  "environment": "prod",
  "input_tokens": 1842,
  "output_tokens": 347,
  "cost_usd": 0.00832,
  "latency_ms": 1247,
  "cache_hit": false,
  "ts": "2026-06-17T14:23:11.000Z",
  "date": "2026-06-17"
}

Outcome Event Schema

{
  "event_type": "outcome",
  "correlation_id": "app-level-uuid",
  "otel_trace_id": "32-hex-trace-id",
  "session_id": "session-uuid",
  "workload_type": "customer_service_deflection",
  "outcome_metric": "ticket_resolved",
  "outcome_value": 1,
  "outcome_signal": "no_reopen_48h",
  "entity_id": "ticket-98765",
  "user_id": "user-anon-hash",
  "team": "cx-platform",
  "environment": "prod",
  "ts": "2026-06-17T15:47:33.000Z",
  "date": "2026-06-17",
  "spend_ts": "2026-06-17T14:23:11.000Z"
}

The spend_ts field on the outcome event is optional but useful for computing inference-to-resolution latency.

Athena Join Query

-- Cost per resolved ticket, last 30 days, with 48h late-arrival window
WITH spend AS (
  SELECT
    correlation_id,
    session_id,
    workload_type,
    team,
    model,
    SUM(cost_usd) AS session_cost_usd,
    SUM(input_tokens) AS total_input_tokens,
    SUM(output_tokens) AS total_output_tokens,
    MIN(ts) AS first_inference_ts
  FROM "ai_economics"."spend"
  WHERE date BETWEEN
    DATE_FORMAT(CURRENT_DATE - INTERVAL '32' DAY, '%Y-%m-%d')
    AND DATE_FORMAT(CURRENT_DATE, '%Y-%m-%d')
    AND workload_type = 'customer_service_deflection'
  GROUP BY correlation_id, session_id, workload_type, team, model
),
outcomes AS (
  SELECT
    correlation_id,
    session_id,
    outcome_metric,
    outcome_value,
    ts AS outcome_ts
  FROM "ai_economics"."outcomes"
  WHERE date BETWEEN
    DATE_FORMAT(CURRENT_DATE - INTERVAL '30' DAY, '%Y-%m-%d')
    AND DATE_FORMAT(CURRENT_DATE, '%Y-%m-%d')
    AND workload_type = 'customer_service_deflection'
)
SELECT
  DATE_TRUNC('day', CAST(s.first_inference_ts AS TIMESTAMP)) AS day,
  s.team,
  s.model,
  COUNT(DISTINCT s.session_id) AS total_sessions,
  COUNT(DISTINCT o.session_id) AS resolved_sessions,
  ROUND(COUNT(DISTINCT o.session_id) * 1.0 /
    NULLIF(COUNT(DISTINCT s.session_id), 0), 3) AS resolution_rate,
  SUM(s.session_cost_usd) AS total_cost_usd,
  ROUND(SUM(s.session_cost_usd) /
    NULLIF(COUNT(DISTINCT o.session_id), 0), 4) AS cost_per_resolved_ticket
FROM spend s
LEFT JOIN outcomes o
  ON s.correlation_id = o.correlation_id
  AND o.outcome_ts > s.first_inference_ts  -- outcome must follow spend
  AND o.outcome_ts < s.first_inference_ts + INTERVAL '48' HOUR  -- late-arrival window
GROUP BY 1, 2, 3
ORDER BY 1 DESC, cost_per_resolved_ticket ASC;

Latency and Late-Arrival Handling

For workloads where outcomes arrive more than 24 hours after inference:

  1. Partition by outcome date, not spend date. The outcome date is the join anchor. Spend records must be queryable by a date range that precedes the outcome date.
  2. Use a configurable late-arrival window in the join condition. For ticket resolution: 48h. For PR merges: 7 days. For document validation batch jobs: match to the batch run timestamp, not wall time.
  3. For real-time dashboards: maintain a rolling “pending” session store (DynamoDB with TTL = late-arrival window). Match outcomes to pending sessions on write; emit a join record to the analytics store when matched.

5. Instrumentation for Specific Stacks

LangChain / LangSmith

LangSmith records token usage and cost natively for OpenAI and Anthropic calls. The gap is connecting LangSmith trace data to outcome events.

Pattern: Use LangSmith run ID as the correlation key.

from langchain_openai import ChatOpenAI
from langchain_core.callbacks import BaseCallbackHandler
from langsmith import Client
import uuid

class OutcomeCorrelationCallback(BaseCallbackHandler):
    """Captures the LangSmith run_id for outcome correlation."""
    
    def __init__(self, ticket_id: str, outcome_store):
        self.ticket_id = ticket_id
        self.outcome_store = outcome_store
        self.run_id = None
    
    def on_llm_start(self, serialized, messages, run_id=None, **kwargs):
        self.run_id = str(run_id) if run_id else str(uuid.uuid4())
        # Register pending session
        self.outcome_store.register_pending(
            correlation_id=self.run_id,
            ticket_id=self.ticket_id,
        )
    
    def on_llm_end(self, response, run_id=None, **kwargs):
        # run_id is now surfaced in LangSmith as the trace anchor
        pass

# Usage
callback = OutcomeCorrelationCallback(ticket_id=ticket_id, outcome_store=store)
llm = ChatOpenAI(model="gpt-4o", callbacks=[callback])
result = llm.invoke(messages)

# Later, when ticket resolves:
def on_ticket_resolved(ticket_id: str):
    pending = store.lookup_by_ticket(ticket_id)
    if pending:
        emit_outcome_event({
            "correlation_id": pending.correlation_id,  # = LangSmith run_id
            "workload_type": "ticket_deflection",
            "outcome_metric": "ticket_resolved",
            "outcome_value": 1,
            "entity_id": ticket_id,
        })

Getting cost from LiteLLM into LangSmith: configure LiteLLM with success_callback = ["langsmith"]. As of 2025, token counts flow correctly but cost display has an intermittent issue on some model providers (reported in LangChain forum). Workaround: compute cost client-side using LiteLLM’s completion_cost() helper and attach to the LangSmith run via client.update_run().

import litellm
from langsmith import Client

ls_client = Client()
cost = litellm.completion_cost(completion_response=response)

ls_client.update_run(
    run_id=callback.run_id,
    extra={"cost_usd": cost, "ticket_id": ticket_id}
)

LlamaIndex

LlamaIndex’s callback system provides TokenCountingHandler for token tracking and LlamaDebugHandler for detailed trace inspection.

from llama_index.core import VectorStoreIndex, Settings
from llama_index.core.callbacks import (
    CallbackManager,
    TokenCountingHandler,
    LlamaDebugHandler,
)
from llama_index.core.callbacks.base import BaseCallbackHandler
from llama_index.core.callbacks.schema import CBEventType, EventPayload
import tiktoken, uuid, time

class OutcomeCorrelationHandler(BaseCallbackHandler):
    """Emits cost events from LlamaIndex sessions to the outcome pipeline."""
    
    def __init__(self, session_id: str, workload_type: str, outcome_store):
        super().__init__([], [])
        self.session_id = session_id
        self.workload_type = workload_type
        self.outcome_store = outcome_store
    
    def on_event_end(self, event_type, payload=None, event_id="", **kwargs):
        if event_type == CBEventType.LLM and payload:
            response = payload.get(EventPayload.RESPONSE)
            if response and hasattr(response, "raw"):
                usage = response.raw.get("usage", {})
                self.outcome_store.record_spend(
                    session_id=self.session_id,
                    workload_type=self.workload_type,
                    input_tokens=usage.get("prompt_tokens", 0),
                    output_tokens=usage.get("completion_tokens", 0),
                    ts=time.time(),
                )
    
    def on_event_start(self, event_type, payload=None, event_id="", **kwargs):
        pass

# Assemble callback stack
token_counter = TokenCountingHandler(
    tokenizer=tiktoken.encoding_for_model("gpt-4o").encode
)
correlation_handler = OutcomeCorrelationHandler(
    session_id=session_id,
    workload_type="rag_qa",
    outcome_store=store,
)

Settings.callback_manager = CallbackManager([token_counter, correlation_handler])

Direct Bedrock SDK

See Section 3 for requestMetadata tagging. The standard pattern for direct SDK usage:

  1. Tag every converse() call with correlation_id, workload_type, team
  2. Enable CloudWatch model invocation logging in Bedrock (once per account)
  3. Ship CloudWatch logs to S3 via subscription filter
  4. Join on correlation_id using the schema in Section 4

CloudWatch invocation log entry structure (relevant fields):

{
  "schemaType": "ModelInvocationLog",
  "modelId": "anthropic.claude-3-5-sonnet-20241022-v2:0",
  "requestId": "aws-internal-uuid",
  "requestMetadata": {
    "correlation_id": "your-app-uuid",
    "workload_type": "doc_classification",
    "team": "finance-ops"
  },
  "input": { "inputTokenCount": 1842 },
  "output": { "outputTokenCount": 347 }
}

6. Dashboard Design for Unit Economics

The CFO-Facing View

A CFO does not need a trace-level debug panel. The CFO needs:

  1. Total AI spend for the period (big number, trend arrow)
  2. Cost per primary outcome by workload type (table: workload / cost per outcome / period-over-period change / status)
  3. Outcome volume by workload type (are we producing more or less?)
  4. Efficiency trend: cost per outcome over time (is it improving or degrading?)

Deliver this as a single Grafana dashboard with four panels. The data source is Athena or a materialized view in a warehouse.

Grafana Panel Configuration

Panel 1: Total Spend + Resolved Outcomes (dual-axis time series)

Metric A (left axis): SUM(cost_usd) GROUP BY day
Metric B (right axis): COUNT(DISTINCT session_id WHERE outcome=resolved) GROUP BY day
Visualization: time series, bar + line overlay

Panel 2: Cost Per Outcome Table

SELECT
  workload_type,
  ROUND(SUM(cost_usd) / NULLIF(SUM(outcome_value), 0), 4) AS cost_per_outcome,
  COUNT(DISTINCT session_id) AS sessions,
  SUM(outcome_value) AS outcomes,
  ROUND(SUM(outcome_value) * 100.0 / COUNT(DISTINCT session_id), 1) AS outcome_rate_pct
FROM joined_economics
WHERE day >= CURRENT_DATE - INTERVAL 30 DAY
GROUP BY workload_type
ORDER BY cost_per_outcome ASC

Panel 3: Cost Per Outcome Trend (time series, one series per workload)

SELECT
  day,
  workload_type,
  ROUND(SUM(cost_usd) / NULLIF(SUM(outcome_value), 0), 4) AS cost_per_outcome
FROM joined_economics
WHERE day >= CURRENT_DATE - INTERVAL 90 DAY
GROUP BY day, workload_type
ORDER BY day, workload_type

Panel 4: Model Efficiency Heatmap

Dimensions: model (Y-axis) × workload_type (X-axis)
Value: cost_per_outcome
Color scale: green (low) → red (high)
Use case: immediately surfaces when a model is expensive relative to peers for a workload

Alerting: Cost-Per-Outcome Regression Detection

The most actionable alert type is not “cost exceeded threshold” — it is “cost per outcome degraded by X% compared to the prior 7-day average.” This fires on regressions caused by prompt changes, context window bloat, model version rollouts, or traffic mix shifts.

# Pseudo-code for regression alert logic
# Run as a scheduled Lambda every 6 hours

def check_cost_per_outcome_regression(workload_type: str, threshold_pct: float = 0.20):
    current_7d = query_cost_per_outcome(workload_type, days=7)
    prior_7d = query_cost_per_outcome(workload_type, days_offset=7, days=7)
    
    if prior_7d == 0:
        return  # no baseline
    
    change_pct = (current_7d - prior_7d) / prior_7d
    
    if change_pct > threshold_pct:
        send_alert(
            title=f"Cost-per-outcome regression: {workload_type}",
            body=(
                f"{workload_type} cost per outcome is {change_pct:.0%} higher "
                f"than the prior 7-day average.\n"
                f"Current: ${current_7d:.4f} | Prior: ${prior_7d:.4f}\n"
                f"Check for: recent prompt changes, model version update, "
                f"context window expansion, traffic mix shift."
            ),
            severity="warning" if change_pct < 0.50 else "critical",
        )

Grafana equivalent: configure an alert on the cost_per_outcome panel using the $__timeFilter macro. Use a Classic Condition: WHEN avg() OF query (A, 7d, now) IS ABOVE avg() OF query (B, 14d, 7d) * 1.20.

A/B Cost Tracking for Model and Prompt Changes

When evaluating a new model or prompt, run both variants simultaneously with feature-flag routing. The metadata or requestMetadata fields carry the variant tag:

variant = "gpt-4o-mini" if experiment_bucket(user_id) == "control" else "gpt-4o"

response = litellm.completion(
    model=variant,
    messages=messages,
    metadata={
        "correlation_id": correlation_id,
        "experiment_id": "model-downgrade-v2",
        "variant": variant,
        "workload_type": "doc_classification",
    }
)

The Athena query adds experiment_id and variant to the GROUP BY. The CFO-facing summary: “Variant B (gpt-4o-mini) reduced cost per classified document from $0.0024 to $0.0009 with no statistically significant change in classification accuracy.”

7. Practical Shortcuts for Teams Not Ready for Full Instrumentation

Full request-level instrumentation requires: (a) propagating correlation IDs through every inference call, (b) instrumenting the application layer to emit outcome events, and © building the join pipeline. For teams in Crawl or early Walk maturity, these shortcuts provide 70–80% of the signal at 20% of the effort.

Shortcut 1: Cohort-Level Attribution

Instead of joining individual spend events to individual outcome events, join at the cohort level. Define a cohort as all inference calls for a given workload_type in a given time window. Join to outcome volumes from the same time window from the ticketing system, CI/CD, or IDE telemetry.

-- Cohort-level: no request-level join required
SELECT
  DATE_TRUNC('week', spend_date) AS week,
  'customer_service' AS workload_type,
  SUM(cost_usd) AS total_spend,
  -- outcome comes from a separate system, loaded weekly
  t.tickets_resolved,
  ROUND(SUM(cost_usd) / t.tickets_resolved, 4) AS approx_cost_per_resolved
FROM spend s
CROSS JOIN (
  SELECT COUNT(*) AS tickets_resolved
  FROM zendesk_export
  WHERE resolved_by = 'ai_agent'
    AND resolution_week = DATE_TRUNC('week', CURRENT_DATE)
) t
WHERE DATE_TRUNC('week', s.spend_date) = DATE_TRUNC('week', CURRENT_DATE)
  AND s.workload_type = 'customer_service'
GROUP BY 1, 2, t.tickets_resolved

Limitation: this produces an average cost per outcome for the cohort, not a distribution. You cannot identify which sessions were expensive and unproductive. It is directionally correct, suitable for executive reporting, and insufficient for optimization decisions at the session level.

Shortcut 2: Proxy Outcomes

For workloads where true outcome instrumentation requires deep application-layer changes, use a proxy outcome that correlates with the true outcome and is available in gateway telemetry.

True outcome Proxy outcome Correlation Data source
Ticket resolved Session length > 3 turns AND no follow-up session within 1h 0.72 (internal estimate) LiteLLM session logs
Code suggestion accepted Response latency < 3s (fast generation = more likely to be accepted) 0.55 Gateway latency
Document classified correctly Confidence score > 0.85 0.80 (calibrated) Model output field
RAG query satisfied Follow-up query absent within 5 min 0.65 Session event log

Proxy outcomes degrade in accuracy when behavior changes. Use them as lagging indicators, not as the primary optimization target. Replace with true outcome instrumentation as soon as the application layer can be modified.

Shortcut 3: Sampling for High-Volume Pipelines

At scale (>100K inference calls/day), full request-level outcome join is expensive. Sample at 5–10% for cost tracking; use the full population for alerting.

import random

SAMPLE_RATE = 0.05  # 5% sampling for cost-per-outcome tracking

def should_instrument(session_id: str) -> bool:
    """Deterministic sampling: same session always sampled or not."""
    return int(session_id[:8], 16) % 100 < (SAMPLE_RATE * 100)

# In the inference path:
if should_instrument(session_id):
    correlation_id = generate_correlation_id()
    # Full instrumentation path
else:
    correlation_id = None
    # Skip outcome emission

Scale the sampled cost-per-outcome by 1/SAMPLE_RATE for aggregate reporting. The statistical noise at 5% sampling is acceptable for weekly trend dashboards; do not use sampled data for day-level alerting.

Shortcut 4: The Two-Metric Bootstrap

If you have zero outcome instrumentation today, the two-metric bootstrap gets you to Crawl+ in one sprint:

  1. Enable LiteLLM SpendLogs (or Bedrock invocation logging). This gives you spend by model, team, and time. Cost: one config change.
  2. Instrument one high-value workload with cohort-level attribution (Shortcut 1). Pick the workload where the CFO is most likely to ask “is this worth it?” — typically customer service deflection or code generation.

With these two in place, you can answer: “We spent $X on AI last month. Approximately $Y went to customer service, which handled Z tickets at roughly $Y/Z per ticket.” That is enough to make a resource allocation decision and justify the next instrumentation sprint.

Summary: The Implementation Sequence

Sprint What to build What it unlocks
1 Enable gateway spend logging (LiteLLM SpendLogs or Bedrock invocation logs) Cost by model, team, time
2 Add workload_type tag to all inference calls Cost by use case
3 Implement cohort-level outcome join for top-2 workloads Approximate cost per outcome
4 Add correlation_id propagation to top-2 workloads Request-level join (exact cost per outcome)
5 Build Athena join table and Grafana dashboard Executive-facing unit economics
6 Add regression alerting Automated detection of cost efficiency degradation
7 Extend request-level instrumentation to all workloads Full unit economics portfolio

The FinOps Foundation model maps Sprint 1–2 to Crawl, Sprint 3–4 to Walk, Sprint 5–7 to Run.

Sources