← Agent Frameworks 🕐 13 min read
Agent Frameworks

LiteLLM Observability and Cost Tracing Integrations (2026)

Executive Summary · March 2026

LiteLLM's callback architecture provides a unified cost and tracing layer across 100+ LLM providers.

LiteLLM’s callback architecture provides a unified cost and tracing layer across 100+ LLM providers. Every request produces a StandardLoggingPayload — a typed struct containing USD cost breakdown, token counts, trace IDs, and business metadata — that is passed to all registered callbacks simultaneously. The same payload feeds Langfuse traces, Datadog dashboards, Prometheus gauges, and custom internal webhooks. The key architectural insight is that LiteLLM is the cost-calculation authority: it resolves provider pricing at call time, populates response_cost, and then distributes that pre-calculated cost to every downstream callback. Observability backends consume, not compute, the cost data.

This matters for production architecture: if LiteLLM’s cost model has a bug (and it does — see Bedrock section), every callback in the chain receives the wrong number. The fix point is always in LiteLLM’s model pricing map, not in the observability backend.

1. Callback Architecture

Lifecycle Hooks

LiteLLM maintains a global callback registry. Callbacks are triggered at four points:

Hook Timing Typical Use
log_pre_api_call Before request leaves proxy Input validation, prompt logging
log_post_api_call After response received, before cost calc Raw response capture
log_success_event After cost calculation completes Cost/token logging to backends
log_failure_event On any exception or timeout Error tracking, alerting

Async variants (async_log_success_event, etc.) are required in proxy mode. The proxy is entirely async; registering a synchronous callback on the proxy will block the event loop.

StandardLoggingPayload

The payload passed to every success callback. Available via kwargs["standard_logging_object"].

# Key fields (simplified)
StandardLoggingPayload:
  id: str                          # Unique log entry ID
  trace_id: str                    # Workflow-level correlation ID
  call_type: str                   # "acompletion", "embedding", etc.
  status: "success" | "failure"
  model: str                       # Requested model name
  
  # Cost breakdown
  response_cost: float             # Total USD (= cost_breakdown.total_cost)
  cost_breakdown:
    input_cost: float              # Prompt tokens (includes cache creation)
    output_cost: float             # Completion tokens (includes reasoning)
    tool_usage_cost: float         # Built-in tool costs (web search, code interp)
    total_cost: float
  
  # Token counts
  prompt_tokens: int
  completion_tokens: int
  total_tokens: int
  
  # Timing
  startTime: float                 # Unix timestamp
  endTime: float
  completionStartTime: float       # TTFT for streaming
  response_time: float
  
  # Business metadata
  request_tags: list[str]          # Custom tags from metadata.tags
  metadata: StandardLoggingMetadata
    user_api_key_alias: str
    user_api_key_team_id: str
    user_api_key_org_id: str
    requester_ip_address: str
    applied_guardrails: list[str]
    mcp_tool_call_metadata: dict   # MCP tool calls with per-tool costs
  
  # Diagnostics
  cache_hit: bool
  cache_key: str
  hidden_params: dict              # Rate limits, model IDs, API endpoints

Supported Callbacks (Full List)

As of mid-2026, the registry maps string identifiers to the following:

Callback Key Type Cost Data Captured
langfuse Tracing (legacy) Yes — from response body
langfuse_otel Tracing (OTEL) Yes — via gen_ai.cost.* attributes
helicone Tracing proxy Yes — forwarded headers
datadog LLM Observability Yes — spend metrics
prometheus Metrics Yes — litellm_spend_metric
otel Generic OTEL Yes — gen_ai.cost.*
arize Tracing Yes — LLM span attributes
langsmith Tracing Partial — token counts
lunary Tracing Yes
traceloop Tracing (OTEL) Yes
athina Eval + tracing Token counts
s3 Blob storage Full payload (JSON)
gcs Blob storage Full payload (JSON)
azure_blob Blob storage Full payload (JSON)
sqs Message queue Full payload (JSON)
sentry Error tracking Error context only
posthog Analytics Custom events
slack Alerting Threshold alerts
logfire Structured logs Yes
grafana_loki Log aggregation Yes
generic_api Webhook Full payload (JSON)

Storage callbacks (S3/GCS/Azure Blob/SQS) capture the complete StandardLoggingPayload as JSON. These are the most complete cost archives and are suitable for FinOps pipelines that need to replay or recompute costs.

2. Langfuse Integration

Two Integration Paths

Path 1 — Legacy langfuse callback

Uses LiteLLM’s own Langfuse SDK client. Cost is read from the LiteLLM response object and written as Langfuse generation usage and cost fields directly.

import litellm
import os

os.environ["LANGFUSE_PUBLIC_KEY"] = "pk-lf-..."
os.environ["LANGFUSE_SECRET_KEY"] = "sk-lf-..."
os.environ["LANGFUSE_HOST"] = "https://us.cloud.langfuse.com"

litellm.callbacks = ["langfuse"]

response = litellm.completion(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Explain CAPM"}],
    metadata={
        "generation_name": "capm-explainer",
        "session_id": "session-finance-001",
        "trace_id": "txn-abc-123",         # Business transaction ID
        "tags": ["prod", "finance-team"],
        "user_id": "analyst-42"
    }
)

Path 2 — OTEL langfuse_otel callback (recommended)

Uses OpenTelemetry export to Langfuse’s OTLP endpoint. Cost flows as gen_ai.cost.* span attributes. This is the path to use when you also want distributed trace context from upstream services.

# config.yaml (LiteLLM Proxy)
litellm_settings:
  callbacks: ["langfuse_otel"]

environment_variables:
  LANGFUSE_PUBLIC_KEY: "pk-lf-..."
  LANGFUSE_SECRET_KEY: "sk-lf-..."
  LANGFUSE_OTEL_HOST: "https://us.cloud.langfuse.com"

Cost Tracking Per Trace/Span

Langfuse tracks cost at the observation (generation) level, not at the trace level. Trace-level cost is the sum of all child generation costs. Key behavior:

  • If LiteLLM sends cost in the response: Langfuse captures it directly without recalculation.
  • If cost is absent (edge case): Langfuse infers from token counts using its built-in model pricing table.
  • Custom models not in Langfuse’s table require manual entry via Project Settings > Models or the Models API.
# Custom model pricing in Langfuse (via API)
POST /api/public/models
{
  "modelName": "my-fine-tuned-gpt4",
  "matchPattern": "(?i)^my-fine-tuned.*",
  "inputPrice": 0.000015,    # USD per token
  "outputPrice": 0.000040,
  "tokenizerId": "cl100k_base"
}

Token Attribution to Business Context

Every Langfuse metadata field is promoted to a searchable attribute. The trace_id field creates a linkage between the Langfuse trace and an external business transaction (e.g., a database row ID, a user session, an order ID).

# Passing business context through to Langfuse
response = litellm.completion(
    model="anthropic/claude-3-5-sonnet",
    messages=messages,
    metadata={
        "trace_id": order_id,           # Links to your order in your DB
        "generation_name": "risk-score",
        "session_id": f"user-{user_id}",
        "tags": ["underwriting", "prod"],
        "user_id": str(user_id)
    }
)

In Langfuse, filtering by tags: ["underwriting"] then grouping by session_id gives per-user LLM cost for the underwriting workflow.

3. Helicone Integration

Status (2026)

Helicone’s LiteLLM callback is maintained but no longer actively developed. Helicone now recommends its own AI Gateway for new integrations. For teams already on LiteLLM Proxy, the callback path still works but lags behind Helicone’s native gateway features.

Callback Setup

import litellm
import os

os.environ["HELICONE_API_KEY"] = "sk-helicone-..."

litellm.success_callbacks = ["helicone"]
litellm.failure_callbacks = ["helicone"]

response = litellm.completion(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Summarize 10-K filing"}],
    metadata={
        "Helicone-Property-Team": "equity-research",
        "Helicone-Property-App": "filing-summarizer",
        "Helicone-Property-Env": "production"
    }
)

Helicone-prefixed properties appear as filterable dimensions in the Helicone dashboard.

Per-Request Cost Attribution

Helicone receives the full request/response pair and calculates its own cost from token counts using its pricing table, independent of LiteLLM’s response_cost. This creates a dual-source cost figure:

Source Where How Calculated
LiteLLM_SpendLogs.spend LiteLLM DB LiteLLM model pricing map
Helicone request cost Helicone cloud Helicone pricing table

The two numbers may diverge when: (a) model pricing tables are out of sync, (b) Bedrock cross-region inference applies regional multipliers, or © custom pricing is configured in one system but not the other. For financial reporting, pick one source and stick to it. LiteLLM’s SpendLogs is the closer-to-provider source; Helicone’s dashboard adds team/app filtering on top.

Comparison: LiteLLM SpendLogs vs Helicone

Capability LiteLLM SpendLogs Helicone
Budget enforcement (reject over-limit requests) Yes No (observability only)
Per-team cost attribution Yes (team_id) Yes (custom properties)
Per-user cost attribution Yes (user param) Yes (user_id)
Custom tag filtering Yes (request_tags) Yes (Properties)
Session/conversation grouping Via session_id metadata Via session_id header
Multi-provider support All 100+ providers Limited without gateway
Self-hostable Yes (open source) Yes (open source)
Dashboard Admin UI Helicone dashboard

4. OpenTelemetry / OpenInference Integration

OTEL Span Hierarchy

Every LiteLLM Proxy request produces a span tree:

Received Proxy Server Request  [SpanKind.SERVER, root]
├── auth                        [infrastructure]
├── router                      [infrastructure]
├── guardrail/<name>            [one per guardrail]
├── litellm_request             [opt-in, USE_OTEL_LITELLM_REQUEST_SPAN=true]
│   └── raw_gen_ai_request      [provider-native payload]
└── redis / postgres            [infrastructure, if enabled]

By default (v1.81+), all gen_ai.* attributes land on the root Received Proxy Server Request span rather than a child span. This is intentional — it reduces span count and puts cost data where APM tools expect it.

Cost Attributes on Spans

gen_ai.cost.input_cost          # USD for prompt tokens
gen_ai.cost.output_cost         # USD for completion tokens
gen_ai.cost.total_cost          # USD total
gen_ai.cost.tool_usage_cost     # USD for built-in tool calls
gen_ai.cost.discount_percent    # Applied discount
gen_ai.cost.discount_amount     # Discount in USD
gen_ai.cost.margin_*            # Margin components (enterprise)

gen_ai.usage.input_tokens
gen_ai.usage.output_tokens
gen_ai.system                   # Provider (openai, anthropic, bedrock, etc.)
gen_ai.request.model
gen_ai.operation.name           # acompletion, embedding, etc.
metadata.user_api_key_team_id   # Team attribution

OTEL Setup (SDK)

from litellm.integrations.opentelemetry import OpenTelemetry, OpenTelemetryConfig
import litellm

# Single backend
litellm.callbacks = [OpenTelemetry(config=OpenTelemetryConfig(
    exporter="otlp_http",
    endpoint="https://your-otel-collector:4318/v1/traces"
))]

# Dual backend (Langfuse + internal collector)
primary = OpenTelemetry(config=OpenTelemetryConfig(
    exporter="otlp_http",
    endpoint="https://us.cloud.langfuse.com/api/public/otel",
    # Headers set via LANGFUSE_PUBLIC_KEY / SECRET env vars
))
secondary = OpenTelemetry(config=OpenTelemetryConfig(
    exporter="otlp_http",
    endpoint="https://internal-collector.corp/v1/traces",
    skip_set_global=True  # CRITICAL: prevents trace context collision
))
litellm.callbacks = [primary, secondary]

OTEL v2 (opt-in, LiteLLM Proxy)

LITELLM_OTEL_V2=true  # Enable in proxy environment

OTEL v2 produces one trace per proxy request encompassing the full lifecycle: HTTP receipt → auth → guardrails → LLM call → DB writes. It follows the official GenAI semantic conventions and ships with presets for Arize, Phoenix, Langfuse, and Weave.

Content Capture Control

# Options: NO_CONTENT | SPAN_ONLY | EVENT_ONLY | SPAN_AND_EVENT
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=NO_CONTENT  # PII-safe default

Per-handler override via OpenTelemetryConfig.capture_message_content allows different content policies for compliance vs debug backends simultaneously.

5. Datadog LLM Observability Integration

Two Integration Modes

Mode 1 — LLM Observability SDK (automatic)

When the Datadog LLM Observability SDK is initialized in the same process as LiteLLM, it instruments LiteLLM automatically with no additional callback registration. The SDK maps each LiteLLM call to a Datadog LLM span.

Mode 2 — Prometheus scrape (proxy)

LiteLLM Proxy exposes /metrics with Prometheus-formatted metrics. The Datadog Agent scrapes this endpoint.

# Datadog Agent check configuration
init_config:
instances:
  - openmetrics_endpoint: http://litellm-proxy:4000/metrics
    # headers:
    #   Authorization: Bearer sk-1234

Kubernetes annotation equivalent:

ad.datadoghq.com/litellm.checks: |
  {
    "litellm": {
      "instances": [{"openmetrics_endpoint": "http://%%host%%:4000/metrics"}]
    }
  }

Minimum Datadog Agent version: 7.68.0

Cost Metrics in Datadog

The following metrics are available after scraping:

Metric Description
litellm.spend.metric.count Total spend over the time period (USD)
litellm.provider.remaining_budget.metric Remaining provider budget
litellm.api.key.max_budget.metric Max budget per API key
litellm.team.max_budget.metric Max budget per team
litellm_total_tokens Token throughput
litellm_input_tokens Input token counts
litellm_output_tokens Output token counts
litellm_requests_total Request volume
litellm_request_duration_seconds Latency histogram
litellm_cache_hits_total Cache performance

Dashboard Setup

Use the official Grafana dashboard ID 19253 as a template when self-hosting Prometheus + Grafana. For Datadog, the LiteLLM Prometheus integration auto-creates a default dashboard. Key panels to add manually for cost visibility:

  1. Cost by model — sum by (model) (rate(litellm.spend.metric.count[5m]))
  2. Cost by team — requires team-tagged requests (metadata.user_api_key_team_id)
  3. Cost anomaly detector — Datadog’s ML-based anomaly monitor on litellm.spend.metric.count
  4. Cache hit rate — litellm_cache_hits_total / (litellm_cache_hits_total + litellm_cache_misses_total)

6. Arize Phoenix: Cost + Quality Correlation

Architecture

Arize Phoenix does not act as a gateway. Teams pair it with LiteLLM via OpenTelemetry: LiteLLM emits OTEL spans (including gen_ai.cost.* attributes), and Phoenix ingests those spans to correlate with evaluation scores.

import phoenix as px
from phoenix.otel import register
from litellm.integrations.opentelemetry import OpenTelemetry, OpenTelemetryConfig

# Register Phoenix OTEL endpoint
tracer_provider = register(
    project_name="finance-agent",
    endpoint="http://phoenix:6006/v1/traces"
)

# Wire LiteLLM to send to Phoenix
litellm.callbacks = [OpenTelemetry(config=OpenTelemetryConfig(
    exporter="otlp_http",
    endpoint="http://phoenix:6006/v1/traces"
))]

Evaluation Setup

from phoenix.evals import llm_classify, OpenAIModel, LiteLLMModel

# Use a cheap judge model to evaluate expensive production outputs
judge = LiteLLMModel(model="gpt-4o-mini", temperature=0.0)

# Run evaluations on traced spans
eval_results = llm_classify(
    dataframe=traced_spans_df,
    template=HALLUCINATION_EVAL_TEMPLATE,
    model=judge,
    rails=["factual", "hallucinated"]
)

Cost-Quality Correlation Pattern

The key pattern: production traces carry gen_ai.cost.total_cost from LiteLLM. After evaluation runs, Phoenix links eval scores back to the same span IDs. This enables queries like:

  • “What is the average cost per correct answer for each model?”
  • “Which prompts produce factual output at the lowest cost?”
  • “At what cost threshold does quality degrade for gpt-4o-mini vs gpt-4o?”
# Example analysis after eval run
import pandas as pd

# spans_df: from Phoenix trace export (includes gen_ai.cost.total_cost)
# evals_df: from phoenix.evals output (includes score per span)
merged = spans_df.merge(evals_df, on="span_id")

cost_quality = merged.groupby("model").agg(
    avg_cost=("gen_ai.cost.total_cost", "mean"),
    quality_rate=("score", lambda x: (x == "factual").mean()),
    p95_cost=("gen_ai.cost.total_cost", lambda x: x.quantile(0.95))
)

# Cost efficiency ratio: quality points per dollar
cost_quality["efficiency"] = cost_quality["quality_rate"] / cost_quality["avg_cost"]

This ratio surfaces the Pareto frontier of cost vs quality across model options — the core input to model selection decisions at scale.

7. Custom Callback for Cost Attribution

Minimal Cost-Emitting Callback

from litellm.integrations.custom_logger import CustomLogger
import litellm
import httpx

class CostAttributionCallback(CustomLogger):
    """
    Emits per-request cost to an internal cost ledger API.
    Tags by team, feature, and environment from request metadata.
    """

    def __init__(self, ledger_url: str, api_key: str):
        self.ledger_url = ledger_url
        self.api_key = api_key
        self.client = httpx.AsyncClient()

    async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
        slp = kwargs.get("standard_logging_object")
        if not slp:
            return

        metadata = slp.get("metadata", {})

        payload = {
            "trace_id": slp.get("trace_id"),
            "request_id": slp.get("id"),
            "model": slp.get("model"),
            "provider": slp.get("custom_llm_provider"),
            "cost_usd": slp.get("response_cost", 0.0),
            "cost_breakdown": {
                "input": slp.get("cost_breakdown", {}).get("input_cost", 0.0),
                "output": slp.get("cost_breakdown", {}).get("output_cost", 0.0),
                "tools": slp.get("cost_breakdown", {}).get("tool_usage_cost", 0.0),
            },
            "tokens": {
                "prompt": slp.get("prompt_tokens"),
                "completion": slp.get("completion_tokens"),
            },
            "team_id": metadata.get("user_api_key_team_id"),
            "tags": slp.get("request_tags", []),
            "cache_hit": slp.get("cache_hit", False),
            "latency_ms": (slp.get("endTime", 0) - slp.get("startTime", 0)) * 1000,
            "timestamp": slp.get("endTime"),
        }

        try:
            await self.client.post(
                f"{self.ledger_url}/v1/cost-events",
                json=payload,
                headers={"Authorization": f"Bearer {self.api_key}"},
                timeout=5.0
            )
        except Exception as e:
            # Log but never raise — callback failures must not fail the request
            print(f"[CostAttributionCallback] Failed to emit: {e}")

    async def async_log_failure_event(self, kwargs, response_obj, start_time, end_time):
        # Still emit on failures — zero cost but track error rate by team
        slp = kwargs.get("standard_logging_object", {})
        metadata = slp.get("metadata", {}) if slp else {}

        payload = {
            "trace_id": slp.get("trace_id") if slp else None,
            "cost_usd": 0.0,
            "error_class": kwargs.get("exception_type"),
            "team_id": metadata.get("user_api_key_team_id"),
            "timestamp": end_time.timestamp() if hasattr(end_time, "timestamp") else end_time,
        }

        try:
            await self.client.post(
                f"{self.ledger_url}/v1/cost-events",
                json=payload,
                headers={"Authorization": f"Bearer {self.api_key}"},
                timeout=5.0
            )
        except Exception:
            pass


# Registration
cost_callback = CostAttributionCallback(
    ledger_url="https://internal-finops.corp",
    api_key=os.environ["FINOPS_API_KEY"]
)
litellm.callbacks = ["langfuse_otel", cost_callback]

Proxy Registration (config.yaml)

Custom callbacks must be loaded as a module path when using the proxy:

litellm_settings:
  callbacks:
    - langfuse_otel
    - custom_callback_module.CostAttributionCallback

environment_variables:
  FINOPS_API_KEY: "..."

8. Trace ID Propagation

Parent Context Resolution Order

When a request arrives at the LiteLLM Proxy, the OTEL handler resolves parent context in this order:

  1. litellm_parent_otel_span in request metadata — explicit parent span object
  2. Inbound traceparent HTTP header (W3C Trace Context standard)
  3. Active span in OTEL global context (thread-local, useful in SDK mode)
  4. None — LiteLLM span becomes the trace root

Injecting Your Business trace_id

The traceparent header approach wires LiteLLM’s spans into your existing distributed trace:

from opentelemetry.propagate import inject
from opentelemetry import trace
import httpx

# Your application already has an active OTEL span for this business transaction
with tracer.start_as_current_span("process-loan-application") as span:
    span.set_attribute("loan.application_id", application_id)

    # Inject W3C traceparent into headers
    headers = {}
    inject(headers)  # headers now contains "traceparent: 00-<trace_id>-<span_id>-01"

    # LiteLLM Proxy call — LiteLLM will become a child span
    response = httpx.post(
        "http://litellm-proxy:4000/v1/chat/completions",
        headers={**headers, "Authorization": f"Bearer {litellm_key}"},
        json={
            "model": "gpt-4o",
            "messages": messages,
            "metadata": {
                "generation_name": "underwriting-decision",
                "trace_id": str(application_id),  # Also stored in SpendLogs
                "tags": ["underwriting", "prod"]
            }
        }
    )

In your trace backend, the LiteLLM span (carrying cost data) appears as a child of process-loan-application. The trace_id in metadata independently appears in LiteLLM_SpendLogs.request_tags for FinOps queries.

W3C Baggage for Downstream Propagation (v1.89.0+)

LiteLLM 1.89.0 added selective promotion of team_metadata sub-keys into W3C Baggage headers:

# Proxy config: which team_metadata keys to propagate downstream
OTEL_BAGGAGE_TEAM_METADATA_KEYS="department,cost_center,project_code"

A team_metadata.cost_center = "CC-4321" set on the LiteLLM team now propagates automatically to downstream services that use the same trace context, without requiring custom propagation code in each service.

Disabling Propagation

OTEL_IGNORE_CONTEXT_PROPAGATION=true  # LiteLLM always creates a new root span

Use this when LiteLLM is behind a load balancer that injects traceparent headers that are not meaningful to your traces.

9. Cost-Quality Correlation

The Pattern

Cost-quality correlation connects three data streams:

  1. LiteLLM SpendLogs — cost and tokens per request, with trace_id / request_tags
  2. Evaluation scores — factuality, relevance, groundedness (from Arize Phoenix, Langfuse evals, or custom LLM-as-judge)
  3. Business outcomes — conversion, resolution, task completion (from your application DB)

All three are joined on trace_id.

Correlation Schema

-- Cost-quality view (DuckDB / BigQuery / Postgres)
SELECT
    sl.trace_id,
    sl.model,
    sl.spend                           AS cost_usd,
    sl.prompt_tokens + sl.completion_tokens AS total_tokens,
    e.factuality_score,
    e.relevance_score,
    b.task_completed,
    
    -- Cost efficiency: quality per dollar
    e.factuality_score / NULLIF(sl.spend, 0) AS quality_per_dollar,
    
    -- Token efficiency: quality per 1K tokens
    e.factuality_score / NULLIF(sl.prompt_tokens + sl.completion_tokens, 0) * 1000 
        AS quality_per_1k_tokens

FROM litellm_spend_logs sl
JOIN eval_scores e ON e.trace_id = sl.trace_id
JOIN business_outcomes b ON b.request_id = sl.id
WHERE sl.startTime > NOW() - INTERVAL '7 days'

Practical Decision Rules

Finding Action
gpt-4o-mini quality_per_dollar > gpt-4o quality_per_dollar Route to mini for this task type
Quality degrades below threshold at cost < $0.001/request Mini has a failure mode at short prompts; add fallback
Cache hit rate > 30% on a specific generation_name Increase cache TTL for that prompt template
Tool calls account for >40% of total cost Audit tool invocation logic for unnecessary calls

Cost Floor vs Quality Ceiling

The critical insight from Phoenix-based analysis: most LLM tasks have a cost floor below which quality collapses and a quality ceiling above which additional spend produces no measurable improvement. The optimal operating point is typically 20–40% above the cost floor. Running A/B tests with different models at the same budget reveals this curve.

10. Production Observability Stack Recommendations

Tier 1 — Startup / Single Team (< 1M requests/month)

Stack: LiteLLM Proxy + Langfuse Cloud + built-in SpendLogs

Component Role
LiteLLM Proxy Gateway, budget enforcement, SpendLogs
Langfuse Cloud (Free/Hobby) Trace explorer, cost per trace, session grouping
LiteLLM Admin UI Spend dashboard, key management

Config:

litellm_settings:
  callbacks: ["langfuse_otel"]
  store_prompts_in_spend_logs: false  # Avoid issue #15641 in v1.77.x

Cost: ~$0 (Langfuse free tier + LiteLLM open source)

Tier 2 — Mid-Scale / Multi-Team (1M–50M requests/month)

Stack: LiteLLM Proxy + Langfuse Cloud Pro + Prometheus + Grafana Cloud

Component Role
LiteLLM Proxy (HA, 2+ replicas) Gateway, multi-team budgets
Langfuse Cloud Pro Trace-level cost, eval integration, team filtering
Prometheus + Grafana Cloud Operational metrics, alerting, cost anomaly detection
S3/GCS callback Full payload archive for FinOps replay

Config:

litellm_settings:
  callbacks:
    - langfuse_otel
    - prometheus
    - s3                         # Cost archive

litellm_settings:
  success_callback_timeouts: 10   # Don't let slow Langfuse writes slow requests

Key Grafana alerts:

  • Error rate > 5% for 5 minutes
  • p99 latency > 30s
  • Daily spend > 110% of 7-day rolling average (anomaly detection)
  • Cache hit rate < 20% (caching regression)

Cost: $200–$800/month (Langfuse Pro + Grafana Cloud)

Tier 3 — Enterprise (50M+ requests/month, regulated)

Stack: LiteLLM Proxy (HA) + Datadog LLM Observability + Arize Phoenix + S3 archive + custom FinOps callback

Component Role
LiteLLM Proxy (HA, autoscaled) Gateway, RBAC, budget enforcement
Datadog LLM Observability APM integration, alerting, compliance dashboards
Arize Phoenix Cost-quality correlation, eval pipelines, model selection
S3 + Athena / BigQuery FinOps chargeback, cost replay, audit trail
Custom CostAttributionCallback Real-time cost ledger for internal chargeback

Config:

litellm_settings:
  callbacks:
    - datadog
    - otel             # → Arize Phoenix
    - s3               # → FinOps archive

environment_variables:
  LITELLM_OTEL_V2: "true"
  DD_API_KEY: "..."
  OTEL_EXPORTER: "otlp_http"
  OTEL_ENDPOINT: "https://phoenix.corp/v1/traces"

Cost: $2,000–$10,000+/month depending on Datadog tier and Phoenix hosting

Stack Comparison Matrix

Dimension Langfuse Helicone Datadog Arize Phoenix Prometheus
Cost per trace capture Free–low Free–low High (per span) Medium Free
Budget enforcement No No No (observability only) No No
Eval score correlation Yes (native) No No Yes (native) No
Self-hostable Yes Yes No Yes Yes
Multi-team filtering Yes Yes Yes Yes Via labels
Real-time alerting Basic Basic Enterprise Basic Yes (via Grafana)
APM integration Via OTEL No Native Via OTEL Via OTEL
FinOps export (FOCUS) No No No No No (LiteLLM native)

Budget enforcement lives only in LiteLLM itself. No observability backend can reject requests over budget — they can only report after the fact.

11. Known Issues and Production Gotchas

Bedrock-Specific Issues

Bedrock prompt caching cost inaccuracy (Issue #15353, v1.75.0)

When using AWS Bedrock with Claude Sonnet 4.5 and prompt caching enabled, cost calculations diverge on cache-hit calls. The first call (cache creation) is priced correctly. Subsequent calls that hit the cache report incorrect costs in LiteLLM’s response_cost field — and therefore in every callback that consumes it (Langfuse, Datadog, custom callbacks). AWS Bedrock’s own usage logs show correct token counts but the LiteLLM cost map misapplies the cache-hit rate.

Workaround: Cross-reference with AWS Cost Explorer for Bedrock cache-heavy workloads. Or compute cost from raw token fields using AWS’s published pricing, bypassing response_cost.

Bedrock cross-region inference profile cost failure (PR #14566)

Cross-region inference profiles (e.g., us.anthropic.claude-3-5-sonnet-20241022-v2:0) were not in LiteLLM’s model pricing map, causing cost callbacks to emit $0.00 or throw unmapped model errors. Fixed in the PR but may resurface for new model variants.

Bedrock unknown region → $0.00 cost (Issue #9306)

When AWS_REGION is not set or the region is unrecognized, streaming requests report zero cost while non-streaming requests are tracked correctly. The root cause is region-based pricing lookup failing silently.

Bedrock passthrough routes (PR #12123, merged June 2025)

/invoke and /converse passthrough routes were not tracked in SpendLogs at all before this PR. If you’re using passthrough routes to avoid LiteLLM’s request transformation, verify your LiteLLM version is post-June 2025.

Callback System-Wide Issues

websearch_interception callback breaks spend tracking (Issue #20179, Jan 2026)

When the websearch_interception built-in callback is enabled, any request containing a tool named “WebSearch” completes successfully but records no entry in LiteLLM_SpendLogs. The callback intercepts the pipeline before cost logging. No workaround available as of the filing date.

Streaming /v1/messages?beta=true SpendLogs gap (Issue #23150, Mar 2026)

Anthropic’s extended output beta endpoint (/v1/messages?beta=true) with streaming enabled never triggers async_success_handler. Result: no SpendLogs entry, no Prometheus metric increment, no cost data in any callback. This affects Claude models accessed via LiteLLM Proxy when using the beta extended-output feature.

store_prompts_in_spend_logs stores empty JSON (Issue #15641, v1.77.2)

Setting store_prompts_in_spend_logs: true in config.yaml results in all rows having messages: {} and response: {}. The feature broke silently in v1.77.2. Avoid this setting for prompt archival; use the S3/GCS callback instead.

Callback exception isolation

If a callback raises an unhandled exception, LiteLLM logs it and continues. The request still succeeds but the callback event is silently dropped. There is no retry mechanism for failed callback invocations. For cost-critical callbacks, wrap all logic in try/except and emit to a fallback (e.g., write to local file / dead-letter queue) rather than letting failures disappear.

Missing SpendLogs on complete request failure (Issue #18510)

When a request fails with no successful fallback, whether a SpendLog entry is created depends on how the failure occurs. Failures before the LLM call (auth failure, rate limit from router) may not record any spend. Budget-based rate limits (LiteLLM rejecting an over-budget key) do create an entry. This means error-rate analysis and cost analysis use different denominators.

OTEL-Specific Gotchas

Dual handler trace context collision

Running two OTEL handlers without skip_set_global=True on the second causes both handlers to compete for the global tracer provider. Both will see traces but span parent relationships may be corrupted. Always set skip_set_global=True on all handlers after the first.

OTEL v1 vs v2 span placement

OTEL v1 (default): cost attributes on root Received Proxy Server Request span. OTEL v2 (LITELLM_OTEL_V2=true): cost attributes on the raw_gen_ai_request child span. If you’re building Datadog monitors or Grafana panels that query specific span names, test which version you’re running — the attribute location differs.

Sources