AI Observability and Evaluation Tooling Landscape (2026)

Audience: Platform engineers choosing observability, tracing, and evaluation tooling for enterprise LLM deployments. Covers architecture, pricing, differentiators, and a decision framework.

---

1. Landscape Overview and Category Map
2. Helicone
3. Portkey
4. Langfuse
5. Arize Phoenix
6. LangSmith
7. Braintrust and Promptfoo
8. OpenTelemetry for LLM Observability
9. W&B Weave and Fiddler (Brief)
10. Decision Framework
11. Recommended Stack Combinations
12. Verification Ledger

---

1. Landscape Overview and Category Map

The LLM observability space has fractured into four overlapping categories. The fracture matters because buying in the wrong category means paying twice or missing a capability entirely.

Category A — LLM Gateways with Observability

Tools: Helicone, Portkey, LiteLLM

What the category solves: Unified entry point for multi-provider LLM routing with observability as a side effect of sitting on the request path. The gateway sees every request and response, so it can log tokens, cost, latency, and metadata without any SDK integration in application code.

Core value: Zero-code cost attribution, caching, fallbacks, and basic tracing from a one-line base URL change.

Where it falls short: Tracing is span-level, not application-level. A gateway sees the LLM call but not the surrounding agent logic, retrieval steps, or chain of tool calls. Evaluation pipelines are absent or shallow.

2026 M&A note: Both Helicone (acquired by Mintlify, now in maintenance mode) and Portkey (acquired by Palo Alto Networks in April 2026 after open-sourcing its gateway under Apache 2.0 in March) have changed ownership. LiteLLM remains independent and actively developed.

Category B — Standalone Tracing and Observability

Tools: Langfuse, Arize Phoenix, W&B Weave, Fiddler

What the category solves: Full distributed tracing of LLM application logic — not just LLM calls but retrieval, agent loops, tool calls, and chain steps. Designed for teams that need to understand why an agent behaved a certain way, not just how much it cost.

Core value: Trace trees that span the full application. Session-level context. Human annotation queues. Production debugging.

Where it falls short: These tools add SDK weight. You instrument your code with decorators or context managers. The gateway-free path means they don’t sit on the request path and can’t enforce caching or fallbacks natively.

Category C — Evaluation Platforms

Tools: LangSmith, Braintrust, Promptfoo, Ragas

What the category solves: Systematic measurement of LLM quality — not just “did it respond” but “did it respond correctly.” Dataset management, LLM-as-judge scoring, regression detection between prompt versions, CI/CD gates.

Core value: A scored history of model/prompt changes. The ability to run a new model against a golden dataset before deploying. Human annotation workflows.

Where it falls short: Eval platforms are weak on production routing, cost attribution, and real-time alerting. They operate largely in experiment space, not the hot path.

Category D — OpenTelemetry-Native

Tools: OpenInference (Arize), OpenLLMetry / traceAI (Traceloop), OTel GenAI semantic conventions

What the category solves: Vendor-neutral trace emission using OpenTelemetry primitives. Ship LLM spans to any OTel-compatible backend — Grafana Tempo, Jaeger, Honeycomb, Datadog, New Relic — without vendor lock-in.

Core value: If you already run an OTel stack for non-LLM services, you can extend it to LLM calls without adopting a new vendor.

Where it falls short: No native evaluation layer. The OTel ecosystem does not define how to run LLM-as-judge evals or manage prompt datasets. You build that layer yourself or bolt on a Category C tool.

Category Overlap Map

                    GATEWAY LAYER
          Helicone ── Portkey ── LiteLLM
               ↓          ↓         ↓
        (routing, cache, fallback, virtual keys)
               ↓          ↓         ↓
                  TRACE TRANSPORT
          OTel OTLP ── OpenInference ── Native SDK
               ↓          ↓                ↓
                  OBSERVABILITY BACKEND
          Langfuse ── Phoenix ── W&B Weave ── LangSmith
               ↓          ↓                    ↓
                    EVALUATION LAYER
          LangSmith ── Braintrust ── Promptfoo ── Ragas

Key overlap zones:

• Langfuse covers B + partial C (tracing + evals in one tool, open-source)
• LangSmith covers B + C (tracing + evals, managed, LangChain-native)
• Phoenix covers B + C (local, open-source, OTel-emitting)
• Portkey covers A + partial B (gateway + observability, no eval)
• W&B Weave covers B + C (for teams already on W&B for ML training)

---

2. Helicone

Status (2026)

Helicone was acquired by Mintlify in March 2026 and is now in maintenance mode. No new features are being added. Existing integrations continue to work. Teams choosing Helicone for new projects should treat it as a stable but terminal choice.

Architecture

Helicone operates in two modes:

Proxy mode (default): Change one URL — e.g., https://api.openai.com/v1 → https://oai.helicone.ai/v1 — and every request passes through Helicone’s infrastructure. Helicone adds an Authorization header for its own auth and a Helicone-Auth header for routing. The proxy logs the full request and response, extracts token counts, computes cost, and applies configured behaviors before forwarding to the provider. Latency overhead is ~10–20ms.

Async logging mode: For teams that cannot proxy traffic (compliance, latency budget), Helicone provides a logging endpoint. The application calls the LLM directly and sends a side-channel log to Helicone. This loses some capabilities (no caching enforcement at the proxy layer) but retains full observability.

Features

• Cost tracking: Per-request cost computed from token counts and provider pricing tables. Supports custom cost overrides. Attributable by user ID, session, and custom properties passed as headers.
• Prompt management: Prompt versioning with template variables. Not as full-featured as LangSmith’s Prompt Hub.
• Caching: Semantic and exact-match caching. Configurable TTL. Reduces cost on repeated or near-identical prompts.
• Bedrock support: AWS Bedrock supported at $3/1M tokens gateway fee. Integration path: configure Bedrock endpoint URL as the target, pass AWS credentials as headers or via environment.
• Open-source: Helicone’s core is open-source (MIT). Self-hosting is documented but not heavily supported post-acquisition.

Pricing (as of acquisition)

Tier	Price	Limits
Free	$0	10,000 requests/month
Pro	~$20/month	500,000 requests/month
Enterprise	Custom	Unlimited, SLA, SAML SSO

Post-acquisition pricing may not be updated. Verify on helicone.ai before committing.

Differentiators vs LiteLLM

Dimension	Helicone	LiteLLM
Primary value	Observability-first proxy	Multi-provider routing
Self-hosting	Documented, not priority	First-class supported
Observability depth	Strong (built-in dashboard)	Weak out of box (integrates with external tools)
Provider coverage	~15 providers	100+ providers
Caching	Built-in	Add-on (via proxy config)
Status (2026)	Maintenance mode	Actively developed
Guardrails	Basic	Basic
Virtual keys	Yes	Yes

When to choose Helicone over LiteLLM: You want a managed proxy with a clean observability dashboard and minimal engineering overhead, you’re on a small team, and you’re not betting on it for new feature development.

---

3. Portkey

Status (2026)

Portkey open-sourced its entire production gateway under Apache 2.0 in March 2026 (processing >1 trillion tokens/day at that point). Acquired by Palo Alto Networks in April 2026. As with Helicone, the acquisition raises long-term roadmap uncertainty, but Portkey is more feature-complete than Helicone was at acquisition.

Architecture

Portkey is a full AI gateway — not just a logging proxy. It acts as a stateful intermediary that routes, transforms, enriches, and enforces policy on every LLM request. The core abstractions are:

• Virtual Keys: Abstract real provider API keys behind workspace-scoped tokens. Engineers never handle raw OpenAI/Anthropic/Bedrock keys. Virtual keys can carry per-key budget limits, rate limits, and provider fallback chains.
• Configs: Declarative routing rules attached to a virtual key. Define fallback order, retry logic, load balancing, and model aliases.
• Guardrails: Request and response filters applied in the gateway layer before and after the LLM call. Includes PII redaction, jailbreak detection, output schema enforcement, and custom regex/LLM-as-judge checks.
• Semantic Cache: Caches responses by semantic similarity of the input prompt. Configurable similarity threshold and TTL. Can reduce cost significantly on RAG pipelines where user questions rephrase the same underlying query.

Features

Feature	Details
Provider coverage	1,600+ LLMs via unified API
Load balancing	Round-robin, weighted, latency-optimized across provider/model combos
Fallbacks	Ordered fallback chains with retry budgets
Semantic cache	Similarity-threshold configurable, provider-agnostic
Guardrails	PII, jailbreak, output schema, custom LLM judge
Observability	Per-request logs, cost, latency, error breakdowns. Exportable to external backends.
Prompt management	Prompt templates versioned and deployed via gateway config
Virtual keys	Workspace-scoped, budget-limited, auditable
Bedrock	Full support — Bedrock endpoints treated as first-class providers
Compliance	HIPAA, SOC2 on Enterprise; audit trails; VPC hosting

Pricing

Tier	Price	Limits
Developer	Free	10,000 logs/month, 3-day retention
Production	$49/month	100,000 logs/month, 30-day retention; $9 per additional 100K requests
Enterprise	Custom	SSO, VPC, custom retention, HIPAA/SOC2, RBAC

Portkey vs LiteLLM vs Helicone

Dimension	Portkey	LiteLLM	Helicone
Primary value	AI gateway + LLMOps	Multi-provider routing	Observability proxy
Open-source	Apache 2.0 (March 2026)	MIT	MIT
Guardrails	Production-grade	None native	None
Semantic cache	Yes	No	Yes (exact + semantic)
Observability	Good (not primary)	Basic	Primary feature
Virtual keys	Yes	Yes	Yes
Bedrock	Yes	Yes	Yes
Status (2026)	Palo Alto Networks acquisition	Actively developed	Maintenance mode
Best for	Enterprise gateway + compliance	Self-hosted multi-provider	Simple proxy + observability

Key insight: If your primary requirement is policy enforcement at the gateway layer (PII redaction, guardrails, audit trails), Portkey is meaningfully ahead of LiteLLM and Helicone. If your primary requirement is pure routing flexibility with no vendor, LiteLLM remains the better open-source choice. Portkey’s Apache 2.0 release means you can self-host the full feature set, making the managed vs. self-hosted comparison a DevOps tradeoff rather than a features tradeoff.

---

4. Langfuse

Overview

Langfuse is the open-source LLM observability platform that has become the default for teams that want self-hosted, framework-agnostic, full tracing + evals in one tool. In January 2026, ClickHouse acquired Langfuse alongside a $400M Series D — the acquisition formalized a technical dependency that already existed in Langfuse v3, which had migrated its core data layer from PostgreSQL to ClickHouse to handle production-scale high-throughput ingestion with fast analytical reads.

License: MIT (including all product features as of June 2025 — tracing, prompts, evals, playground, annotation queues).

Architecture

Langfuse is an SDK + backend model, not a proxy. Your application code calls the Langfuse SDK to emit traces. The SDK is available in Python, TypeScript/JavaScript, and via REST. There is no traffic proxy; Langfuse sits beside the request path rather than on it.

Tracing model:

• A Trace is a full request lifecycle (one user query → final response)
• A Span is a unit of work within a trace (LLM call, retrieval, tool call, custom step)
• Spans have type annotations: LLM, RETRIEVAL, TOOL, DEFAULT
• Nested spans form a tree; the root span is the trace

from langfuse.decorators import observe, langfuse_context

@observe()  # creates a span for each call
def retrieve_context(query: str) -> list[str]:
    ...

@observe()  # creates the root trace
def answer(user_query: str) -> str:
    context = retrieve_context(user_query)
    response = openai_client.chat.completions.create(...)
    return response.choices[0].message.content

The @observe() decorator automatically captures inputs, outputs, and nests spans via Python’s contextvars. No manual span management for most cases.

Evaluation Features

Langfuse covers evaluation in three modes:

1. Human annotation: Annotation queues surface traces to human reviewers. Configurable scoring rubrics. Results stored as Scores on spans.
2. LLM-as-judge: Define a judge prompt template, attach it to a dataset or live trace sample. Langfuse runs the judge and stores the score. Built-in templates for hallucination, faithfulness, relevance, and toxicity.
3. Dataset management: Upload or curate datasets from production traces (select traces → add to dataset). Run experiments: pick a dataset, a prompt version, a model, and a scorer — Langfuse runs all combinations and tracks score deltas.

Cost Tracking

Langfuse does not sit on the request path, so cost tracking requires either:

• Passing usage metadata when logging (token counts from the LLM response)
• Using the OpenAI SDK integration which auto-captures usage
• Custom cost attributes on spans

For Bedrock specifically: Bedrock responses return token counts in the response body; the Langfuse SDK captures these automatically via the Bedrock integration or OpenAI-compatible endpoint mapping.

Bedrock Integration

Two paths:

1. OpenAI-compatible endpoint: Route Bedrock calls through LiteLLM’s OpenAI-compatible proxy, then use the standard Langfuse OpenAI SDK integration.
2. Direct SDK: Langfuse’s CallbackHandler for LangChain works natively with Bedrock-backed LangChain models. For raw Boto3 calls, manual span creation with usage metadata is required.

Prompt Management

Langfuse’s Prompt Management feature stores prompt templates with version history, variable injection, and deployment labels (production/staging). Prompts are fetched at runtime via the SDK, enabling prompt rollbacks without code deploys:

from langfuse import Langfuse
lf = Langfuse()
prompt = lf.get_prompt("my-rag-prompt", version=None, label="production")
compiled = prompt.compile(question=user_question, context=context_docs)

Pricing

Tier	Price	What you get
Free (Cloud)	$0	50K events/month
Core (Cloud)	$29/month	Higher limits, basic evals
Pro (Cloud)	$199/month	Full evals, annotation queues, longer retention
Self-hosted (MIT)	$0 license	Full feature set, no seat caps, no retention limits
Self-hosted (Enterprise)	Custom	Support SLA, SSO, audit logs

Self-hosting reality check: MIT self-hosting requires running Langfuse + ClickHouse. ClickHouse at production scale is real ops work — estimated 20–30% of a senior engineer’s time, translating to $3,000–$6,000/month in staffing cost at Series A+ salaries. Factor this into TCO comparisons with the $199/month Pro cloud tier.

Key Differentiators

• Best open-source option that combines full tracing + evals + prompt management + annotation queues in one tool
• Framework-agnostic: integrates with LangChain, LlamaIndex, OpenAI SDK, Anthropic SDK, raw HTTP, or any framework via REST
• MIT license means full product is self-hostable with no feature gating
• ClickHouse acquisition does not change pricing or licensing (confirmed as of June 2026)

---

5. Arize Phoenix

Overview

Arize Phoenix is an open-source, locally-runnable AI observability and evaluation platform. It is the reference implementation for OpenInference — the OTel-aligned semantic convention for LLM spans. Phoenix is designed to run on a developer’s laptop or a team’s private infrastructure with zero external data transmission.

import phoenix as px
px.launch_app()  # starts local UI at http://localhost:6006

No API keys, no cloud account, no vendor lock-in. Data lives in local SQLite (dev) or PostgreSQL (production deployment).

Tracing Architecture

Phoenix uses OpenInference as its trace format — a set of OTel semantic conventions that extends the base OTel GenAI spec with:

• openinference.span.kind: LLM, CHAIN, RETRIEVER, RERANKER, EMBEDDING, AGENT, TOOL
• llm.model_name, llm.input_messages, llm.output_messages, llm.token_count.*
• retrieval.documents (array of retrieved docs with content and score)
• embedding.embeddings (input text + vector)

Auto-instrumentation packages exist for: OpenAI, Anthropic, Bedrock, LangChain, LlamaIndex, DSPy, Guardrails AI, and more. One import enables tracing for the entire framework:

from openinference.instrumentation.langchain import LangChainInstrumentor
from opentelemetry import trace as otel_trace
from opentelemetry.sdk.trace import TracerProvider

provider = TracerProvider()
otel_trace.set_tracer_provider(provider)
LangChainInstrumentor().instrument()
# All LangChain calls now emit OpenInference spans to Phoenix

Evaluation Library

Phoenix ships phoenix.evals — a standalone evaluation library that operates on dataframes of spans. This is a genuine differentiator: evals are not a UI workflow but a code-first library.

from phoenix.evals import (
    HallucinationEvaluator,
    QAEvaluator,
    RelevanceEvaluator,
    run_evals,
)
import pandas as pd

# Pull traces into a DataFrame
spans_df = px.active_session().get_spans_dataframe()

# Run evaluators in parallel
eval_results = run_evals(
    dataframe=spans_df,
    evaluators=[HallucinationEvaluator(model), QAEvaluator(model)],
    provide_explanation=True,
)

Built-in evaluators: HallucinationEvaluator, QAEvaluator, RelevanceEvaluator, ToxicityEvaluator, SummarizationEvaluator, SQL_EVAL. All use LLM-as-judge internally with configurable judge models.

Embedding Visualization

Phoenix renders embedding spaces via UMAP, enabling visual clustering of:

• Input query embeddings colored by eval result (to find failure clusters)
• Retrieved document embeddings vs. query embeddings (to diagnose retrieval gaps)
• Session-level drift visualization

This is the one feature with no equivalent in Langfuse or LangSmith.

Bedrock Integration

OpenInference ships openinference-instrumentation-bedrock — auto-instruments boto3 Bedrock calls:

from openinference.instrumentation.bedrock import BedrockInstrumentor
BedrockInstrumentor().instrument()
# boto3 bedrock_runtime.invoke_model() calls now emit OpenInference spans

Phoenix vs Langfuse Decision

Criterion	Use Phoenix	Use Langfuse
Data residency required	Yes — runs fully local	Cloud or self-hosted ClickHouse
Primary concern	Evaluation research + embedding analysis	Production tracing + annotation queues
Team size	Solo / small team	Team with annotation workflow
OTel backend	Want to emit to Grafana/Honeycomb too	Langfuse-native backend
Eval approach	Code-first, dataframe-centric	UI-first, annotation queues
Prompt management	Not a Phoenix feature	First-class feature

Phoenix is the right tool for: RAG debugging, evaluation research, local development, and teams that want code-first evaluation without a managed backend. Langfuse wins for production annotation queues, prompt versioning, and team-scale observability.

---

6. LangSmith

Overview

LangSmith is LangChain’s managed observability and evaluation platform. It is not open-source. Its strongest advantage is zero-configuration tracing for LangChain and LangGraph applications — set LANGCHAIN_TRACING_V2=true and every LangChain component emits traces automatically.

Architecture

LangSmith is a managed SaaS backend. There is no self-hosted option (an enterprise on-premises version exists but is not publicly documented). The SDK wraps LangChain’s callback system; for non-LangChain code, the @traceable decorator provides similar functionality to Langfuse’s @observe().

import os
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"] = "..."

# Zero additional code — all LangChain/LangGraph calls are traced automatically
from langchain_openai import ChatOpenAI
llm = ChatOpenAI()
response = llm.invoke("What is the capital of France?")
# → trace appears in LangSmith UI

For non-LangChain code:

from langsmith import traceable

@traceable
def my_retriever(query: str) -> list[str]:
    ...

Evaluation Features

LangSmith’s eval story centers on:

• Datasets: Upload gold examples or curate from production traces
• Evaluators: LLM-as-judge, exact match, string contains, custom Python functions
• Experiments: Run a prompt + model combination against a dataset, get per-example scores and aggregate stats
• Prompt Hub: Version prompt templates, compare variants, deploy by label

LangSmith’s evaluate() function:

from langsmith.evaluation import evaluate

results = evaluate(
    my_pipeline,  # callable that takes example input
    data="my-dataset",
    evaluators=[correctness_evaluator, groundedness_evaluator],
    experiment_prefix="v2-prompt-tuning",
)

LangGraph Integration

LangSmith has deep, automatic tracing for LangGraph agent graphs — every node execution, state transition, and conditional edge is captured as a structured span. This is the most complete automatic tracing of agent graphs available in any platform as of mid-2026.

Pricing

Tier	Price	Limits
Developer (Free)	$0	5,000 traces/month, 1 seat, 14-day retention
Plus	$39/user/month	100,000 traces/month, 10 users, 400-day retention
Enterprise	Custom	Unlimited, SSO, dedicated support

Overage: $0.50 per 1,000 traces on Plus. A team running 500K traces/month pays $39 + $200 in overage per seat.

LangSmith vs Langfuse Decision

Criterion	Use LangSmith	Use Langfuse
Framework	LangChain/LangGraph	Any / multi-framework
Self-hosting required	No (no option)	Yes — strong self-hosted option
Pricing model	Per-seat + trace overage	Usage-based, free self-hosted
Eval UX	Polished, UI-first	Slightly less polished, more flexible
Provider agnosticism	LangChain-centric	Fully provider-agnostic
Prompt management	Prompt Hub, well integrated	Full prompt management
Data ownership	LangChain Cloud	Self-hostable with full data control

The practical rule: If your stack is LangChain-native, LangSmith is the path of least resistance. If you need self-hosting, multi-framework support, or want to avoid LangChain ecosystem lock-in, Langfuse is the better default.

---

7. Braintrust and Promptfoo

Braintrust

Category: Evaluation platform (managed, eval-first)

Braintrust is a managed LLM evaluation platform built around the idea that evaluation should be a software engineering workflow, not a data science one-off. Its core primitives are:

• Experiments: Named, versioned eval runs. Compare experiment A vs experiment B on the same dataset. Scores and diffs are first-class.
• Datasets: Versioned collections of (input, expected output) pairs. Non-technical team members can contribute examples via UI.
• Scorers: LLM-as-judge, exact match, regex, embedding similarity, custom Python/JS functions. Braintrust’s “Loop” feature automatically generates better-performing prompt variants by analyzing eval results.
• CI/CD gates: Every PR can trigger an eval run. Results post as PR comments showing which test cases improved, which regressed, and by how much. Quality gates block merges if score drops below threshold.
• Tracing: Braintrust added production tracing, but it’s secondary to the eval story.

Pricing: Not publicly listed; contact-sales model. Free tier available for individuals.

When Braintrust wins over LangSmith:

• You want eval as the primary product, not observability
• You want CI/CD eval gates with automated regression detection
• Your team includes non-engineers who need to contribute test cases via UI
• You want prompt optimization automation (Loop feature)

Promptfoo

Category: Open-source evaluation CLI (developer-driven, CI-native)

2026 status: OpenAI acquired Promptfoo in March 2026. Future of the open-source project under OpenAI ownership is not yet clarified.

Promptfoo is a CLI tool and library for evaluating, comparing, and red-teaming LLM applications. Configuration is YAML-first:

# promptfooconfig.yaml
providers:
  - id: openai:gpt-4o
  - id: anthropic:claude-opus-4

prompts:
  - "Summarize the following text: {{text}}"
  - file://prompts/summary_v2.txt

tests:
  - vars:
      text: "The Fed raised rates by 25bps..."
    assert:
      - type: contains
        value: "Federal Reserve"
      - type: llm-rubric
        value: "The summary should be factually accurate and under 50 words"

  - vars:
      text: "file://test_cases/earnings_call.txt"
    assert:
      - type: not-contains
        value: "I don't know"

Run with: promptfoo eval → generates a comparison table with pass/fail rates across providers and prompts.

Red teaming: promptfoo redteam generate creates adversarial inputs targeting 50+ vulnerability types (prompt injection, PII extraction, jailbreaks, hallucination triggers). The red team command reads your prompts and generates stress-test cases automatically.

When Promptfoo wins:

• You want eval integrated into CI without adopting a managed platform
• Your team is developer-driven (comfortable with YAML + CLI)
• You need automated red teaming as part of CI
• You want MIT-licensed, self-contained eval with no backend dependency
• You’re OK with pre-acquisition Promptfoo (post-OpenAI acquisition roadmap TBD)

Braintrust vs Promptfoo vs LangSmith for Evals

Criterion	Braintrust	Promptfoo	LangSmith
Primary purpose	Eval platform	Eval CLI	Tracing + eval
Open-source	No	Yes (MIT, pre-acquisition)	No
CI/CD integration	First-class	First-class	Good
Red teaming	No	Yes (50+ vuln types)	No
Non-engineer UI	Yes	CLI-only	Yes
Tracing in production	Secondary	None	Primary
Prompt optimization	Automated (Loop)	Manual	Prompt Hub
LangChain integration	Provider-agnostic	Provider-agnostic	Tight
Self-hosted	No	Yes	No

---

8. OpenTelemetry for LLM Observability

The Standards Landscape

As of mid-2026, three overlapping semantic convention standards exist for LLM spans in OTel:

Standard	Maintainer	Namespace	Status
OTel GenAI Conventions	OpenTelemetry project	gen_ai.*	Stable in OTel 1.29+
OpenInference	Arize	openinference.*, llm.*	Widely adopted, OTel-aligned
OpenLLMetry / traceAI	Traceloop	gen_ai.* (OTel GenAI aligned)	Active

The convergence direction: OTel GenAI is the upstream standard; OpenInference is a superset that adds span kinds and attributes for retrieval, embedding, and agent metadata that OTel GenAI does not cover. The two are not in conflict — OpenInference spans are valid OTel spans; they just carry additional attributes that OTel GenAI doesn’t define. The long-term expectation is that OTel GenAI absorbs retrieval and agent attributes as the spec matures, making OpenInference less necessary.

OTel GenAI Semantic Conventions (1.29+)

Key attributes for LLM spans:

gen_ai.system              # "openai" | "anthropic" | "aws.bedrock" | etc.
gen_ai.operation.name      # "chat" | "text_completion" | "embeddings"
gen_ai.request.model       # "gpt-4o" | "claude-opus-4" | etc.
gen_ai.response.model      # model actually used (may differ from request)
gen_ai.usage.input_tokens
gen_ai.usage.output_tokens
gen_ai.usage.total_tokens
gen_ai.request.temperature
gen_ai.request.max_tokens

Message content is captured as span events (not attributes) to avoid attribute size limits:

Event: gen_ai.user.message    { content: "..." }
Event: gen_ai.assistant.message { content: "..." }

Emitting LLM Spans to Any Backend

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Point to any OTel-compatible backend
exporter = OTLPSpanExporter(endpoint="http://grafana-tempo:4317")
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)

# Instrument frameworks
from openinference.instrumentation.openai import OpenAIInstrumentor
OpenAIInstrumentor().instrument()
# All OpenAI calls now emit OTel spans to Grafana Tempo

Compatible backends: Grafana Tempo, Jaeger, Honeycomb, Datadog, New Relic, Dynatrace, OpenObserve, Signoz, Langfuse (OTLP ingestion), Phoenix.

Cost: Full OTel Stack vs Managed Tools

Approach	Monthly Cost (10M traces)	Engineering Overhead
Managed tool (LangSmith Plus)	$390+/mo (10 seats) + overages	Low
Managed tool (Langfuse Pro)	$199/mo	Low
Self-hosted Phoenix + OTel	$0 license, ~$50-200 infra	Medium
Self-hosted Grafana + Tempo + OTel	$0 license, ~$100-500 infra	High
Full OTel stack (Grafana Cloud)	~$100-300/mo usage-based	Low-medium

The case for OTel-native: If you already operate an OTel stack for non-LLM services, extending it to LLM calls via OpenInference auto-instrumentation is the lowest additional cost path. The tradeoff is that you lose purpose-built LLM UX (trace trees with message rendering, cost attribution dashboards, eval integration) and must build or bolt those on.

The case against full OTel-native: LLM-specific UX (rendered message history, cost attribution, human annotation queues, eval runs) is expensive to build on top of a generic trace backend. Unless your team already has OTel expertise and a running backend, the managed tools pay for themselves in engineering time.

---

9. W&B Weave and Fiddler (Brief)

W&B Weave

Category: B (tracing) + C (evaluation), tightly coupled to Weights & Biases ecosystem

Weave extends W&B — the industry standard for ML experiment tracking — to LLM applications. Teams already using W&B for training run tracking, artifact versioning, and model registry find Weave a natural extension.

Core features:

• @weave.op() decorator for automatic trace capture
• Cost, latency, and token tracking per call
• Evaluation framework with Weave Scorers (toxicity, PII, hallucination, coherence)
• Integration with W&B artifacts for dataset versioning
• Works alongside W&B training run tracking

When to choose Weave: Your team uses W&B for ML training and you want unified tooling across the train/eval/deploy lifecycle. Not the best standalone LLM observability tool — its strength is integration depth with the W&B platform.

Pricing: Shares W&B pricing. Free tier: 100GB storage. Team tier: $50/seat/month. Enterprise: custom.

Fiddler

Category: B (observability), ML monitoring heritage

Fiddler originated as an ML model monitoring platform for traditional ML (drift detection, bias monitoring, explainability). It has expanded to LLM monitoring — hallucination detection, toxicity scoring, custom metrics — but carries the complexity and pricing of an enterprise ML platform. Best suited for organizations that need unified ML + LLM monitoring and have existing Fiddler contracts. Not a first-choice for greenfield LLM deployments.

---

10. Decision Framework

Primary Decision Tree

Start: What is your primary need?

├── Gateway / routing / cost control
│   ├── Self-hosted required? → LiteLLM
│   ├── Need guardrails + PII redaction? → Portkey (self-hosted Apache 2.0)
│   └── Managed, simple proxy, small team? → Portkey managed (Helicone in maintenance)
│
├── Tracing / observability
│   ├── LangChain/LangGraph stack? → LangSmith
│   ├── Self-hosted required? → Langfuse
│   ├── Local dev / air-gapped? → Arize Phoenix
│   ├── Already on W&B? → W&B Weave
│   └── Need OTel-native (emit to Grafana/Honeycomb)? → OpenInference + Phoenix or Langfuse OTLP
│
└── Evaluation
    ├── LangChain stack? → LangSmith
    ├── CI/CD gates + regression detection, managed? → Braintrust
    ├── CI/CD gates + open-source + red teaming? → Promptfoo (pre-OpenAI caution)
    ├── Code-first eval research, local? → Arize Phoenix (phoenix.evals)
    └── Full tracing + eval in one open-source tool? → Langfuse

Tool Selection Matrix

Tool	Self-hosted	Open-source	Gateway	Tracing	Evals	Bedrock	LangChain-native	2026 status
LiteLLM	First-class	MIT	Yes	Basic	No	Yes	Good	Active
Helicone	Documented	MIT	Yes	Good	No	Yes	Good	Maintenance
Portkey	Apache 2.0	Apache 2.0	Yes	Good	No	Yes	Good	Palo Alto acq.
Langfuse	First-class	MIT	No	Excellent	Good	Via LiteLLM/SDK	Good	ClickHouse acq.
Phoenix	First-class	Apache 2.0	No	Excellent	Excellent	Auto-instrumented	Good	Active (Arize)
LangSmith	Enterprise only	No	No	Good	Good	Via LangChain	Best-in-class	Active
Braintrust	No	No	No	Secondary	Excellent	Via SDK	Good	Active
Promptfoo	Yes	MIT	No	No	Good	Via config	Good	OpenAI acq.
W&B Weave	No	No	No	Good	Good	Via SDK	Good	Active

Cost Comparison at Scale (100K traces/month, 10 engineers)

Tool	Monthly cost	Notes
LiteLLM self-hosted	~$50-200 infra	Engineering overhead to maintain
Portkey managed	$49 + overages ≈ $100-150	100K logs included
Langfuse cloud (Pro)	$199	All features, no seat cap
Langfuse self-hosted	~$200-500 infra + $500-1000 eng time	ClickHouse ops overhead
LangSmith Plus	$390/month (10 seats) + $0 overage at 100K	Overage kicks in above 100K
Phoenix self-hosted	~$50-100 infra	No managed option
Braintrust	Contact sales	Likely $500-2000/month at team scale

---

11. Recommended Stack Combinations

Stack 1: Open-Source, Self-Hosted, Maximum Control

LiteLLM (gateway + routing)
  → Langfuse (tracing + evals + prompt management)
    → Langfuse ClickHouse backend

Optional: Phoenix for evaluation research + embedding visualization

Best for: Teams with data residency requirements, financial/healthcare, or enterprises that cannot send traces to third parties. Requires ClickHouse + LiteLLM ops capacity.

Stack 2: Managed, LangChain-Native

LangChain / LangGraph (framework)
  → LangSmith (tracing + evals, zero-config)
  → Braintrust (CI/CD eval gates on top)

Best for: LangChain-native teams that want managed infrastructure and are willing to pay per seat. LangSmith handles observability; Braintrust handles eval regression CI gates where LangSmith’s CI integration is weaker.

Stack 3: Enterprise Gateway + Open-Source Observability

Portkey (gateway: guardrails, virtual keys, semantic cache, fallbacks)
  → Langfuse via OTLP (tracing + evals)

Portkey can emit traces to Langfuse via its observability export. This gives you enterprise-grade gateway features with open-source observability backend. Portkey’s Apache 2.0 license means you can self-host both layers.

Best for: Enterprises needing guardrails + PII redaction at the gateway layer AND full tracing with annotation queues.

Stack 4: OTel-Native, Existing Infrastructure

OpenInference auto-instrumentation (openai, anthropic, bedrock, langchain)
  → OTel Collector
    → Grafana Tempo (traces)
    → Grafana + dashboards (metrics)
  + Phoenix (local eval runs, not for production tracing)
  + Promptfoo (CI eval gates)

Best for: Platform teams already running OTel for non-LLM services who want to extend their existing stack rather than adopt a new vendor. Requires investment in LLM-specific dashboards and eval tooling.

Stack 5: Local Dev + Research

Phoenix (local, pip install arize-phoenix)
  → phoenix.evals (hallucination, QA, relevance evaluators)
  → Jupyter notebooks for embedding visualization

Promptfoo (CI eval gates on PRs)

Best for: Individual developers, ML researchers, or teams building RAG pipelines who want zero-cost, zero-vendor-dependency local observability. Graduate to Langfuse or LangSmith when team scale requires annotation workflows.

---

12. Verification Ledger

Claim	Source	Confidence
Helicone acquired by Mintlify, maintenance mode	ChatForest review, helicone.ai changelog	Official docs verified
Portkey open-sourced Apache 2.0 March 2026	PR Newswire via financialcontent.com	Official announcement
Portkey acquired by Palo Alto Networks April 2026	ChatForest review	Official docs verified
Langfuse acquired by ClickHouse January 2026	Multiple pricing teardown sources, consistent	Official docs verified
Langfuse MIT license — all features	Langfuse blog (June 2025 announcement)	Official docs verified
Langfuse Free: 50K events, Core $29, Pro $199	CheckThat.ai, CostBench, Coverge	Community pattern (verify on langfuse.com)
LangSmith Plus: $39/user/month, 100K traces	Multiple pricing sources, consistent	Official docs verified
LangSmith Free: 5,000 traces, 14-day retention	Multiple pricing sources	Official docs verified
Portkey Developer: free 10K logs, Production $49/month	TrueFoundry pricing guide, Portkey docs	Official docs verified
Promptfoo acquired by OpenAI March 9, 2026	AppSec Santa review, multiple sources	Official docs verified
OTel GenAI stable in OTel 1.29+	OpenObserve OTel guide, Datadog blog	Official docs verified
Phoenix runs local, SQLite/Postgres	Phoenix GitHub repo, Phoenix docs	Source/code verified
OpenInference BedrockInstrumentor exists	openinference PyPI, Arize docs	Official docs verified
Portkey covers 1,600+ LLMs	Portkey docs, GitHub models repo	Official docs verified
LiteLLM covers 100+ providers	LiteLLM docs	Official docs verified

Known Gaps

• Helicone exact current pricing post-Mintlify acquisition: pricing page may be stale
• Portkey post-Palo Alto acquisition product roadmap: unclear
• Promptfoo post-OpenAI acquisition: open-source future not clarified
• Braintrust pricing: not publicly listed; contact-sales only
• Fiddler LLM-specific pricing: enterprise-only, not public
• W&B Weave exact trace pricing model: complex, shared with W&B platform billing

Next Research Lanes

1. Acquisition outcomes (6-month check): How Portkey under Palo Alto Networks, Promptfoo under OpenAI, and Helicone under Mintlify have evolved. Product kills and feature migrations are the most likely outcomes.
2. OTel GenAI spec coverage gaps: Whether the OTel 1.30+ spec has absorbed retrieval and agent span kinds from OpenInference, reducing the need for OpenInference-specific auto-instrumentation.
3. Ragas deep dive: The open-source RAG evaluation library was not covered in depth here. Ragas provides metric-specific eval (context recall, context precision, faithfulness, answer relevancy) that complements Langfuse/Phoenix eval pipelines rather than replacing them.
4. Cost at 1B+ tokens/day: The decision framework above assumes moderate scale. At very high volume, self-hosted LiteLLM + Langfuse self-hosted economics change significantly; the infra cost of ClickHouse becomes negligible relative to the per-event cloud costs.
5. MLflow LLM tracing: MLflow 2.12+ added LLM tracing. For teams already using MLflow for model tracking, this is a natural extension. Not covered here because it is not yet competitive with Langfuse or LangSmith on UX.