← Agent Frameworks 🕐 4 min read
Agent Frameworks

OSS and Independent Agent Observability/Eval Platforms 2026

Brandon Sneider · June 2026

> Primary sources are vendor docs, open-source repos, and current product docs

See also (wiki): wiki/semantic-observability.md · wiki/agent-operations-observability.md · wiki/mlops-ai-platform-engineering.md · wiki/agentic-ai-governance.md

Source credibility: MEDIUM-HIGH. Primary sources are vendor docs, open-source repos, and current product docs for Langfuse, W&B Weave, Braintrust, Helicone, OpenLIT, OpenLLMetry, DeepEval, Ragas, TruLens, and Promptfoo. Capability claims are docs-verified, not benchmark-verified. Treat pricing, hosted-retention, and maintenance posture as drift-prone.

Executive Summary

The missing player layer in the agent observability standard is not one more trace viewer. It is the independent trace-to-eval ecosystem: tools that ingest or create traces, attach scores, build datasets, manage prompts, and make human review possible. The durable standard should treat these products as workbench views over a raw vault and warehouse, not as the sole system of record.

The strongest current pattern is:

  1. preserve native raw logs and local agent transcripts;
  2. emit OpenTelemetry/OpenInference-style spans for app causality;
  3. send a view to one or more workbenches such as Langfuse, Weave, Braintrust, Phoenix/Arize, LangSmith, or Helicone;
  4. create curated datasets, eval rows, and training candidates outside the trace UI.

Platform Contract Matrix

Platform Best role Trace contract Eval/dataset contract Main gap
Langfuse OSS/self-hostable LLM engineering workbench OTel-native SDKs and OTLP ingestion; traces include LLM calls, tool invocations, retrieval steps, and observations Prompts, datasets, evals, experiments, annotation, playground Need per-version export and cost/retention verification; full local desktop-agent replay still requires native raw logs
W&B Weave ML experiment + LLM trace/eval workspace Calls are span-like; OTel ingestion maps GenAI, OpenInference, Vercel AI SDK, MLflow, and Traceloop attributes; thread grouping supported Online evals, experiments, model monitoring, call review Better for instrumented apps than raw local CLI replay; exact export shape needs test
Braintrust Eval-first trace and scorer workbench Trace hierarchy, timeline, conversation view, raw trace data, llm, tool, score, task, and eval span types Datasets, experiments, scorers, human review, Loop improvement suggestions Strong scorer/dataset loop, but cross-system raw transcript joins still need warehouse keys
Helicone Gateway/request observability and cost lens Gateway logs request/response, latency, cost, sessions/workflows, routing, prompts Evals API and prompt/version workflows Request gateway is not a full multi-step local-agent trajectory unless joined to app spans
OpenLIT OpenTelemetry-native auto-instrumentation SDK/CLI emits LLM traces, metrics, events, prompt/response capture, DB parameters, system metrics Evaluation export options and Prompt Hub/Vault features Great instrumentation layer; downstream semantic normalization still needed
OpenLLMetry / Traceloop OTel instrumentation convention Emits OTel traces with LLM/framework attributes; Weave and Phoenix can map conventions Instrumentation first, evals through downstream backends Convention drift and tool-call ID consistency need testing
DeepEval CI/eval harness Trace storage is not its core job LLM/RAG evaluation, regression checks, scorers Feed evaluation rows; do not use as trace vault
Ragas RAG evaluation harness Not a trace vault RAG metrics and dataset/eval workflows Needs trace/retrieval lineage from warehouse
TruLens RAG/LLM observability and eval App-level traces and feedback functions Feedback/evaluation for RAG/LLM apps Fit depends on app framework and export path
Promptfoo Prompt/model test runner Test-run artifacts, not production trace trees Prompt evals, red teaming, CI regression suites Excellent golden/canary testing; not production raw event storage

What This Adds to the Standard

These tools clarify the middle layer between raw logs and model training:

Standard table Tool contribution
agent_spans Langfuse, Weave, Braintrust, OpenLIT, OpenLLMetry, Phoenix, LangSmith
tool_calls Braintrust tool spans, Langfuse observations, Weave mapped tool-call attributes, OpenLIT GenAI/tool spans
evaluations Braintrust scorers, Weave online evals, Langfuse evals/annotations, DeepEval, Ragas, TruLens, Promptfoo
training_candidates Langfuse datasets, Braintrust datasets/experiments, Weave evaluations, Promptfoo regressions
artifacts Prompt versions, datasets, trace exports, score reports, eval result files

Adoption Guidance

Use one of these tools for one or two primary workflows, not every workflow:

  • Need OSS/self-hosted trace/eval/prompt management: start with Langfuse or Phoenix plus a warehouse.
  • Need ML experiment tracking and LLM traces together: evaluate W&B Weave.
  • Need eval-first scorer/dataset discipline: evaluate Braintrust.
  • Need gateway request/cost/routing visibility: evaluate Helicone.
  • Need framework auto-instrumentation: use OpenLIT or OpenLLMetry and export into the selected backend.
  • Need CI regression and red-team tests: use Promptfoo/DeepEval/Ragas/TruLens as eval producers, not trace stores.

Underdeveloped Gaps

  • Tool-call IDs are still inconsistent across UI protocols, model SDKs, framework spans, and local executors.
  • Trace UIs show what happened, but rarely explain which prompt, retrieval source, tool result, or state transition caused failure without extra evals.
  • Dataset conversion is product-specific; there is no single open schema for a curated agent episode with provenance, eval labels, artifacts, and redaction.
  • Cost controls need explicit sampling and retention policy. OTel-native capture can accidentally ingest unrelated framework traces.
  • Human review queues must preserve lineage from raw event -> trace -> scorer -> dataset example -> prompt/model change.

Source Notes

  • Langfuse docs and repo describe an open-source LLM engineering platform with tracing, prompt management, evals, experiments, datasets, annotations, and OpenTelemetry integrations.
  • Weave docs describe calls as span-like traces, thread grouping with thread_id, OTel ingestion, and mappings from GenAI/OpenInference/Vercel AI SDK/MLflow/Traceloop attributes.
  • Braintrust docs define traces, spans, span types including tool and score, raw trace views, datasets, experiments, scorers, and OTel ingestion.
  • OpenLIT docs define zero-code instrumentation, prompt/response capture, database parameter capture, events, metrics, and system metrics.

Sources

Brandon Sneider | brandon@brandonsneider.com June 2026