← Findings 🕐 45 min read
Findings

Agent Observability Standardization Playbook

Brandon Sneider · June 2026

This cluster has several June 2026 appendices because the research unfolded in

Executive Summary

  • LangSmith is applicable, but it should not be the only standard. It is strong for LangChain/LangGraph traces, run trees, datasets, experiments, feedback, evaluator correction loops, and trace querying. It now accepts OpenTelemetry traces and can bulk-export run data to Parquet. That makes it useful as an application/eval workbench, not as the universal raw log vault.
  • The durable standard should be layered. Preserve native raw records, normalize operational traces to OpenTelemetry, normalize AI semantics to either OpenInference or the OTel-community GenAI semantic conventions depending on backend, and create separate curated datasets for skills, KB defects, prompt optimization, and fine-tuning. Do not treat OpenInference as a neutral superset: in 2026 this is still a standards split, and Phoenix/Arize-style OpenInference traces and plain OTel GenAI traces do not render equivalently in every backend without translation.
  • OTel GenAI must be version-pinned. Treat gen_ai.* as a moving semantic-convention family, not a timeless schema. Record the OTel semantic-convention version, instrumentation version, and whether OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental or a dual-emission/translation path was used.
  • One place is the warehouse, not one UI. The highest-use system is a local or cloud data lake with raw_events, agent_spans, turns, tool_calls, artifacts, evaluations, and training_candidates. Phoenix/Arize, LangSmith, CloudWatch, and BI/search tools should read or receive views from that ledger.
  • Memory and context are first-class telemetry. Persistent memory, memory read/write audit logs, compaction, context-window utilization, tool-output offloading, code-execution filtering, and programmatic tool calling are now core reliability and cost signals.
  • Snippets need to be safe and representative. The right article should show redacted examples of Claude Code OTel, Codex OTel/config and local JSONL/SQLite rows, Bedrock invocation logs, Strands/AgentCore spans, AG-UI events, A2A messages, OpenInference spans, ATIF trajectories, and LangSmith runs with all relevant telemetry features enabled. Do not publish raw prompts, tool bodies, file contents, emails, or cloud identifiers.
  • Vocabulary matters. A trace is a causality tree. A span is one timed operation in that tree. A turn is a user-visible exchange. A run is a platform-specific execution record. A tool call is a structured request/result pair. Confusing these terms leads directly to bad datasets.

Reading Order

This cluster has several June 2026 appendices because the research unfolded in iterations. Use this article as the canonical synthesis. Use Agent Observability Format Standard 2026 as the canonical technical appendix for current copy/paste configs, schemas, raw snippets, and vocabulary. Treat the older/narrower pages as evidence appendices:

Page Current role
How to Get Agent Logs That Actually Matter Superseded first-pass narrative
Coder Agent OTEL Observability Coder-agent framing appendix
Agent Trace Format Crosswalk 2026 Source-by-source evidence crosswalk
Local Agent Log Schema Observations 2026 Local schema-only evidence appendix
Desktop Agent Observability 2026 Desktop/local-state scope appendix
Redacted ATIF and OpenInference Conversion 2026 One local conversion experiment
Agent Telemetry Config Verification Harness 2026 Field-verification plan for copied configs

Optimized Research Prompt

Research LangSmith, OpenTelemetry, OpenInference, Phoenix/Arize, Claude Code/Cowork, Codex CLI/Desktop, Bedrock invocation logs, Bedrock AgentCore/Strands, AG-UI, A2A, Microsoft Scout, and Claude SDK trace settings for a cross-vendor agent logging standard.

For each source, find the official or code-level contract for:

  • telemetry enablement and privacy toggles;
  • native raw record shape;
  • trace/span/run/message/tool-call schema;
  • export/query APIs;
  • dataset, evaluator, prompt-optimization, and fine-tuning conversion paths;
  • fields needed for skill mining, corporate KB mining, domain YAML generation, and reliability metrics;
  • gaps where the vendor does not expose enough data.

Then synthesize:

  1. Whether LangSmith should be used, and for which purposes.
  2. Whether the organization should standardize on one format or multiple formats.
  3. The canonical warehouse schema and join keys.
  4. Query patterns for self-improving skills, hard-to-find KB questions, training/curation, YAML-generation optimization, and reliability metrics.
  5. Safe raw-format snippets with all relevant feature gates enabled.
  6. A glossary defining trace, span, run, session, turn, message, event, tool call, artifact, trajectory, eval, and training candidate.

Publish both human HTML and agent Markdown, and link the result from the existing agent log mining article.

Recommendation

Use four layers.

Layer Standard Why
Raw vault Native JSONL, SQLite extracts, CloudWatch/S3 logs, protocol event streams Keeps provenance and lets future converters recover fields the first parser missed
Trace layer OpenTelemetry trace IDs, span IDs, parent span IDs, W3C trace context Gives cross-service timing, causality, and collector compatibility
AI semantic layer OpenInference or OTel GenAI semantic-convention spans, plus LangSmith run-tree view where useful Makes LLM, tool, retriever, chain, agent, and evaluator operations queryable, but backend support differs
Memory/context layer memory spans/events, compaction events, context utilization, code-execution filtering Explains cross-session state, context rot, retry divergence, token cost, and memory poisoning
Curated learning layer Task/eval/training datasets, not raw chats Produces safe skills, KB fixes, prompt-optimization examples, and fine-tuning records

LangSmith belongs in layers three and four. It is especially useful when the system already uses LangChain or LangGraph, when human feedback and evaluator correction loops matter, and when teams need datasets/experiments rather than only traces. Phoenix/OpenInference belongs in layer three as a strong local/self-hostable trace workbench and AI semantic convention, but it is not the same contract as the OTel-community GenAI semantic conventions. Phoenix uses OpenInference as its standard display format, so plain OTel GenAI spans may need translation or can show up with weaker/unknown AI semantics. The warehouse owns joins, retention, redaction, compliance, and cross-vendor analytics.

Visual Explainers

The diagrams below are intentionally static SVGs instead of a runtime diagram library. They render in the human HTML article, survive copy/paste into the agent-readable Markdown, and keep the technical edges visible.

Four-layer agent observability and learning architecture Native logs flow into a raw vault, then OpenTelemetry traces, then OpenInference or OTel GenAI AI semantics, then curated learning datasets. Do not collapse logs, traces, and training data into one object Preserve evidence first, add causality, add AI semantics, then curate learning examples. 1. Raw Vault Native evidence Claude/Codex JSONL Bedrock invocation logs AG-UI/A2A events Screenshots, files, diffs Keep: raw_ref + sha256 Risk: secrets and PII 2. Trace Layer OpenTelemetry trace_id span_id parent_span_id timing, status, service Answers: what happened when? 3. AI Semantics OpenInference or OTel GenAI LLM, AGENT, TOOL RETRIEVER, EVALUATOR tokens, cost, model tool args/results refs Answers: which AI step failed? 4. Learning Layer Curated datasets golden sets and evals SFT / DPO / GRPO rows skills and KB defects validator-backed YAML Answers: what should improve?

Production failures and eval results feed the next curation cycle, not raw chat fine-tuning.

Figure 1. The durable architecture is layered: raw evidence, trace causality, AI semantics, then curated learning data.
Trace tree compared with turns, runs, and tool calls A user turn can contain one trace tree with agent, retriever, LLM, tool, evaluator, and artifact spans. Tool calls need IDs to pair arguments with results. Vocabulary diagram: trace vs span vs turn vs run vs tool call A trace is a causality tree. A turn is the human-visible exchange. A run is vendor-specific. A tool call is a request/result pair. turn_id = user-visible exchange span: AGENT root trace_id=abc, span_id=root span: RETRIEVER docs, scores, citations span: LLM messages, tokens, cost span: TOOL tool_call_id=t42 tool result status, error, output_ref span: EVALUATOR label, score, reason artifact YAML, file, answer, diff

LangSmith run tree, Phoenix trace tree, CloudWatch trace view, and local JSONL are different views over this shape. Rule: never mine tool reliability from final chat text alone. Pair tool request, streamed chunks, result, error, and side effects by tool_call_id.

Figure 2. Trace, turn, run, and tool-call vocabulary are not interchangeable; tool calls need explicit request/result pairing.
Trace-to-training conversion with hard gates Raw traces should pass through redaction, labeling, validation, split assignment, and late export before becoming training data. Training data is not a renamed trace Trace formats explain execution. Training formats define what tokens/actions should be learned. Raw trace messages, tools, docs timing, cost, errors Redaction gate PII, secrets, code policy + hash refs Canonical IR task_type, labels loss masks, rewards Validation schema, evaluator golden set split Late exports ChatML / ShareGPT DPO / KTO / GRPO Training libs TRL, Axolotl Unsloth, LLaMA-Factory

Hard edge: ShareGPT is a useful SFT export, but it usually drops span timing, retries, retrieval lineage, evaluator labels, and tool-call side effects.

Figure 3. The safe path is raw trace to redacted canonical IR to validated late export, not direct trace-to-ShareGPT conversion.
Where the major tools fit in the observability and learning stack Vendors and formats are grouped by native capture, cloud audit, observability workbench, warehouse orchestration, and training export roles. Competing tools are layers, not substitutes The decision is what to preserve, where to query, and when to export training rows.

Native capture Claude JSONL Codex JSONL AG-UI A2A ATIF

Cloud audit Bedrock logs CloudWatch S3 AgentCore

Trace workbench OTel OpenInference Phoenix Arize LangSmith

Curation and training Dagster HF Datasets TRL Unsloth LLaMA-Factory

Use multiple views: raw vault for proof, trace workbench for debugging, warehouse for joins, curated exports for training.

Figure 4. The tools are complementary layers. Replacing the warehouse with any single trace UI loses either provenance, joins, or training controls.

Diagram source basis: Arize describes traces as trees of spans with AI span kinds such as LLM, tool, retriever, agent, guardrail, reranker, evaluator, and embedding; OpenInference defines an AI semantic convention on top of OpenTelemetry; OpenTelemetry separately maintains GenAI semantic conventions for model, retrieval, tool, agent, and workflow spans; NVIDIA AI-Q documents Phoenix, LangSmith, Weave, OpenTelemetry Collector, and verbose logging as separate observability backends. The hand-drawn Ian Xiaohei illustration skill is useful as inspiration for approachable visual language, but it is not a precise schema diagram standard for this article.

Pipeline Update: AWS, Dagster, and Hugging Face

The latest pipeline recommendation is now split out in research/19-agent-frameworks/agent-telemetry-pipelines-huggingface-dagster-aws-2026.md. The short version:

Layer Recommended home Why
Regulated raw vault AWS S3, optionally CloudWatch for hot metadata IAM/KMS, Bedrock/AgentCore joins, long-retention audit
Curation orchestration Dagster assets over source/date/tenant partitions Redaction, conversion, eval, review, and dataset lineage
Trace semantics OTel plus either OpenInference or OTel GenAI semantics Cross-vendor causality with an explicit semantic-convention choice
Interactive workbench Phoenix/Arize, LangSmith, Langfuse, Weave, Braintrust Debugging, annotation, evals, and run exploration
ML-native datasets Hugging Face private Datasets/Buckets, then curated public/private releases Agent trace viewer, mounted storage, TRL-compatible training formats

Hugging Face materially changes the trace story: its Hub docs now say Claude Code ~/.claude/projects, Codex ~/.codex/sessions, and Pi ~/.pi/agent/sessions JSONL traces are natively supported in the Hub trace viewer. That is useful for review and sharing, but it is not safe as an unfiltered upload path because those traces can contain prompts, tool inputs, command output, local paths, screenshots, secrets, private code, and personal data.

Training remains a separate approval workflow. Raw traces become candidates; curated datasets become training material. For domain YAML generation, the best format is usually TRL-compatible conversational SFT rows plus validator-backed golden sets. For tool-call reliability, use tool-call SFT rows, preference pairs, or RLVR/GRPO environments with programmatic rewards. For KB mining, create eval/golden-set questions and corpus-fix tickets before any fine-tune.

ShareGPT changes the framing but not the core architecture. It is a convenient training-conversation export, not an observability standard. OpenInference and AWS/OTel GenAI semantics keep trace IDs, spans, timing, token/cost attributes, tool-call IDs, retrieval lineage, and evaluator spans. ShareGPT usually keeps only the conversation text. Use ShareGPT-like JSON only as a curated gold-layer SFT export, with provenance stored in metadata or a sidecar table.

There is an important observability nuance here: OpenInference and the OTel-community GenAI semantic conventions are competing/converging contracts, not one clean stack. OpenInference is OTel-compatible but uses its own required AI span-kind attribute, while OTel GenAI uses gen_ai.* operation names and attributes that are still marked developmental in current OTel docs. Phoenix standardizes on OpenInference display semantics and documents translation from OpenTelemetry GenAI/OpenLIT/OpenLLMetry into OpenInference. In practice, a “valid OTLP trace” can still be semantically weak or show up as unknown in a Phoenix-style UI unless it carries the expected OpenInference attributes.

OTel GenAI also needs version handling, not just naming. Current OpenTelemetry docs describe migration behavior for instrumentations that emitted v1.36-era GenAI conventions and introduce OTEL_SEMCONV_STABILITY_OPT_IN, including gen_ai_latest_experimental, to opt into newer experimental GenAI conventions. For production, record otel_semconv_version, ai_semantic_convention, instrumentation_name, instrumentation_version, and semconv_stability_opt_in on every semantic span. Treat model/client spans, agent/framework spans, and MCP spans as separate stability surfaces.

MCP is now a separate OTel semantic-convention layer to track, not only a source of generic tool calls. The OTel MCP semantic conventions recommend MCP-specific spans and metrics because MCP streaming messages, protocol version, method names, prompts, resources, and tool targets are not fully captured by generic RPC conventions. Tool reliability mining should therefore preserve both tool_call_id and MCP attributes such as server identity, method name, target, and protocol version.

The harder edge is loss masking and model-native serialization. A trace can contain user secrets, retrieved documents, tool observations, failed tool calls, validator output, and rejected answers. A training example decides which tokens the model should predict. For that reason the pipeline should not convert directly from traces to ShareGPT. It should convert traces to a canonical training IR with task type, messages, tool calls, labels, rewards, loss masks, and provenance, then export late to TRL, Axolotl, LLaMA-Factory, Unsloth, or a hosted API format.

Training formats differ by both library and model family: raw text for continued pretraining, Alpaca/instruction rows for simple SFT, OpenAI/ChatML-style messages, ShareGPT conversations, prompt-completion, DPO preference pairs, KTO labels, GRPO/RLVR prompt-only environment cases, stepwise supervision, and multimodal content-part formats. The same curated example may need different chat templates for Llama, Qwen, Gemma, Phi, DeepSeek, GLM, GPT-OSS, or a VLM. That is a serialization and masking problem, not just a JSON key-renaming problem.

Why Not One Format

One format loses evidence.

  • Bedrock invocation logs prove cloud model invocation but not local shell activity.
  • Claude and Codex local session logs preserve local agent behavior but not AWS account/region audit.
  • AG-UI preserves frontend stream lifecycle but not model internals.
  • A2A preserves delegation/task state but not the delegated agent’s full reasoning or local tool effects.
  • LangSmith runs are excellent app/eval records but are not guaranteed to contain every local desktop-agent event unless instrumented.
  • OpenTelemetry spans preserve timing and parentage but do not by themselves define “turn”, “trajectory”, “accepted answer”, or “training-safe example”.

The right standard is multiple compatible representations with stable join keys.

Format Gap and Tool-Call Matrix

Source Useful by default Must toggle on for useful logs What is still missing Tool-call handling
Claude Code local JSONL Local session chronology, user/assistant/tool-like records, task replay clues Keep local transcripts; optionally pair with OTel for fleet metrics Cloud request IDs, org-wide fleet view, guaranteed redaction, cross-service trace context unless joined Tool activity appears in local session records and can be joined to OTel tool events when telemetry is enabled
Claude Code OTel Metrics/logs/events; optional traces depending on config CLAUDE_CODE_ENABLE_TELEMETRY=1; OTLP endpoint/exporters; OTEL_LOG_USER_PROMPTS=1, OTEL_LOG_TOOL_DETAILS=1, OTEL_LOG_TOOL_CONTENT=1 for content-bearing experiments Local file artifacts and subprocess telemetry unless separately instrumented; raw content if gates remain off Tool details/content are gated; tool result events need tool_call_id/span correlation added downstream if the emitted schema is not enough
Claude Agent SDK App trace plus child Claude Code process linkage SDK tracing plus W3C trace context propagation; child process telemetry Local CLI transcript completeness and enterprise admin metadata unless joined Tool calls should become child spans under app/agent spans; MCP/subprocess tools need propagated trace context
Claude Cowork OTel Org/admin prompt and tool activity with prompt/session/account metadata Admin OTLP endpoint; privacy review because prompt/tool/user attributes can be sensitive Not currently a full Compliance API substitute; local desktop replay is not guaranteed Tool result events are grouped by prompt.id; inputs may include paths, URLs, patterns, and arguments
Codex CLI OTel Metrics/traces/logs when configured otel.trace_exporter, otel.metrics_exporter, OTLP endpoint/headers/protocol; otel.log_user_prompt=true for prompt content Public emitted schema detail is thinner than Claude Code; entrypoint behavior can vary Tool/function calls need call_id extraction from local sessions and span/log correlation from OTel
Codex local state Session index, JSONL sessions, SQLite logs/state, shell snapshots, skills, memories Nothing cloud-side; run local redacted extractors Not a vendor stability contract; no cloud identity; Desktop paths/behavior need validation Function/tool calls are visible as local records; arguments/results must be redacted and paired by call ID or sequence
Codex Desktop Potential app-server/local state and user/global config behavior Likely user-level ~/.codex/config.toml OTel until project config behavior is validated Exact Desktop schema, local DB paths, and project-vs-user OTel behavior Treat as unverified until sampled; expect app-server tool events plus local session records
Microsoft Scout Session history, memory, skills, shell/browser/M365/WorkIQ activity in preview docs Unknown public OTel/export toggles; enable any admin/export controls Microsoft exposes Complete local/export/admin telemetry contract is not public enough yet Tool surface is broad: shell, browser, MCP filesystem, M365, subagents; capture needs per-tool type, arguments, approvals, and sensitivity labels
Bedrock invocation logging Supported model-runtime request/response metadata and optional payloads Enable invocation logging per account/region and destination; choose CloudWatch/S3 and payload logging policy No local agent actions; no shell/browser/file/MCP/tool replay; bedrock-mantle Responses API endpoint gap Tool calls only appear if they are in model payloads; Bedrock logs do not know whether the local agent executed them
AgentCore / Strands AWS/ADOT-backed OTel spans, sessions, Bedrock/tool activity ADOT/OTel instrumentation, CloudWatch GenAI observability, session baggage such as session.id AgentCore is not its own semantic convention; local desktop actions outside the framework; raw prompt/tool bodies depending on config Tool invocations become OTel/GenAI spans or events in the agent trace; join to Bedrock invocation logs by request metadata where possible
AG-UI UI protocol lifecycle, message streaming, tool-call lifecycle, state deltas Preserve event stream IDs and attach trace/session IDs at the app boundary Model internals, local filesystem effects, cloud request IDs Tool calls are first-class UI events with tool-call IDs, names, argument/result streaming boundaries
A2A Agent-to-agent task/message/status/artifact protocol Preserve taskId, contextId, message IDs, artifact IDs; attach trace context in metadata Delegated agent’s internal trace unless that agent also exports telemetry Tool-like delegation is represented as tasks/messages/artifacts, not local tool execution
OpenTelemetry GenAI Vendor-neutral trace/span transport and GenAI attributes Instrument app/framework; propagate W3C trace context; export to collector Curation labels, turn boundaries, protocol semantics unless added Tool calls should be spans/events with stable tool_call_id, arguments/results as redacted refs
OpenInference / Phoenix AI semantic spans for LLM, tool, retriever, embedding, chain, agent-style work Use OpenInference instrumentation or span processors; translate OTel GenAI/OpenLLMetry/OpenLIT where needed Not a neutral superset of OTel GenAI; native local transcript provenance unless raw refs are preserved Tool spans have OpenInference semantic kind TOOL; input/output should be redacted or referenced
LangSmith Run trees, Messages view, datasets, feedback, evals, query/export, Parquet bulk export LANGSMITH_TRACING or OTel endpoint; LANGSMITH_OTEL_ENABLED for LangChain/LangGraph OTel path; metadata such as thread_id, ls_provider, ls_model_name Not a raw desktop-agent vault; only sees what is instrumented; sensitive data needs sampling/redaction policy Tool runs are child runs with run_type="tool"; Messages view pairs LLM tool calls and tool results when payload shape/detection rules match
ATIF Agent trajectory for replay/eval Export or convert trajectories with ordered steps, actions, observations, outcome labels Not a live collector by itself; needs conversion to OTel/OpenInference for tracing tools Tool calls are trajectory steps with action/observation pairs and terminal outcome context

The operational rule is simple: if the goal is audit, keep native raw records; if the goal is debugging, use traces/spans; if the goal is learning, require turn/task labels and validation artifacts; if the goal is tool reliability, extract every tool request/result/error into a tool_calls table.

Cost and Storage Tradeoffs

Bedrock invocation logging into CloudWatch or S3 is not “free observability.” It is a model-call audit pipeline with AWS log storage economics. The useful question is which data belongs in expensive hot logs, which belongs in cheaper object storage, and which belongs in an AI observability/eval workbench.

Option Cost drivers Benefits Hard tradeoffs
Bedrock to CloudWatch Logs Log ingestion GB, retained log storage, Logs Insights scans, optional streaming/export paths Native AWS account/Region audit, CloudWatch Logs Insights, IAM/KMS integration, near-real-time operations Can become expensive with prompt/output bodies; no local tool execution; AWS-centric query model; payloads create privacy burden
Bedrock to S3 S3 storage class, PUT/GET requests, Athena/Glue/OpenSearch/ETL scans, lifecycle management Cheaper long-term vault, gzipped JSON batches, large/binary payload support, Athena/Glue-friendly Slower operational debugging; requires partitioning/catalog/ETL; not a live trace UI
Bedrock to CloudWatch plus S3 Double destination cost surface: CloudWatch hot logs plus S3 archive Best split for regulated production: hot incident response and durable archive Must control duplication, retention, and redaction or bills and risk grow quickly
CloudWatch to Firehose/OpenSearch/warehouse CloudWatch delivery/streaming plus Firehose ingest, destination storage/indexing Near-real-time fanout to analytics/search systems Every hop adds cost and schema drift risk
Prometheus / managed Prometheus Low-cardinality metrics samples, remote write/OTLP metrics ingest, recording/alert rules Fleet health, SLOs, token/cost rates, tool failure rates, Bedrock invocation/token counters Not a raw log, trace, prompt, payload, or replay store; high-cardinality labels such as request IDs, prompts, user emails, file paths, and trace IDs must stay out
LangSmith Seats/plan, trace volume, retention tier, bulk export, enterprise/self-host terms Excellent app trace UI, LangGraph/LangChain fit, datasets, evals, feedback, trace queries Only sees instrumented data; not a Bedrock/local raw vault; SaaS data policy and trace volume economics matter
Arize/Phoenix Phoenix OSS infrastructure if self-hosted; Arize cloud spans/GB/retention plan if SaaS OpenInference/OTel-native, local/self-hostable Phoenix, strong LLM trace/eval workflow You operate storage/retention if OSS; SaaS costs scale with spans and ingestion
OSS OTel/Phoenix/Langfuse/Grafana/Loki/OpenSearch Compute, storage, index size, backups, SRE time, upgrades Maximum control, local raw retention, vendor-neutral joins Operating burden, schema ownership, and query/annotation UX are now your problem

Practical default:

  1. Send Bedrock invocation logs to S3 for durable audit, with lifecycle rules and Athena/Glue partitions.
  2. Send only sampled or metadata-heavy Bedrock logs to CloudWatch for hot debugging unless compliance requires full CloudWatch retention.
  3. Send application and agent traces through OTel/OpenInference to Phoenix/Arize/LangSmith for debugging and eval workflows.
  4. Keep prompt/tool bodies out of shared backends by default; store hashes and redacted refs, then promote selected examples into curated datasets.
  5. Use LangSmith or Arize/Phoenix for evaluation and annotation, not as the only source of truth for cloud invocation audit.

The cost trap is full-content logging at high volume. A single agent turn can create multiple model calls, retrieval calls, tool results, file diffs, and protocol events. If every prompt, completion, tool body, and retrieved document is copied into CloudWatch, a logging bill can scale faster than the model bill. CloudWatch is valuable for hot operations; S3/warehouse storage is usually the right place for long-retention raw audit; LangSmith/Arize/Phoenix are better for interactive trace analysis and eval loops.

Prometheus belongs beside this stack as the alerting and low-cardinality metrics plane, not as a replacement for CloudWatch/S3 or a trace/log backend. Prometheus can ingest OTLP metrics when its OTLP receiver is enabled, but the data should be derived metrics such as agent_model_invocations_total, agent_input_tokens_total, agent_tool_calls_total, bedrock_invocations_total, bedrock_input_tokens_total, and bedrock_output_tokens_total. Keep Bedrock requestId, prompts, completions, tool arguments, user emails, file paths, and raw ModelInvocationLog JSON in CloudWatch/S3/warehouse/log storage. Put only bounded labels such as provider, model_id, operation, region, source, workflow, and status on Prometheus series.

The newer cost trap is context waste, not only log storage. Tool manifests, large tool results, repeated memory files, stale retrievals, and long histories consume the model’s attention budget before the agent starts useful work. Instrument context engineering directly: record tokens_before, tokens_after, context_window_tokens, tool_manifest_tokens, memory_tokens, retrieval_tokens, compaction_id, and whether code execution filtered or transformed data before it reached the model. Anthropic’s code-execution-with-MCP pattern is the clearest example: execute joins, filters, loops, and data movement in code, then return only high-signal results to the model.

Canonical Schema

Minimum tables:

Table Unit Examples
raw_events One native record Claude JSONL row, Codex SQLite log row, Bedrock invocation record, AG-UI event
agent_spans One timed operation LLM call, tool call, retriever call, shell command, protocol step
turns One user-visible conversational exchange User asks, assistant/tool sequence answers
tool_calls One structured request/result pair Bash, MCP, browser, file edit, M365, retrieval
memory_events One persistent-memory operation memory read, write, version, rollback, consolidation, stale-memory cleanup
context_events One context-management action compaction, tool-result clearing, code-execution filtering, context offload
artifacts One produced or consumed object YAML output, markdown article, source citation, generated file
evaluations One quality judgment pass/fail, score, failure mode, human label, validator output
training_candidates One curated example candidate skill example, KB eval, prompt optimization row, SFT row

Required join keys:

Key Use
trace_id, span_id, parent_span_id Causality and timing
session_id, thread_id, turn_id Human conversation grouping
run_id, prompt_id, task_id Vendor and protocol grouping
tool_call_id Pair arguments, streamed chunks, result, error, and side effects
artifact_id Link outputs, files, citations, YAML, screenshots, datasets
memory_store_id, memory_version_id, memory_event_id Link persistent state reads/writes, audit logs, rollback, and poisoning review
context_event_id, compaction_id, context_window_tokens Attribute cost/reliability to compaction, offload, and context-budget decisions
cloud_request_id, aws_account_id, aws_region Cloud audit join
repo, git_commit, workspace_hash Reproducibility without broad path exposure
model_provider, model_id, prompt_version Reliability and optimization attribution
redaction_version, policy_version Training and compliance review

Query Patterns

Self-Improving Skills

Find repeated successful task trajectories:

select task_family, count(*) as successes
from training_candidates
where candidate_type = 'skill'
  and validation_status = 'passed'
  and human_approved = true
group by task_family
having count(*) >= 3;

Promote only when the traces share a trigger, action sequence, recovery pattern, and validation artifact. A skill should include counterexamples from failed traces, not just the happy path.

Hard Corporate KB Questions

Find answerable questions that were expensive to retrieve:

select turn_id, question_hash, final_source_uri, retrieval_attempts, elapsed_ms
from turns
where retrieval_attempts >= 5
  and final_source_uri is not null
  and eval_label = 'answered_with_citation';

Convert these into KB defects: missing alias, stale title, missing owner, poor access permissions, no canonical page, no citation anchor, or no answer exists.

Reliability Metrics

Measure task-level reliability, not just tokens:

select task_family,
       avg(case when validation_status = 'passed' then 1 else 0 end) as pass_rate,
       avg(retry_count) as avg_retries,
       avg(tool_error_count) as avg_tool_errors,
       avg(cost_usd) as avg_cost,
       percentile_cont(0.95) within group (order by elapsed_ms) as p95_elapsed_ms
from turns
group by task_family;

YAML Generation Optimization

Mine validator failures:

select schema_name, failure_code, count(*) as n
from evaluations
where artifact_type = 'yaml'
  and evaluator_type = 'deterministic_validator'
group by schema_name, failure_code
order by n desc;

The output should feed prompt optimization or GEPA/DSPy style instruction refinement before fine-tuning.

Memory and Context Reliability

Find failures where memory state or context management likely contributed:

select t.turn_id, t.validation_status, c.event_type,
       c.tokens_before, c.tokens_after,
       m.operation as memory_operation, m.memory_store_id
from turns t
left join context_events c on t.turn_id = c.turn_id
left join memory_events m on t.turn_id = m.turn_id
where t.validation_status = 'failed'
  and (c.event_type in ('compaction', 'tool_result_clearing', 'code_execution_filter')
       or m.operation in ('read', 'write', 'rollback'));

Track stale memory, poisoned memory, missing memory, conflicting memories, unauthorized writes, bad compactions, and over-broad tool manifests as distinct failure modes.

LangSmith Fit

LangSmith should be used when:

  • the application uses LangChain, LangGraph, or an easily instrumented custom app;
  • trace trees, Messages view, datasets, experiments, feedback, and evaluator improvement are needed in one workflow;
  • engineers need SDK/API trace queries, trace filters, tree filters, metadata filters, or bulk Parquet export;
  • human corrections should become few-shot evaluator examples;
  • filtered traces should become datasets for regression testing.

LangSmith should not be the sole system of record when:

  • local desktop-agent logs, shell history, file edits, approvals, M365 data, and protocol events need independent retention;
  • the organization needs vendor-neutral trace storage;
  • raw content must remain local;
  • Bedrock invocation logs, CloudTrail, Scout, Claude/Codex local state, or AG-UI/A2A events are part of the evidence chain.

Glossary

Term Definition
Trace A causally related tree or graph of operations for one request, task, or workflow
Span One timed operation inside a trace, with start/end time, parent, status, and attributes
Run A platform-specific execution record; in LangSmith, a root run can represent a trace and child runs are span-like
Rollout A concrete execution episode of an agent or workflow against a task, prompt, dataset row, or production request; it may produce one or more traces, turns, tool calls, artifacts, and eval labels
Session A longer-lived user/agent interaction container, often many turns or traces
Thread A conversational grouping key, often used for UI history and multi-turn retrieval
Turn One user-visible exchange: user input plus assistant/model/tool work needed to answer it
Message A role-tagged conversational item such as system, user, assistant, or tool
Event A point-in-time record, often lifecycle, streaming, status, or telemetry metadata
Tool call A structured request from model/agent to a tool plus its result, error, or side effects
Memory event A persistent-memory read, write, version, rollback, consolidation, or permission decision
Context event An explicit change to what the model sees or avoids seeing, such as compaction, tool-result clearing, context offload, or code-execution filtering
Artifact A produced or consumed object: file, YAML, citation, image, report, dataset row, or patch
Trajectory A full agent episode with actions, observations, decisions, and terminal outcome
Evaluation A deterministic, human, or model-judge quality signal linked to a trace, turn, artifact, or dataset example
Benchmark A named evaluation protocol with fixed tasks, scoring rules, runner configuration, and reporting expectations
Golden set A stable, reviewed set of examples or tasks used to detect regressions and compare prompts/models/agents over time
Experiment A controlled comparison of variants such as model, prompt, retrieval config, tool policy, or agent architecture against the same eval set
Prompt version A stable identifier for the prompt/instruction/template used in a rollout, needed to attribute changes in quality, cost, and latency
Dataset A named collection of examples with schema, version, split, provenance, and intended use
Example One dataset row: input, expected/reference output or label, metadata, and optional trace/artifact refs
Split Partition of a dataset such as train, validation, test, golden, canary, or holdout
Holdout Examples intentionally withheld from training or prompt optimization, used to detect overfitting
Canary set Small high-risk eval subset run frequently before broader rollout or production promotion
Workload A stable logical operation or agent node, such as tool routing, query rewriting, summarization, retrieval, YAML generation, or citation formatting
Lineage The provenance chain from raw event to curated example to trained adapter/model/eval result
Dedupe Exact, fuzzy, or semantic removal of duplicate examples so metrics and training are not distorted
Contamination Leakage of benchmark/golden/test examples into training, prompt selection, retrieval, or judge context
Human-in-the-loop Human review, approval, or labeling before promotion to dataset, skill, model, or production
Judge Human, deterministic validator, or model evaluator that scores an output against criteria
Reward model A model trained or used to score candidate outputs, often for preference optimization or filtering
Preference pair Two or more candidate outputs for the same input with a preference label or ranking
SFT Supervised fine-tuning on instruction/output examples
PEFT / LoRA Parameter-efficient fine-tuning; LoRA stores a small adapter rather than a full model checkpoint
Adapter Fine-tuned parameter delta mounted on a base model for inference or evaluation
Model promotion Governance step that approves a candidate model/prompt/adapter for deployment after eval and review
Training candidate A redacted, licensed, labeled, deduped example proposed for prompt optimization, SFT, preference data, or skill creation

A rollout is the practical unit for agent improvement. A trace tells you what happened; a rollout says “this exact agent/prompt/model/tool configuration was run on this task at this time and produced these artifacts and scores.” Evals attach quality labels to rollouts. Benchmarks define repeatable eval protocols. Golden sets are the smaller trusted slices you run frequently to catch regressions before a broad benchmark or production deploy.

NVIDIA Data-Curation Jargon Mapping

NVIDIA’s Data Flywheel blueprint was deprecated in April 2026, so it should not be treated as current production software. The pattern still matters as a reference architecture: production traffic becomes curated datasets, experiments, evals, adapters, and promotion decisions. For production work, map the concepts to current NeMo Microservices or your own OTel/warehouse/eval stack rather than extending the deprecated blueprint.

NVIDIA / curation term Maps to this standard Notes
Data flywheel Rollout/eval/training-candidate loop Production traffic becomes datasets, experiments, evals, and model/prompt candidates
Production traffic Raw events plus turns/tool calls Must be logged with enough request/response/tool structure to replay
Log store Raw vault / raw_events NVIDIA reference used Elasticsearch; this standard prefers native raw vault plus warehouse
workload_id workload, task_family, agent.node_id Critical key: each agent node needs its own stable workload ID
client_id application/deployment/tenant ID Multi-tenant or multi-app isolation key
Datastore Dataset/artifact registry Stores eval datasets, fine-tuning datasets, and adapter artifacts
Customizer Fine-tuning job/service Produces LoRA/adapters or customized model artifacts
Evaluator Eval runner/judge Scores base, ICL, customized, prompt, or adapter variants
Deployment Manager / DMS Runtime orchestration Starts/stops candidate model deployments for evaluation
NIM Serving runtime Where base or adapted models run; not itself a curation method
Base eval Benchmark rollout against unmodified model Establishes baseline before prompt/ICL/fine-tuning changes
ICL eval Rollout with few-shot examples in context Tests whether examples in prompt beat fine-tuning cost
Customized eval Rollout against fine-tuned adapter/model Tests SFT/LoRA impact against held-out tasks
Workload tagging Required trace/dataset metadata Without this, examples from different agent nodes get mixed and become weak training data
Class-aware stratified split Dataset split by label/tool/workload class Prevents rare tool calls or task classes from disappearing from eval/holdout sets
LLM-as-judge Model evaluator Useful but should be calibrated with golden/human/deterministic checks
Tool-calling accuracy Tool-call eval metric Usually exact function name plus exact/semantic argument matching
LoRA adapter artifact Adapter/artifact Needs lineage to base model, dataset, prompt version, eval score, and deployment
Model promotion Promotion gate Human/governed approval after eval, cost, safety, and regression review

The key translation: NVIDIA’s workload_id is the curation equivalent of an agent-node label. If every step in an agent chain logs the same workload, the flywheel cannot tell whether tool routing, retrieval, summarization, YAML generation, or citation formatting needs improvement. For our agent logging standard, workload_id, rollout_id, trace_id, turn_id, tool_call_id, prompt_version, and eval_id all need to survive the conversion.

Evals and Benchmarks for Our Domains

Anthropic’s “Demystifying evals for AI agents” is directly relevant here. The main lesson is that agent evals are not just prompt/response checks. A serious agent eval gives the agent tools, a task, and an environment; lets it run a multi-turn loop; then grades the resulting state, artifacts, tool behavior, and cost. That maps exactly to our observability standard:

Domain Eval target Golden-set shape Required logs
Self-improving skills Repeated successful task trajectories 3+ validated successes plus counterexamples per skill rollout, trace, tool calls, final artifacts, validation command, human label
Corporate KB needles Hard-to-find answers with verifiable citations Questions that require aliases, owners, old names, permissions, or multi-hop retrieval turns, retrieval spans, query rewrites, final citation artifact, elapsed time
Domain YAML generation Schema-valid and semantically correct YAML Adversarial schema cases, invalid/valid examples, deterministic validator failures prompt version, generated artifact, validator output, correction trace
Finance/source research Source acquisition and evidence quality Tasks with expected primary sources, citation anchors, and artifact checks browser/search/tool calls, raw source ledger, final article, build/search verification
Tool routing / SLM replacement Correct tool/function and arguments Stratified examples by tool/class, including rare tools tool-call request/result/error rows, workload ID, judge/deterministic score
Coding/agent reliability Environment state after multi-step work Repo fixtures with tests, files, UI/browser checks, and failure modes shell/file/browser spans, git state, test output, resource config

Practical guidance:

  1. Write small golden sets before broad benchmarks.
  2. Grade outcomes and artifacts, not just assistant text.
  3. Keep trajectory metrics: extra turns, failed tools, retries, cost, latency, and unnecessary edits.
  4. Calibrate LLM judges against deterministic checks and human review.
  5. Treat infrastructure configuration as part of the benchmark contract.
  6. Split eval suites into regression checks, quality benchmarks, and production monitoring; do not expect one suite to do all three.

What NeMo Microservices Add

The newer/current NVIDIA NeMo Microservices stack adds production services around the deprecated Data Flywheel pattern. It does not change the logging standard; it adds concrete places where curated data, evaluation, customization, deployment, and guardrails can run.

NeMo microservice concept What it adds How it maps to our standard
Entity Store / Data Store Managed entity, dataset, and artifact storage datasets, artifacts, lineage, model/adapter refs
Data Designer Synthetic/curated data generation workflows training_candidates, synthetic examples, curation labels
Customizer Fine-tuning and customization jobs SFT/LoRA jobs linked to dataset version, prompt version, base model, evals
Evaluator Benchmark/custom eval/LLM-as-judge workflows evaluations, benchmark runs, golden-set regression, judge metadata
Deployment Management Starts/stops candidate NIM deployments rollout infrastructure, model candidate runtime metadata
NIM Model serving runtime model endpoint spans, model ID, latency/cost metrics
Guardrails / Safe Synthesizer Safety checks and synthetic-data controls GUARDRAIL and EVALUATOR spans, policy version, rejection labels

So the answer is: yes, NeMo Microservices add a production path if you are already in the NVIDIA ecosystem, especially for self-hosted model customization and evaluation. They do not remove the need for OpenTelemetry/OpenInference, raw-event retention, warehouse joins, or external eval design. They are a backend for parts of the lifecycle, not the universal trace format.

OpenInference in Plain English

OpenTelemetry gives you the envelope: trace IDs, span IDs, parent/child relationships, timestamps, status, attributes, and OTLP export. OpenInference adds the AI-specific vocabulary inside that envelope.

OpenInference span kind What it means Typical model architecture mapping
LLM A model generation call Chat model, completion model, reasoning model, local SLM, Bedrock/NIM/OpenAI/Claude call
EMBEDDING Text/image/etc. converted into vectors Embedding model in RAG, memory, clustering, semantic dedupe
RETRIEVER Fetch context candidates BM25, vector DB, SQL, graph retrieval, web search, KB search
RERANKER Reorder/prune retrieved candidates Cross-encoder, ColBERT, LLM relevance classifier, NVIDIA reranker NIM
TOOL External function/tool execution Shell, browser, MCP, database, API, file edit, calculator, M365, Bedrock tool result
CHAIN Deterministic orchestration step Prompt assembly, parser, router, formatter, validator, agent graph node
AGENT Autonomous loop or agent node Claude/Codex session segment, LangGraph node, Strands agent, A2A delegate
GUARDRAIL Safety/policy check NeMo Guardrails, policy engine, content filter, permission check
EVALUATOR Quality/scoring step YAML validator, LLM judge, benchmark scorer, human label ingestion
PROMPT Prompt/template construction Prompt version, system template, few-shot assembly

This matters because different model architectures produce different span shapes:

  • Single LLM call: one LLM span under a rollout trace.
  • RAG: RETRIEVER -> RERANKER -> LLM, with document refs attached.
  • Tool-calling agent: AGENT root with repeated LLM and TOOL children.
  • Graph/workflow agent: AGENT or CHAIN spans per node, with model/tool spans inside each node.
  • Mixture/router system: CHAIN router span, then one or more LLM or model-endpoint spans for selected experts/routes.
  • SLM replacement/flywheel: rollout span compares large production model against candidate SLM/NIM spans; EVALUATOR spans attach accuracy/cost labels.
  • Fine-tuned adapter: LLM span carries base model, adapter ID, dataset lineage, and prompt version.
  • Guardrailed system: GUARDRAIL spans appear before/after model/tool calls.

The rule: every rollout starts as one OTel trace. Every model, retrieval, rerank, tool, guardrail, evaluator, and orchestration step becomes a span. OpenInference tells downstream systems what those spans mean. Vendor tools then map differently: Phoenix/Arize display OpenInference spans natively; LangSmith maps similar concepts into root/child runs; Bedrock invocation logs join through request IDs; Claude/Codex local logs join through session/thread/tool-call IDs; NeMo joins through dataset/model/adapter/evaluator/deployment IDs.

Comparable Lab and Platform Stacks

NVIDIA is not alone. The major labs and cloud platforms are converging on the same lifecycle, but each uses different product nouns.

Platform Closest NVIDIA-like services Native nouns Maps to our ontology
OpenAI Platform Agent evals, trace grading, graders, fine-tuning/custom-model workflows, Agents SDK integrations graders, traces, evals, datasets, fine-tuning, agents EVALUATOR, rollout, benchmark, training candidate, tool spans
Anthropic Claude Agent SDK, Claude Code/Cowork OTel, Skills, Managed Agents, empirical eval guidance skills, hooks, subagents, sessions, prompts, tool results, evals skill, rollout, session, tool call, OTel trace, golden set
Google Vertex / Gemini Enterprise Agent Engine, ADK, Gen AI Evaluation, Model Garden, Agent Registry/Runtime/Gateway/Identity agents, eval datasets, metrics, templates, runtime, registry rollout, benchmark, dataset, deployment, identity, guardrail
AWS Bedrock AgentCore Runtime, Memory, Gateway, Browser/Code Interpreter, Observability, Evaluations, Optimization, Policy sessions, traces, online evals, CloudWatch results, policy, optimization rollout, trace, eval, policy/guardrail, tool call, promotion loop
Microsoft Foundry Agent Service, external-agent registration, Azure Monitor/Application Insights tracing, trace-to-dataset, evals connected agents, traces, Application Insights, datasets, evaluators OTel trace, external agent, dataset, eval, benchmark, cost logs
Mistral Enterprise Observability suite and fine-tuning Explorer, Judges, Campaigns, Datasets, fine-tuning raw event explorer, judge, annotation campaign, dataset, SFT
Databricks Mosaic AI Agent Framework, MLflow tracing/eval/monitoring, Vector Search, Unity Catalog/lineage traces, scorers, evaluation datasets, monitors, lineage trace, eval, benchmark, retriever, artifact lineage, warehouse
IBM watsonx Orchestrate Agentic control plane, governance/observability, pre-deployment evaluation agents, tools, monitoring, governance, evaluations agent registry, tool calls, evals, guardrails, policy gates
Phoenix / Arize OpenInference tracing/evals spans, traces, datasets, experiments, annotations open semantic trace/eval workbench
LangSmith LangChain/LangGraph tracing/evals runs, child runs, datasets, feedback, experiments trace, span, eval, dataset, benchmark

Common pattern:

production traffic / traces
  -> inspect and label failures
  -> create datasets and golden sets
  -> run evals or trace grading
  -> compare prompts/models/agents/tools
  -> customize, optimize, or skill-edit
  -> promote with governance
  -> monitor in production

The taxonomy differences matter less than preserving join keys. Every platform needs a way to carry trace_id, rollout_id, turn_id, tool_call_id, dataset_example_id, prompt_version, model_id, artifact_id, and eval_id. If those survive, OpenTelemetry/OpenInference plus the warehouse can normalize the rest.

Open Standards Mapping

If the organization standardizes on open standards, use OpenTelemetry for transport/correlation, then explicitly choose and version the AI semantic convention: OpenInference for Phoenix/Arize-first workflows, OTel GenAI for AWS/ADOT/standard OTel backends, and MCP semantic conventions for MCP tool, prompt, and resource traffic. Then map vendor records into those conventions rather than trying to force every vendor to emit the same native payload.

Open standard concept OTel/OpenInference representation Claude/Codex mapping Bedrock/Strands mapping AG-UI/A2A mapping LangSmith/Phoenix mapping
Request/workflow trace trace_id, root span Session/turn/root agent task AgentCore/Strands root agent span; Bedrock request joined by request ID AG-UI runId; A2A taskId/contextId attached as attributes LangSmith root run; Phoenix trace
Operation span span_id, parent_span_id, start_time, end_time, status Model call, tool call, file edit, shell command, retrieval Bedrock model span, Strands tool/model span Protocol lifecycle event promoted to span when it has duration LangSmith child run; Phoenix span
LLM call openinference.span.kind=LLM, model/provider/token attributes Claude/Codex model response step Bedrock invocation metadata plus model ID/token counts Usually absent unless app emits it LangSmith run_type="llm"; Phoenix LLM span
Tool call OpenInference TOOL, OTel GenAI execute_tool, or MCP semantic-convention span with input/output refs Bash/MCP/browser/file/tool function call Strands tool span; Bedrock payload only if model requested tool AG-UI tool-call events; A2A delegated task/message LangSmith run_type="tool"; Phoenix TOOL span after translation
Memory event Memory span/event with store/version/actor/policy IDs Local skill/memory file read/write AgentCore or Managed Agent memory read/write/version A2A artifact/context update when memory is delegated/shared Custom span, annotation, or warehouse row
Context event Compaction/tool-result-clearing/code-execution-filter span with token counts Local compaction and shell/code filtering Code interpreter, tool-output offload, context-summary events Protocol payload reduction or state snapshot Trace metadata or warehouse row
Retrieval openinference.span.kind=RETRIEVER, document refs Search/read/source-acquisition tools Bedrock KB/agent retrieval if instrumented UI/delegation metadata only unless app emits retriever span LangSmith run_type="retriever"; Phoenix RETRIEVER span
Conversation turn Custom turn_id, message events, session/thread metadata User prompt plus assistant/tool sequence App-layer session metadata; not Bedrock invocation alone AG-UI message lifecycle; A2A messages LangSmith thread metadata; Phoenix session metadata
Rollout Custom rollout_id, prompt_version, model/tool config, dataset/example refs One agent episode or CLI session segment One AgentCore/Strands workflow execution or Bedrock-backed app run One UI run or delegated task execution LangSmith experiment run; Phoenix trace plus eval metadata
Eval result Custom eval span/event or warehouse evaluations row Build/test/human label attached to session/turn/artifact AgentCore/Phoenix/LangSmith eval or external harness result Protocol success is not enough; attach separate eval LangSmith feedback/evaluation; Phoenix annotation/eval
Golden-set example Dataset row with stable ID and split Curated local trace/task converted to eval Bedrock APO/eval sample or external benchmark row Protocol replay case if UI/delegation matters LangSmith dataset example; Phoenix dataset/experiment
Benchmark run Experiment/rollout collection with fixed scorer StateBench or local harness run over agent traces Bedrock eval/prompt optimization or external harness AG-UI/A2A conformance/e2e harness LangSmith experiment; Phoenix experiment

OpenTelemetry calls are the emission mechanics: start a root span for the rollout, create child spans for model/tool/retrieval/protocol/memory/context operations, attach session/thread/turn/tool/memory IDs as attributes, and export via OTLP to a collector. OpenInference, OTel GenAI, and MCP semantic conventions then give those spans AI-specific meaning. Record the convention name and version because backend rendering and collector recognition are not stable across all combinations.

Raw Snippets

Missing Players and Underdeveloped Gaps

The June 2026 article is directionally right, but the source base was too weighted toward Claude/Codex/Bedrock/LangSmith/Phoenix. A broader pass adds four missing buckets.

Gap Why it matters What to do next
OTel GenAI version drift Current OTel GenAI conventions are marked development and include a v1.36 transition path with OTEL_SEMCONV_STABILITY_OPT_IN; collectors and dashboards can disagree after instrumentation upgrades Pin semantic-convention version, record instrumentation version, and test dual/translated emission before upgrading
MCP semantic conventions Tool-call reliability lives at the MCP boundary, and MCP protocol spans are not fully described by generic LLM spans Add MCP-specific span attributes for method, server identity, target, protocol version, approval, argument/result refs, and trace context
Persistent agent memory Managed agent memory can now be shared, audited, versioned, redacted, and exported; memory reads/writes explain many cross-session successes and failures Add memory_events, memory spans, memory store/version IDs, and memory quality/security evals
Context engineering observability Compaction, context rot, tool-output offload, code-execution filtering, and token-budget choices explain cost and reliability more directly than storage tiers alone Add context_events, token-before/after fields, compaction IDs, and context regression metrics
GRPO-family frontier and reward design GRPO/RLVR is not the end state; DAPO, Dr. GRPO, GSPO, and step-level/process-supervised variants target real failure modes, but reward design remains the bottleneck Treat GRPO as a baseline for verifiable tasks; document reward contract before choosing an RL algorithm
Environment-as-package layer Prime/Verifiers is one example of a broader pattern where the environment is eval harness, reward source, rollout substrate, and synthetic-data generator Standardize environment package metadata and link rollout traces to reward/eval artifacts
OSS observability and eval platforms Langfuse, Weave, Braintrust, Helicone, OpenLIT/OpenLLMetry, DeepEval, Ragas, TruLens, Promptfoo, and related tools are where many teams actually run trace-to-eval loops Add them as optional views over the same warehouse, then test which one preserves tool calls, datasets, feedback, and export best
China cloud and lab stacks Alibaba/AgentScope, Tencent ADP, Baidu Qianfan, Volcengine ADK/Ark, Huawei ModelArts/Pangu, Coze, Zhipu, Moonshot/Kimi, and DeepSeek are important for global agent deployments Treat public docs as lifecycle/product coverage until they publish raw trace/export contracts comparable to OTel/OpenInference/LangSmith
Operator lore from X/YouTube/community posts Practitioners repeatedly point out that trace UIs show what happened, but not whether the run was correct or how state drifted Mine credible public threads/videos for failure taxonomies, then convert them into eval labels and warehouse queries
Research prototypes and benchmarks AgentTrace, CoderForge-style trajectory mining, failure-aware observability, eBPF boundary tracing, and standardized agent exams point beyond dashboard tracing Track these as benchmark/eval design inputs, not production standards until schemas and data rights are clear

Additional Players to Track

Player Current public signal Fit in this standard
Langfuse Open-source LLM engineering platform; docs and repo emphasize OTel-based tracing, tool invocations, retrieval steps, prompt management, datasets, evals, experiments, and human annotation Strong candidate for OSS trace/eval UI over OTel-like data; still keep native raw vault and warehouse exports
W&B Weave W&B positions Weave around LLM observability, traces, online evaluations, experiments, and model monitoring Useful when ML experiment tracking and production trace scoring need one workspace
Braintrust Docs expose trace hierarchy/timeline/conversation views, raw trace JSON, tool spans, datasets, experiments, scorers, and human review Strong eval-first complement to LangSmith/Phoenix, especially for systematic scorer and dataset workflows
Helicone Gateway-style observability for LLM requests, costs, latency, routing, dashboards, alerts, prompt management, and request/response controls Good request gateway and cost lens; weaker as a full local-agent raw replay system unless joined to local traces
OpenLIT / OpenLLMetry / Traceloop OTel-style LLM instrumentation ecosystems that Phoenix can translate into OpenInference Useful instrumentation layer; standardize downstream semantics to OpenInference and warehouse tables
DeepEval, Ragas, TruLens, Promptfoo Evaluation, RAG assessment, prompt testing, and CI-oriented quality checks Feed evaluations, golden_sets, and training_candidates, not a substitute for trace storage
NVIDIA AI-Q Blueprint Documents Phoenix, Weave, tracing of agent execution, LLM calls, tool invocations, token usage, and dataset construction from traced runs Confirms the same architecture: traces first, eval datasets second, backend choice third

China and Asia Gap Map

Platform What public docs show Current gap
AgentScope / Alibaba ecosystem AgentScope docs describe built-in Studio traces, token usage metadata, and tracing_url export to Phoenix, Langfuse, Alibaba CloudMonitor, or any OTLP endpoint This is the strongest China-adjacent OTel/Phoenix/Langfuse bridge found so far; verify exact emitted attributes and tool-call payload shape
Tencent Cloud Agent Development Platform Public docs describe app analytics, conversation history, Q&A optimization, evaluation set export/upload, workflows, multi-agent mode, knowledge base, plugin marketplace, approval center, and permissions Good lifecycle coverage, but raw trace/export/OTel contract is not public enough to treat as standard-grade
Baidu Qianfan Product pages describe an agent-centered platform with agent engine, tools and MCP, model services, and enterprise capabilities Need public trace/export/eval schema before mapping beyond product lifecycle
Volcengine ADK / Ark ADK tracing docs expose trace IDs and GenAI attributes; Volcengine articles discuss SLI-driven full-chain agent observability Promising technical signal; needs English/product contract validation and raw export samples
Huawei ModelArts / Pangu ModelArts/Pangu docs cover data preprocessing, labeling, training, deployment, monitoring metrics, and agent/application development Strong AI lifecycle coverage, but agent trace/tool-call schema is under-specified publicly
Coze Relevant as a widely used bot/agent builder with workflows, plugins, knowledge, and app publishing Need deeper docs pass for logs, exports, evals, and trace IDs
Moonshot/Kimi and DeepSeek Public docs/product pages emphasize model/API access, agent tasks, web search/tool use, pricing, and model capability Treat as model/API providers unless separate agent telemetry/export contracts emerge

The practical conclusion does not change: use native raw vaults plus OTel/OpenInference plus curated eval/training datasets. What changes is the watchlist. Any provider claiming “agent platform” status should be evaluated against this checklist before adoption:

  1. Can it emit or export full trace trees with stable trace_id, span_id, parent_span_id, turn_id, and tool_call_id?
  2. Does it preserve tool arguments, tool results, errors, approvals, artifacts, retrieval refs, and state deltas with configurable redaction?
  3. Can traces be converted into datasets, golden sets, evaluator corrections, and training candidates?
  4. Can data leave the product as JSONL, Parquet, OTLP, OpenInference, LangSmith-compatible runs, or warehouse tables?
  5. Are costs and retention clear for full-content logging versus metadata-only tracing?

X and YouTube Research Notes

Public X/web snippets and YouTube search results are useful for discovery, but they are weaker evidence than official docs and code. The credible patterns worth mining are consistent:

  • Teams want natural-language and SQL-style querying over traces: “what do failing runs have in common?”, cost breakdowns, latency percentiles, and before/after diffs.
  • Agent failures often hide in state, retrieval, tool sequence, repeated commands, missing tests, and silent non-work, not just bad final answers.
  • Trace views must be paired with evals, diagnostics, and run review; otherwise they move log-diving into a prettier UI.
  • Public trajectory datasets such as coding-agent traces can reveal behavioral predictors of success, but they also raise training-rights, contamination, and task-distribution questions.

Those notes should become a separate operator-lore appendix if we continue: collect official talks and high-signal public threads, classify failure modes, and convert each repeated claim into an eval label or warehouse query.

Config Appendix Status

The technical appendix now includes copy/paste max-telemetry configs and host file locations for the main harnesses. The important correction is that these are not all equally verified:

Harness Copy/paste config included Host location included Verified status
Claude Code Yes: managed JSON and shell exports Yes: macOS, Linux/WSL, Windows managed settings plus user/project .claude paths Docs verified; local binary version checked as 2.1.150; collector ingestion still needs a lab run
Claude Agent SDK Yes: Python and TypeScript app-level OTel plus child Claude Code env App/container environment and ClaudeAgentOptions.env; not a fixed host dotfile Docs verified for W3C trace propagation into the child Claude Code process; collector ingestion still needs SDK sample runs
Codex CLI Yes: user-level config.toml Yes: $CODEX_HOME / ~/.codex / Windows user home; project .codex caveat Docs verified; local binary version checked as 0.135.0; collector ingestion still needs a lab run
Codex Desktop / IDE Partial: same managed/local config family Yes at config-family level Must be sampled per app build before claiming complete Desktop/IDE coverage
Bedrock invocation logging Representative JSON/IaC shape Cloud account/Region setting, not host-local Docs verified; AWS account enablement not run here
Strands / AgentCore Yes: Python StrandsTelemetry and TypeScript setupTracer() examples Application/container environment plus SDK code; not a fixed host dotfile Docs verified for OTel traces, model/tool/retriever spans, semantic-convention opt-in, OTLP and console exporters; collector ingestion still needs sample runs
Langfuse / Braintrust / Weave / OpenLIT Yes: backend/SDK OTel environment snippets Application environment or collector config Docs verified; per-version ingestion tests still needed
AG-UI / A2A Protocol ID propagation JSON App code/protocol stream, not a host config file Event contracts verified; they do not capture full model/local tool behavior alone

See the Copy/Paste Max-Telemetry Config Appendix.

Operationally, the next missing artifact is a small harness test suite: start an OTLP collector, run one minimal session per agent surface, store raw OTLP payloads and native local logs, and verify every claimed field appears. Until that exists, the article should distinguish “docs verified” from “collector verified.”

Gap-Ingest Follow-Ups

The missing-player work is now split into three durable technical pages:

Raw Snippets

The linked technical appendix includes safe representative snippets for:

  • Claude Code telemetry environment with prompt/tool gates enabled;
  • Codex OTel config with prompt logging enabled;
  • Codex local session/log record shape;
  • Bedrock invocation log shape;
  • Strands/AgentCore OpenTelemetry span shape;
  • AG-UI lifecycle/tool/state events;
  • A2A task/send and status/artifact events;
  • OpenInference span attributes;
  • ATIF trajectory;
  • LangSmith run tree and bulk export fields.

See Agent Observability Format Standard 2026.

Sources

Brandon Sneider | brandon@brandonsneider.com June 2026