← Agent Frameworks 🕐 31 min read
Agent Frameworks

Agent Observability Format Standard 2026

Brandon Sneider · June 2026

This appendix answers the format question left open by the first agent-log

See also (wiki): wiki/agent-telemetry-format-standardization.md · wiki/agent-operations-observability.md · wiki/semantic-observability.md

Executive Summary

This appendix answers the format question left open by the first agent-log mining playbook: do not collapse Claude, Codex, Bedrock, Strands, AG-UI, A2A, OpenInference, ATIF, Phoenix, and LangSmith into one lossy JSON object. Keep native raw records, normalize trace causality to OpenTelemetry, normalize AI semantics to OpenInference or the OTel-community GenAI semantic conventions, and expose LangSmith-compatible run/dataset views where they add value. The OpenInference-vs-OTel framing is not clean in practice: both are OTel-compatible ways to describe AI work, but they use different attributes and backend display contracts.

OTel GenAI is also not a fixed target. Current OpenTelemetry docs still mark the GenAI conventions as development and document a v1.36 transition path using OTEL_SEMCONV_STABILITY_OPT_IN, including gen_ai_latest_experimental. Therefore every OTel/OpenInference/MCP span conversion should preserve otel_semconv_version, ai_semantic_convention, ai_semantic_convention_version, instrumentation_name, instrumentation_version, and semconv_stability_opt_in. MCP semantic conventions are a third layer to track for tool/protocol spans, not a footnote inside generic LLM telemetry.

LangSmith is applicable because current docs describe OpenTelemetry ingestion, LangSmith-specific OTel attribute mapping, run/tree query syntax, message-view trace extraction, datasets, experiments, few-shot evaluator correction loops, and bulk Parquet trace export. It is strongest as an application observability and eval workbench. It is not a complete replacement for a raw local transcript vault, Bedrock invocation logs, protocol logs, or a warehouse.

Standardization Decision

Need Best format Why
Compliance provenance Native raw record plus hash Preserves exact source contract and future re-parsing options
Cross-service timing OpenTelemetry trace/span Standard parentage, propagation, exporter/collector support
LLM/tool/retriever semantics OpenInference attributes or OTel GenAI gen_ai.* attributes AI operation vocabulary on top of OTel; choose per backend and translation path
LangChain/LangGraph debugging LangSmith run tree Native Messages view, run filters, datasets, feedback, experiments
Agent trajectory replay/eval ATIF-like trajectory plus spans Keeps multi-step action/observation sequences
Skill mining Curated task episodes Skills need repeated validated patterns, not raw transcripts
Corporate KB mining Turn/query/retrieval defect rows The question is findability, not model-call telemetry alone
YAML optimization Artifact plus validator feedback dataset Deterministic failure codes beat generic chat fine-tuning
Reliability metrics Evaluation rows joined to spans Pass/fail and failure modes are labels, not trace fields

Format Gap and Toggle Matrix

Source Enablement needed Native unit Tool-call representation Main blind spot
Claude Code local JSONL Local transcript retention; redacted extractor Session JSONL event/record Local tool records, usually pairable by sequence/session and sometimes tool IDs No cloud identity or org fleet metrics
Claude Code OTel CLAUDE_CODE_ENABLE_TELEMETRY=1, OTLP exporters/endpoints, optional prompt/tool content gates OTel metric/log/event/span Tool events/spans when details/content gates are enabled Subprocesses need their own telemetry config
Claude Agent SDK SDK tracing plus propagated trace context into Claude Code child process App trace plus child spans Tool/MCP/subagent work as child spans/events Local CLI transcript completeness still needs joining
Claude Cowork OTel Admin OTLP endpoint and privacy controls OTel event grouped by prompt.id Tool result events with potentially sensitive arguments Compliance API and Cowork OTel are not identical coverage
Codex CLI OTel otel.trace_exporter, otel.metrics_exporter, OTLP endpoint/headers/protocol, otel.log_user_prompt=true if needed OTel logs/metrics/traces Tool/function calls as spans/logs where emitted; local session join recommended Public emitted schema and entrypoint behavior need validation
Codex local JSONL/SQLite Local redacted extractors Session row, session index row, SQLite log row Function/tool calls in session rows, logs, and shell snapshots Local observed schema is not a stable vendor contract
Codex Desktop Likely global ~/.codex/config.toml OTel; verify installed build App-server/local session records, TBD TBD; expect app-server tool events and local records Exact Desktop export/local schema is unresolved
Microsoft Scout Unknown public telemetry toggles; preserve session history/admin exports if exposed Preview session/history/memory/tool activity Shell/browser/MCP/M365/subagent actions need typed tool records Public export/admin/OTel contract is incomplete
Bedrock invocation logs Enable invocation logging per account/region/destination and payload policy Cloud model invocation record Tool calls only inside model payloads, if logged No local tool execution; bedrock-mantle endpoint gap
AgentCore/Strands ADOT/OTel instrumentation, CloudWatch GenAI observability, session baggage OTel spans/logs/metrics Agent/tool/model spans AgentCore is not a separate schema; it is AWS/ADOT/OTel GenAI transport plus AWS storage/query surfaces
AG-UI Preserve event stream and attach trace/session IDs Protocol event First-class tool-call lifecycle events No model/cloud/local execution proof alone
A2A Preserve JSON-RPC task/message/artifact IDs and attach trace context Task/message/status/artifact Delegation as task/message/artifact, not local tool execution Delegated agent internals need separate telemetry
OpenTelemetry GenAI App/framework instrumentation and collector export Trace/span/event Tool spans/events with attributes No learning labels or turn semantics by default
MCP semantic conventions MCP client/server instrumentation and trace-context propagation Protocol/tool span MCP method/server/tool spans with protocol attributes Host approval logs, raw tool results, and training labels still need joins
OpenInference/Phoenix OpenInference instrumentation or span translation OTel-compatible AI span TOOL semantic span kind Plain OTel GenAI spans may need translation; otherwise Phoenix can lose AI span semantics or show unknown kinds
LangSmith SDK tracing or OTel ingestion; metadata such as thread_id, ls_provider, ls_model_name; dataset/eval configuration Run tree; root run and child runs Child runs with run_type="tool"; Messages view pairs tool calls/results for supported payload shapes Only sees instrumented data; not a full raw desktop-agent vault
ATIF Trajectory export/conversion with steps and outcomes Trajectory Action/observation steps Needs conversion to trace spans for observability backends

Tool-Call Normalization

Every source should be normalized into one tool_calls row per request/result pair:

{
  "tool_call_id": "tool_hash",
  "trace_id": "trace_hash",
  "span_id": "span_hash",
  "turn_id": "turn_hash",
  "source_system": "codex_local_jsonl",
  "tool_type": "shell",
  "tool_name": "exec_command",
  "request_time": "2026-06-07T13:02:10.481Z",
  "response_time": "2026-06-07T13:02:12.927Z",
  "status": "success",
  "arguments_ref": "raw_args_redacted_hash",
  "result_ref": "raw_result_redacted_hash",
  "side_effects": [
    { "kind": "file_write", "artifact_id": "artifact_hash" }
  ],
  "approval": {
    "required": false,
    "decision": "not_required"
  }
}

Tool calls are not all the same:

  • Claude/Codex local records are closest to replay, because they can include the local tool request/result and surrounding assistant turn.
  • Claude/Codex OTel is best for timing and fleet metrics, but sensitive tool details often require explicit opt-in.
  • Bedrock invocation logs can show that a model requested or described a tool call in the payload, but they do not prove the local tool actually ran.
  • AG-UI shows user-interface tool-call lifecycle events and streaming boundaries.
  • A2A shows delegated task/message/artifact flow; the delegated agent needs its own trace to prove its internal tool calls.
  • LangSmith represents tools as child runs and can render tool calls in the Messages view when the trace payload matches supported extraction rules.
  • OpenInference represents tools as semantic spans; Phoenix can display them, but labels and raw provenance still need joins. Plain OTel GenAI tool spans are a different convention and should be translated or dual-emitted before assuming Phoenix/Arize rendering parity.

Bedrock, CloudWatch, and Observability Cost Tradeoffs

Amazon Bedrock model invocation logging can deliver logs to CloudWatch Logs, S3, or both. AWS documents that it can collect model input data, output data, and metadata for supported calls in an account and Region, but only for supported bedrock-runtime operations. CloudWatch log events contain invocation metadata plus input/output JSON bodies up to 100 KB. S3 receives gzipped JSON batches and is also the destination for large JSON bodies and binary data.

That creates a real design decision:

Pattern Use when Cost surface Technical cost
CloudWatch only Hot operations and incident response inside AWS Log ingestion, retained storage, Logs Insights scans, optional alarms/dashboards Fast queries, but higher hot-log cost and AWS-centric analysis
S3 only Durable audit vault, long retention, offline mining S3 storage class, PUT/GET, Athena/Glue/OpenSearch scans, lifecycle transitions Cheaper archive, but requires partitioning and ETL
CloudWatch plus S3 Regulated production where hot and cold paths both matter Both hot-log and object-storage costs Best coverage, but duplicate retention and redaction policies are mandatory
CloudWatch to Firehose/warehouse Near-real-time analytics outside CloudWatch CloudWatch delivery/streaming, Firehose ingest, destination indexing/storage Good fanout, but every destination adds schema and cost governance
Prometheus / managed Prometheus Alerting and SLOs over derived counters, gauges, and histograms Prometheus scrape/remote-write/OTLP metrics ingest, storage, and cardinality costs Metrics-only plane; not a store for raw Bedrock invocation JSON, Claude logs, traces, prompts, or tool bodies
OTel/OpenInference to Phoenix/Arize/LangSmith Agent/app debugging, evals, datasets, feedback SaaS span/GB/retention costs or self-hosted infra/SRE costs Better AI semantics, but not a substitute for Bedrock audit logs

Current pricing pages make the shape of the bill clear even when exact numbers vary by Region and plan:

  • CloudWatch pricing is usage-based. AWS examples show CloudWatch Logs ingestion and archival charges, Logs Insights scanning, and separate application-observability span ingestion/indexing surfaces.
  • S3 pricing separates storage class, request/data retrieval, transfer, and management/query features.
  • Firehose pricing is volume-based ingestion/delivery, with optional format conversion and destination features.
  • LangSmith pricing is plan/usage based and positions trace retention, bulk data export, datasets, feedback, and enterprise controls as product features.
  • Arize pricing separates free/self-hosted Phoenix, free/pro cloud tiers, span counts, ingestion volume, retention, and enterprise support.

The engineering conclusion:

  1. Use S3 as the Bedrock invocation raw vault for long retention.
  2. Use CloudWatch for operationally necessary hot logs, not indiscriminate full-content retention.
  3. Export application/agent spans to OpenInference/Phoenix/Arize or LangSmith for debugging and evaluation.
  4. Store raw prompt/tool bodies only under explicit policy. Use hashes, byte counts, redacted refs, and sampled content in shared stores.
  5. Create a warehouse join between Bedrock request IDs, OTel trace IDs, local session IDs, tool call IDs, and eval labels. No single backend gives all of those by default.

Prometheus Boundary

Prometheus should ingest metrics derived from agent and Bedrock telemetry, not the raw telemetry itself. Prometheus can receive OTLP metrics when its OTLP receiver is enabled; traces and logs still need an OTel Collector plus a trace/log backend or warehouse.

Recommended split:

Claude Code / Claude Agent SDK / app OTel
  -> metrics pipeline -> Prometheus
  -> traces pipeline  -> trace backend
  -> logs pipeline    -> log store / warehouse

Bedrock ModelInvocationLog in CloudWatch Logs or S3
  -> parser / ETL
  -> derived metrics -> Prometheus
  -> raw JSON        -> CloudWatch/S3/warehouse

Example derived metrics:

agent_model_invocations_total{source,provider,model,operation,status}
agent_input_tokens_total{source,provider,model}
agent_output_tokens_total{source,provider,model}
agent_tool_calls_total{tool_name,tool_type,status}
bedrock_invocations_total{region,model_id,operation}
bedrock_input_tokens_total{region,model_id,operation}
bedrock_output_tokens_total{region,model_id,operation}

Prometheus labels must stay bounded. Do not use prompts, completions, requestId, trace_id, user email, local file path, raw tool arguments, or tool output bodies as labels. Preserve those values in CloudWatch/S3/raw vaults, trace stores, or warehouse side tables, then use Prometheus for rates, alerts, SLOs, and fleet health.

Cost Model Inputs

Before enabling full Bedrock/CloudWatch logging, estimate:

monthly_log_gb =
  invocations_per_month
  * avg_logged_bytes_per_invocation
  / 1_073_741_824

cloudwatch_hot_cost =
  monthly_log_gb * region_log_ingest_price
  + retained_gb * region_log_storage_price
  + logs_insights_scanned_gb * region_query_price

s3_archive_cost =
  retained_gb_by_storage_class * s3_storage_price
  + put_get_request_counts * request_prices
  + athena_or_opensearch_scan_or_index_cost

The largest variable is not the number of invocations; it is whether you log full prompts, completions, retrieved documents, tool outputs, images, and large JSON bodies. For agentic systems, payload multiplication is common: one user turn can fan out into several model calls and many tool/retrieval events.

Canonical Warehouse Tables

raw_events

{
  "raw_event_id": "raw_01jz...",
  "source_system": "codex_local_jsonl",
  "native_schema_version": "observed-2026-06-07",
  "native_timestamp": "2026-06-07T13:02:10.481Z",
  "observed_at": "2026-06-07T13:02:11.000Z",
  "join_keys": {
    "session_id": "sess_hash",
    "thread_id": "thread_hash",
    "trace_id": "trace_hash",
    "tool_call_id": "tool_hash"
  },
  "payload_redacted": {
    "type": "response_item",
    "item_type": "tool_call",
    "content_hash": "sha256:..."
  },
  "payload_hash": "sha256:...",
  "redaction_version": "stateofai-redaction-v1"
}

agent_spans

{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "parent_span_id": "b9c7c989f97918e1",
  "span_kind": "TOOL",
  "name": "shell.exec",
  "start_time": "2026-06-07T13:02:10.481Z",
  "end_time": "2026-06-07T13:02:12.927Z",
  "status": "OK",
  "attributes": {
    "tool.name": "exec_command",
    "agent.system": "codex",
    "repo.name": "stateofai",
    "git.commit": "unknown-dirty",
    "privacy.redaction_version": "stateofai-redaction-v1"
  },
  "input_ref": "raw_01jz...",
  "output_ref": "raw_01jz..."
}

memory_events

{
  "memory_event_id": "mem_01jz...",
  "trace_id": "trace_hash",
  "span_id": "span_hash",
  "session_id": "sess_hash",
  "agent_id": "agent_hash",
  "memory_store_id": "workspace_or_user_store_hash",
  "memory_version_id": "version_hash",
  "operation": "write",
  "scope": "workspace",
  "content_ref": "redacted_memory_ref",
  "previous_hash": "sha256:...",
  "next_hash": "sha256:...",
  "audit_actor": "agent_or_user_hash",
  "redaction_version": "stateofai-redaction-v1"
}

context_events

{
  "context_event_id": "ctx_01jz...",
  "trace_id": "trace_hash",
  "span_id": "span_hash",
  "turn_id": "turn_hash",
  "event_type": "compaction",
  "compaction_id": "compact_hash",
  "tokens_before": 178000,
  "tokens_after": 24000,
  "context_window_tokens": 200000,
  "tool_manifest_tokens": 14000,
  "memory_tokens": 9000,
  "retrieval_tokens": 22000,
  "artifact_refs_created": ["artifact_01jz..."],
  "reason": "token_trigger"
}

turns

{
  "turn_id": "turn_01jz...",
  "session_id": "sess_hash",
  "thread_id": "thread_hash",
  "root_trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "user_intent_label": "research_publish_article",
  "question_hash": "sha256:...",
  "retrieval_attempts": 8,
  "tool_call_count": 41,
  "validation_status": "passed",
  "final_artifact_id": "artifact_01jz...",
  "elapsed_ms": 4708000
}

Representative Raw Snippets

These examples are intentionally redacted and schema-focused. They show the fields that should exist when telemetry is maximally useful; they are not raw private logs.

Copy/Paste Max-Telemetry Config Appendix

These snippets are intentionally “maximum telemetry” configs for lab verification, not a production default. They can include prompts, tool arguments, tool outputs, file paths, commands, approvals, and errors. Use them only in a controlled environment with a redaction policy and a disposable OTLP collector.

Verification Status

Harness Version checked locally Config contract status End-to-end ingestion status
Claude Code 2.1.150 Verified against current Claude Code monitoring docs Not yet collector-verified in this article
Codex CLI 0.135.0 Verified against current OpenAI Codex config docs Not yet collector-verified in this article
Codex Desktop / IDE Not locally version-sampled here OpenAI says local managed configs apply across desktop app, CLI, and IDE extension, but exact local telemetry behavior still needs app-specific sampling Unverified
Bedrock invocation logging AWS account/Region dependent Verified against AWS Bedrock invocation logging docs Not locally enabled here
Strands / AgentCore Not locally sampled here OTel support verified from AWS/Strands public docs Not locally sampled here
Langfuse / Weave / Braintrust / OpenLIT Backend/SDK version dependent Current docs provide OTel endpoints or SDK config Not locally sampled here
AG-UI / A2A Protocol version dependent Event schemas/protocol IDs verified from public docs They preserve protocol events, not full model/local tool execution by themselves

Config File Locations

Tool macOS / Linux Windows Notes
Claude Code managed settings /Library/Application Support/ClaudeCode/managed-settings.json on macOS; /etc/claude-code/managed-settings.json on Linux/WSL C:\Program Files\ClaudeCode\managed-settings.json Admin managed settings have high precedence; the legacy Windows C:\ProgramData\ClaudeCode\managed-settings.json path is no longer supported as of Claude Code v2.1.75
Claude Code user/project settings ~/.claude/settings.json, ~/.claude/settings.local.json, <repo>/.claude/settings.json, <repo>/.claude/settings.local.json %USERPROFILE%\.claude\settings.json and project .claude\settings.json Use managed settings for fleet telemetry; local settings are easier for lab tests
Codex user config/state $CODEX_HOME/config.toml, default ~/.codex/config.toml %USERPROFILE%\.codex\config.toml unless CODEX_HOME is set Telemetry keys must live in user-level config, not project .codex/config.toml
Codex project config <repo>/.codex/config.toml <repo>\.codex\config.toml OpenAI docs say project config cannot override otel
Codex managed requirements /etc/codex/requirements.toml or organization-managed config paths Enterprise/admin managed requirements locations vary Requirements constrain security-sensitive settings; allow_managed_hooks_only belongs here, not in config.toml
Claude Agent SDK app config App/container environment and SDK ClaudeAgentOptions.env App/container environment and SDK ClaudeAgentOptions.env SDK propagates W3C trace context into Claude Code child process when query() runs inside an active OTel span
Strands Agents SDK config App/container environment plus Python/TypeScript OTel setup App/container environment plus Python/TypeScript OTel setup Use strands-agents[otel] for Python or OTel peer deps for TypeScript; config is app code/env, not a fixed host dotfile
OTel Collector Any path you run it from, commonly /etc/otelcol/config.yaml or repo ops/otel-collector.yaml Any path, commonly C:\ProgramData\otelcol\config.yaml Collector config is the shared sink that makes vendor configs testable

Claude Code Managed Settings JSON

{
  "env": {
    "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
    "CLAUDE_CODE_ENHANCED_TELEMETRY_BETA": "1",
    "OTEL_METRICS_EXPORTER": "otlp",
    "OTEL_LOGS_EXPORTER": "otlp",
    "OTEL_TRACES_EXPORTER": "otlp",
    "OTEL_EXPORTER_OTLP_PROTOCOL": "grpc",
    "OTEL_EXPORTER_OTLP_ENDPOINT": "http://127.0.0.1:4317",
    "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Bearer REDACTED",
    "OTEL_METRIC_EXPORT_INTERVAL": "10000",
    "OTEL_LOGS_EXPORT_INTERVAL": "5000",
    "OTEL_LOG_USER_PROMPTS": "1",
    "OTEL_LOG_TOOL_DETAILS": "1",
    "OTEL_LOG_TOOL_CONTENT": "1",
    "OTEL_LOG_RAW_API_BODIES": "file:/var/log/claude-code-api-bodies",
    "ENABLE_BETA_TRACING_DETAILED": "1",
    "BETA_TRACING_ENDPOINT": "http://127.0.0.1:4317",
    "OTEL_RESOURCE_ATTRIBUTES": "service.name=claude-code,deployment.environment=lab,agent.telemetry_profile=max"
  }
}

Claude Code nuance: current docs say traces require both CLAUDE_CODE_ENABLE_TELEMETRY=1 and CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1, plus OTEL_TRACES_EXPORTER. Tool details are gated by OTEL_LOG_TOOL_DETAILS; tool input/output body span events require OTEL_LOG_TOOL_CONTENT=1 and are truncated. Full raw Messages API request/response body events require OTEL_LOG_RAW_API_BODIES; use file:<dir> in lab captures when untruncated bodies are needed. Those bodies include the full conversation history and should never be enabled casually. Detailed hook spans also require ENABLE_BETA_TRACING_DETAILED=1 and BETA_TRACING_ENDPOINT. In interactive CLI sessions, current Claude Code docs also say detailed hook spans require organization allowlisting; Agent SDK and non-interactive claude -p sessions are not gated the same way.

Claude Code Shell Export

export CLAUDE_CODE_ENABLE_TELEMETRY=1
export CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_TRACES_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer REDACTED"
export OTEL_METRIC_EXPORT_INTERVAL=10000
export OTEL_LOGS_EXPORT_INTERVAL=5000
export OTEL_LOG_USER_PROMPTS=1
export OTEL_LOG_TOOL_DETAILS=1
export OTEL_LOG_TOOL_CONTENT=1
export OTEL_LOG_RAW_API_BODIES=file:/var/log/claude-code-api-bodies
export ENABLE_BETA_TRACING_DETAILED=1
export BETA_TRACING_ENDPOINT=http://127.0.0.1:4317
export OTEL_RESOURCE_ATTRIBUTES="service.name=claude-code,deployment.environment=lab,agent.telemetry_profile=max"

Claude Agent SDK Python Max Telemetry

The Agent SDK embeds Claude Code as a child process. The key extra behavior is trace propagation: when query() runs while an application OTel span is active, the SDK injects TRACEPARENT and TRACESTATE into the child process so the Claude Code claude_code.interaction span nests under the app span. The CLI also forwards TRACEPARENT into Bash and PowerShell tool commands unless you override TRACEPARENT in options.env.

pip install claude-agent-sdk opentelemetry-api opentelemetry-sdk \
  opentelemetry-exporter-otlp-proto-http

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

from claude_agent_sdk import ClaudeAgentOptions, query

resource = Resource.create({
    "service.name": "support-triage-agent",
    "service.version": "1.4.0",
    "deployment.environment": "lab",
    "agent.telemetry_profile": "max",
})
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(
    OTLPSpanExporter(
        endpoint="http://127.0.0.1:4318/v1/traces",
        headers={"Authorization": "Bearer REDACTED"},
    )
))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("support-triage-agent")

options = ClaudeAgentOptions(
    env={
        "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
        "CLAUDE_CODE_ENHANCED_TELEMETRY_BETA": "1",
        "OTEL_TRACES_EXPORTER": "otlp",
        "OTEL_METRICS_EXPORTER": "otlp",
        "OTEL_LOGS_EXPORTER": "otlp",
        "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
        "OTEL_EXPORTER_OTLP_ENDPOINT": "http://127.0.0.1:4318",
        "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Bearer REDACTED",
        "OTEL_LOG_USER_PROMPTS": "1",
        "OTEL_LOG_TOOL_DETAILS": "1",
        "OTEL_LOG_TOOL_CONTENT": "1",
        "OTEL_LOG_RAW_API_BODIES": "file:/var/log/claude-agent-sdk-api-bodies",
        "ENABLE_BETA_TRACING_DETAILED": "1",
        "BETA_TRACING_ENDPOINT": "http://127.0.0.1:4318",
        "OTEL_RESOURCE_ATTRIBUTES": (
            "service.name=claude-agent-sdk-child,"
            "deployment.environment=lab,"
            "agent.telemetry_profile=max,"
            "enduser.id_hash=user_hash"
        ),
    }
)

with tracer.start_as_current_span("app.agent_request") as span:
    span.set_attribute("session.id", "session_hash")
    span.set_attribute("enduser.id_hash", "user_hash")
    for message in query("Summarize the ticket and inspect policy docs.", options=options):
        print(message)

Fields to verify:

  • app root span app.agent_request;
  • child claude_code.interaction span;
  • child claude_code.llm_request spans;
  • claude_code.tool, claude_code.tool.blocked_on_user, and claude_code.tool.execution;
  • claude_code.hook only when detailed beta tracing is enabled;
  • session.id;
  • propagated TRACEPARENT in Bash/PowerShell subprocesses;
  • application end-user attribution via resource attributes.

Claude Agent SDK TypeScript Max Telemetry

npm install @anthropic-ai/claude-agent-sdk \
  @opentelemetry/api @opentelemetry/sdk-trace-node \
  @opentelemetry/sdk-trace-base @opentelemetry/resources \
  @opentelemetry/exporter-trace-otlp-http

import { trace } from "@opentelemetry/api";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { resourceFromAttributes } from "@opentelemetry/resources";
import { query } from "@anthropic-ai/claude-agent-sdk";

const provider = new NodeTracerProvider({
  resource: resourceFromAttributes({
    "service.name": "support-triage-agent",
    "service.version": "1.4.0",
    "deployment.environment": "lab",
    "agent.telemetry_profile": "max",
  }),
  spanProcessors: [
    new BatchSpanProcessor(
      new OTLPTraceExporter({
        url: "http://127.0.0.1:4318/v1/traces",
        headers: { Authorization: "Bearer REDACTED" },
      })
    ),
  ],
});
provider.register();

const tracer = trace.getTracer("support-triage-agent");

await tracer.startActiveSpan("app.agent_request", async (span) => {
  span.setAttribute("session.id", "session_hash");
  span.setAttribute("enduser.id_hash", "user_hash");

  for await (const message of query({
    prompt: "Summarize the ticket and inspect policy docs.",
    options: {
      env: {
        CLAUDE_CODE_ENABLE_TELEMETRY: "1",
        CLAUDE_CODE_ENHANCED_TELEMETRY_BETA: "1",
        OTEL_TRACES_EXPORTER: "otlp",
        OTEL_METRICS_EXPORTER: "otlp",
        OTEL_LOGS_EXPORTER: "otlp",
        OTEL_EXPORTER_OTLP_PROTOCOL: "http/protobuf",
        OTEL_EXPORTER_OTLP_ENDPOINT: "http://127.0.0.1:4318",
        OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer REDACTED",
        OTEL_LOG_USER_PROMPTS: "1",
        OTEL_LOG_TOOL_DETAILS: "1",
        OTEL_LOG_TOOL_CONTENT: "1",
        OTEL_LOG_RAW_API_BODIES: "file:/var/log/claude-agent-sdk-api-bodies",
        ENABLE_BETA_TRACING_DETAILED: "1",
        BETA_TRACING_ENDPOINT: "http://127.0.0.1:4318",
        OTEL_RESOURCE_ATTRIBUTES:
          "service.name=claude-agent-sdk-child,deployment.environment=lab,agent.telemetry_profile=max,enduser.id_hash=user_hash",
      },
    },
  })) {
    console.log(message);
  }

  span.end();
});

Nuance: interactive Claude Code sessions ignore inbound TRACEPARENT; Agent SDK and claude -p runs honor it. Span names and attributes remain beta and may change between releases.

Codex User Config TOML

Put this in user config, not project config:

  • macOS/Linux: ~/.codex/config.toml, or $CODEX_HOME/config.toml
  • Windows: %USERPROFILE%\.codex\config.toml, or %CODEX_HOME%\config.toml
#:schema https://developers.openai.com/codex/config-schema.json

[otel]
environment = "lab"
log_user_prompt = true
metrics_exporter = "otlp-http"

[otel.exporter.otlp-http]
endpoint = "http://127.0.0.1:4318/v1/logs"
protocol = "binary"
headers = { Authorization = "Bearer REDACTED" }

[otel.trace_exporter.otlp-http]
endpoint = "http://127.0.0.1:4318/v1/traces"
protocol = "binary"
headers = { Authorization = "Bearer REDACTED" }

Codex nuance: OpenAI docs say Codex stores local state under CODEX_HOME defaulting to ~/.codex. Project .codex/config.toml files are read only for trusted projects, but telemetry keys such as otel are ignored in project config and must be set in user-level ~/.codex/config.toml. Current reference docs list OTel log exporter, metrics exporter, trace exporter, TLS options, and otel.log_user_prompt; OpenAI’s safety article shows Codex logs for user prompts, approval decisions, tool results, MCP server usage, and network proxy allow/deny events. A public GitHub issue reported differences between interactive codex, codex exec, and codex mcp-server in older 0.105.0; this article has not yet collector-verified those surfaces on local 0.135.0.

Bedrock Invocation Logging

This is not a local host file. It is an AWS account/Region setting configured through Bedrock APIs, console, or IaC. For maximum audit coverage, enable both S3 and CloudWatch destinations, payload logging where policy allows, and KMS encryption. Keep the separate OTel app/agent trace because Bedrock invocation logs do not prove local tool execution.

Representative IaC shape:

{
  "loggingConfig": {
    "cloudWatchConfig": {
      "logGroupName": "/aws/bedrock/model-invocations",
      "roleArn": "arn:aws:iam::123456789012:role/BedrockInvocationLoggingRole"
    },
    "s3Config": {
      "bucketName": "company-bedrock-invocation-logs",
      "keyPrefix": "bedrock/model-invocations/"
    },
    "kmsKeyArn": "arn:aws:kms:us-east-1:123456789012:key/REDACTED",
    "logPromptAndCompletion": true
  }
}

Strands / AgentCore Python Environment

The Strands SDK has a richer contract than the previous baseline snippet showed. Current docs say Strands traces use OpenTelemetry and capture agent lifecycle, LLM interactions, retrievers, tool usage, and event-loop processing. Install the OTel extra:

pip install 'strands-agents[otel]'

Environment:

export OTEL_SERVICE_NAME=strands-agent
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=lab,agent.framework=strands,agent.telemetry_profile=max,service.version=1.0.0"
export OTEL_TRACES_EXPORTER=otlp
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer REDACTED"
export OTEL_SEMCONV_STABILITY_OPT_IN="gen_ai_latest_experimental,gen_ai_tool_definitions,gen_ai_use_latest_invocation_tokens"

Python:

from os import linesep

from strands import Agent
from strands.telemetry import StrandsTelemetry

telemetry = StrandsTelemetry()

# Send traces to OTLP.
telemetry.setup_otlp_exporter()

# Also write local JSONL spans for schema verification.
logfile = open("/tmp/strands-traces.jsonl", "wt")
telemetry.setup_console_exporter(
    out=logfile,
    formatter=lambda span: span.to_json() + linesep,
)

# Metrics can also be exported through OTel.
telemetry.setup_meter(
    enable_console_exporter=True,
    enable_otlp_exporter=True,
)

agent = Agent(
    model="us.anthropic.claude-sonnet-4-20250514-v1:0",
    system_prompt="You are a governed support triage agent.",
)

response = agent("Find the policy and draft a cited answer.")
print(response)
logfile.close()

Fields to verify:

  • top-level agent span with gen_ai.system and gen_ai.agent.name;
  • event-loop cycle spans;
  • model invocation spans with model ID, prompt/completion token counts, cache token attributes where supported, latency start/end attributes;
  • tool spans with tool.status, gen_ai.tool.name, gen_ai.tool.call.id, gen_ai.operation.name, start/end times, and formatted tool result;
  • retriever spans when using retrieval;
  • custom trace attributes for session.id, rollout_id, prompt_version, policy_version, and Bedrock request IDs where the app can attach them.

Strands TypeScript Max Telemetry

npm install @strands-agents/sdk \
  @opentelemetry/api @opentelemetry/sdk-trace-node \
  @opentelemetry/sdk-trace-base @opentelemetry/resources \
  @opentelemetry/exporter-trace-otlp-http

export OTEL_EXPORTER_OTLP_ENDPOINT="http://127.0.0.1:4318"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer REDACTED"
export OTEL_SEMCONV_STABILITY_OPT_IN="gen_ai_latest_experimental,gen_ai_tool_definitions,gen_ai_use_latest_invocation_tokens"

import { Agent } from "@strands-agents/sdk";
import { setupTracer } from "@strands-agents/sdk/telemetry";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { BatchSpanProcessor, ConsoleSpanExporter } from "@opentelemetry/sdk-trace-base";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";

const provider = new NodeTracerProvider({
  spanProcessors: [
    new BatchSpanProcessor(
      new OTLPTraceExporter({
        url: "http://127.0.0.1:4318/v1/traces",
        headers: { Authorization: "Bearer REDACTED" },
      })
    ),
    new BatchSpanProcessor(new ConsoleSpanExporter()),
  ],
});

setupTracer({
  provider,
  exporters: { otlp: true, console: true },
});

const agent = new Agent({
  systemPrompt: "You are a governed support triage agent.",
});

const response = await agent("Find the policy and draft a cited answer.");
console.log(response);

For Bedrock-backed Strands agents, attach session.id, Bedrock request IDs, model IDs, tool names, tool-call IDs, and guardrail/policy versions as span attributes, baggage, or custom trace attributes. The SDK emits many spans, but the app still owns business keys such as turn_id, rollout_id, dataset_example_id, and policy_version.

Langfuse as OTLP Backend

export OTEL_EXPORTER_OTLP_ENDPOINT="https://cloud.langfuse.com/api/public/otel"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic REDACTED,x-langfuse-ingestion-version=4"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://cloud.langfuse.com/api/public/otel/v1/traces"
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=Basic REDACTED,x-langfuse-ingestion-version=4"
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_RESOURCE_ATTRIBUTES="langfuse.environment=lab,service.name=agent-app"

Langfuse docs state OTLP over HTTP JSON/protobuf is supported and gRPC is not currently supported for this endpoint.

Braintrust as OTLP Backend

export BRAINTRUST_API_KEY=REDACTED
export BRAINTRUST_PARENT="project_name:agent-observability-lab"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.braintrust.dev/otel"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://api.braintrust.dev/otel/v1/traces"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer REDACTED, x-bt-parent=project_name:agent-observability-lab"

Braintrust docs require braintrust[otel] >= v0.3.5 for Python distributed tracing and braintrust >= v1.0.0 plus @braintrust/otel >= v0.1.0 for TypeScript distributed tracing.

W&B Weave OTel Collector

Weave supports sending OTel traces directly or through a collector and maps OpenTelemetry GenAI, OpenInference, Vercel AI SDK, MLflow, and Traceloop style attributes. Use wandb.thread_id or supported session/thread attributes to group turns.

export WANDB_API_KEY=REDACTED
export OTEL_SERVICE_NAME=agent-app
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=lab,wandb.thread_id=thread-lab"

OpenLIT Max Capture

export OTEL_SERVICE_NAME=my-ai-app
export OTEL_DEPLOYMENT_ENVIRONMENT=lab
export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer REDACTED"
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true
export OPENLIT_MAX_CONTENT_LENGTH=0
export OPENLIT_DISABLE_METRICS=false
export OPENLIT_DISABLE_EVENTS=false
export OPENLIT_COLLECT_SYSTEM_METRICS=true
export OPENLIT_CAPTURE_DB_PARAMETERS=true
export OTEL_RESOURCE_ATTRIBUTES="service.version=lab,team=ai-platform,cost.center=engineering"

openlit-instrument python your_app.py

OpenLIT docs warn that database parameter capture can expose sensitive values; it should be a lab-only setting unless policy explicitly allows it.

Protocol Harnesses: AG-UI and A2A

AG-UI and A2A do not have a single “turn telemetry on” host config. They are protocol/event contracts. To make them observable, the application must attach trace context and preserve protocol IDs:

{
  "traceparent": "00-TRACEID-SPANID-01",
  "agui": {
    "runId": "run_hash",
    "messageId": "message_hash",
    "toolCallId": "tool_call_hash"
  },
  "a2a": {
    "taskId": "task_hash",
    "contextId": "context_hash",
    "messageId": "message_hash"
  }
}

AG-UI docs define lifecycle, text message, tool-call, state, activity, reasoning, raw, and custom events. Use those IDs as join keys; do not expect AG-UI alone to prove model internals or local executor side effects.

Claude Code OTel Enablement

CLAUDE_CODE_ENABLE_TELEMETRY=1
OTEL_METRICS_EXPORTER=otlp
OTEL_LOGS_EXPORTER=otlp
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector.internal:4318
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer REDACTED
OTEL_LOG_USER_PROMPTS=1
OTEL_LOG_TOOL_DETAILS=1
OTEL_LOG_TOOL_CONTENT=1
OTEL_METRIC_EXPORT_INTERVAL=10000

Claude Code Event-Like OTel Log

{
  "resource": {
    "service.name": "claude-code",
    "service.version": "redacted"
  },
  "scope": "claude-code",
  "body": {
    "event.name": "claude_code.tool.result",
    "session.id": "sess_hash",
    "tool.name": "Bash",
    "tool.input.hash": "sha256:...",
    "tool.output.hash": "sha256:...",
    "content.redacted": true
  },
  "attributes": {
    "user.account_hash": "user_hash",
    "workspace.path_hash": "path_hash",
    "privacy.prompt_logged": true,
    "privacy.tool_details_logged": true,
    "privacy.tool_content_logged": true
  },
  "time_unix_nano": 1780847584000000000
}

Codex OTel Config

[otel]
log_user_prompt = true
metrics_exporter = "otlp-http"
trace_exporter = "otlp-http"
exporter_otlp_endpoint = "http://collector.internal:4318"
exporter_otlp_headers = { Authorization = "Bearer REDACTED" }
exporter_otlp_protocol = "http/protobuf"

Codex Local JSONL-Like Session Row

{
  "type": "response_item",
  "thread_id": "thread_hash",
  "timestamp": "2026-06-07T13:02:10.481Z",
  "item": {
    "type": "function_call",
    "name": "exec_command",
    "call_id": "tool_call_hash",
    "arguments_hash": "sha256:...",
    "arguments_redacted": true
  }
}

Codex Local SQLite Log Row

{
  "timestamp": "2026-06-07T13:02:10.481Z",
  "thread_id": "thread_hash",
  "process_uuid": "process_hash",
  "target": "codex_core::exec",
  "level": "INFO",
  "body_hash": "sha256:...",
  "module_path": "codex_core::exec",
  "file": "redacted",
  "line": 123
}

Bedrock Invocation Log

{
  "schemaType": "ModelInvocationLog",
  "accountId": "123456789012",
  "region": "us-east-1",
  "requestId": "aws_request_hash",
  "operation": "Converse",
  "modelId": "anthropic.claude-redacted",
  "input": {
    "inputBodyJsonHash": "sha256:...",
    "contentRedacted": true
  },
  "output": {
    "outputBodyJsonHash": "sha256:...",
    "contentRedacted": true
  },
  "metadata": {
    "inputTokenCount": 1234,
    "outputTokenCount": 456,
    "latencyMs": 789
  }
}

Important boundary: Bedrock invocation logging covers supported bedrock-runtime calls. AWS documents that calls through other endpoints, such as the bedrock-mantle Responses API endpoint, are not currently captured by Bedrock invocation logging.

Strands / AgentCore OTel Span

{
  "trace_id": "trace_hash",
  "span_id": "span_hash",
  "parent_span_id": "parent_hash",
  "name": "StrandsAgent.invoke",
  "kind": "SPAN_KIND_INTERNAL",
  "attributes": {
    "gen_ai.system": "bedrock",
    "gen_ai.request.model": "anthropic.claude-redacted",
    "session.id": "session_hash",
    "agent.name": "research_agent",
    "tool.name": "retrieve_sources",
    "cloud.account.id": "account_hash",
    "aws.region": "us-east-1"
  },
  "status": { "code": "OK" }
}

AG-UI Event

{
  "type": "TOOL_CALL_START",
  "runId": "agui_run_hash",
  "messageId": "msg_hash",
  "toolCallId": "tool_call_hash",
  "toolCallName": "search_kb",
  "parentMessageId": "assistant_msg_hash",
  "timestamp": "2026-06-07T13:02:10.481Z"
}

A2A Task Event

{
  "jsonrpc": "2.0",
  "id": "rpc_hash",
  "method": "message/send",
  "params": {
    "message": {
      "messageId": "msg_hash",
      "role": "user",
      "parts": [{ "kind": "text", "text_hash": "sha256:..." }],
      "contextId": "context_hash",
      "taskId": "task_hash"
    }
  }
}

OpenInference Span Attributes

{
  "name": "llm.chat",
  "attributes": {
    "openinference.span.kind": "LLM",
    "llm.provider": "openai",
    "llm.model_name": "gpt-redacted",
    "input.value": "{\"messages\":\"sha256:...\"}",
    "output.value": "{\"message\":\"sha256:...\"}",
    "llm.token_count.prompt": 1200,
    "llm.token_count.completion": 300,
    "session.id": "session_hash"
  }
}

ATIF-Like Trajectory

{
  "trajectory_id": "traj_hash",
  "task": {
    "task_id": "task_hash",
    "intent": "publish_agent_observability_research",
    "input_hash": "sha256:..."
  },
  "steps": [
    {
      "index": 0,
      "type": "llm",
      "span_id": "span_hash",
      "input_hash": "sha256:...",
      "output_hash": "sha256:..."
    },
    {
      "index": 1,
      "type": "tool",
      "tool_name": "web.search",
      "arguments_hash": "sha256:...",
      "result_hash": "sha256:..."
    }
  ],
  "outcome": {
    "status": "passed",
    "artifact_id": "artifact_hash",
    "evaluator": "human_and_build"
  }
}

LangSmith Run Tree

{
  "id": "run_hash",
  "trace_id": "trace_hash",
  "parent_run_id": null,
  "name": "research_and_publish",
  "run_type": "chain",
  "start_time": "2026-06-07T13:02:10.481Z",
  "end_time": "2026-06-07T14:20:38.000Z",
  "status": "success",
  "inputs": { "request_hash": "sha256:..." },
  "outputs": { "artifact_url": "https://stateofai.pages.dev/..." },
  "extra": {
    "metadata": {
      "thread_id": "thread_hash",
      "ls_provider": "openai",
      "ls_model_name": "gpt-redacted",
      "task_family": "research_publish"
    }
  },
  "events": [],
  "tags": ["agent-observability", "published"],
  "feedback_stats": { "correctness": { "passed": 1 } },
  "child_runs": [
    {
      "id": "child_run_hash",
      "parent_run_id": "run_hash",
      "run_type": "tool",
      "name": "search_docs",
      "inputs": { "query_hash": "sha256:..." },
      "outputs": { "result_count": 10 }
    }
  ]
}

LangSmith bulk export fields documented for Parquet include identifiers and hierarchy such as id, session_id, trace_id, parent_run_id, parent_run_ids, and reference_example_id; metadata such as name, run_type, start_time, end_time, status, is_root, dotted_order, and trace_tier; run data such as inputs, outputs, error, extra, and events; tags and feedback; and token/cost fields.

Query Cookbook

LangSmith SDK

List root runs:

from langsmith import Client

client = Client()
root_runs = client.list_runs(project_name="agent-prod", is_root=True)

Find runs in one conversation/thread:

thread_id = "thread_hash"
filter_string = (
    'and(in(metadata_key, ["session_id","conversation_id","thread_id"]), '
    f'eq(metadata_value, "{thread_id}"))'
)
thread_runs = client.list_runs(
    project_name="agent-prod",
    filter=filter_string,
    is_root=True,
)

Find retrieval spans inside successful traces:

runs = client.list_runs(
    project_name="agent-prod",
    filter='eq(run_type, "retriever")',
    trace_filter='and(eq(feedback_key, "correctness"), eq(feedback_score, 1))',
)

Search all string fields:

runs = client.list_runs(
    project_name="agent-prod",
    filter='search("missing policy exception")',
)

Warehouse Queries

Needle KB candidates:

select t.turn_id, t.question_hash, a.uri, t.retrieval_attempts, t.elapsed_ms
from turns t
join artifacts a on a.artifact_id = t.final_artifact_id
where t.retrieval_attempts >= 5
  and t.validation_status = 'passed'
  and a.artifact_type = 'citation';

Skill candidates:

select task_family, count(*) as validated_runs
from training_candidates
where candidate_type = 'skill'
  and validation_status = 'passed'
  and human_approved = true
group by task_family
having count(*) >= 3;

Reliability by tool:

select tool_name,
       count(*) as calls,
       avg(case when status = 'error' then 1 else 0 end) as error_rate,
       percentile_cont(0.95) within group (order by elapsed_ms) as p95_ms
from tool_calls
group by tool_name
order by error_rate desc, calls desc;

YAML failure coverage:

select schema_name, failure_code, count(*) as failures
from evaluations
where artifact_type = 'yaml'
  and evaluator_type = 'deterministic_validator'
group by schema_name, failure_code
order by failures desc;

Conversion Rules

Raw Chat To Skill

Do not train on raw chats. Convert a trace into a skill candidate only when:

  • the task family recurs;
  • the user intent and repo state can be summarized without sensitive content;
  • the tool sequence is stable enough to teach;
  • the final artifact passed a deterministic or human validation;
  • at least one counterexample explains when not to use the skill.

Trace To KB Defect

Use retrieval churn as the signal:

  • many search attempts;
  • many query rewrites;
  • late discovery of the right source;
  • repeated related questions across sessions;
  • answer found only through a synonym, acronym, owner name, stale title, or old path.

Trace To YAML Optimization

Keep:

  • schema version;
  • prompt version;
  • generated YAML hash or redacted body;
  • validator failure codes;
  • corrected YAML;
  • final pass/fail;
  • field-level coverage tags.

For most domain YAML tasks, start with 10-20 adversarial eval examples, then 30-100 prompt-optimization examples, then 100-300 coverage examples. Fine-tune only after schema-constrained prompting, retrieval, deterministic validation, and prompt optimization fail the reliability target.

Definitions

Term Operational definition
Trace One causality tree for a task/request/workflow; identified by trace_id
Span Timed operation inside a trace; identified by span_id and parent
Run LangSmith/platform execution object; root run is trace-like, child run is span-like
Rollout One concrete execution of an agent/workflow/prompt/model configuration against a task, dataset row, or production request
Session Longer user-agent container spanning one or more traces
Thread Conversation grouping key, often used by chat UIs and LangSmith metadata
Turn User-visible exchange, potentially containing many spans/tool calls
Message Role-tagged conversational content
Event Point-in-time lifecycle/status/streaming record
Tool call Model/agent request to external capability plus result/error
Artifact File, structured output, citation, screenshot, dataset row, or report
Trajectory Ordered action/observation episode with terminal outcome
Evaluation Quality label or score linked to run, turn, artifact, or dataset example
Feedback Human or model-provided evaluation signal, often attached after execution
Dataset example Curated input/reference/output row used for evaluation or few-shot prompting
Benchmark Named evaluation protocol with task set, runner, scorer, config, and report conventions
Golden set Small trusted eval set with reviewed expected behavior, used for frequent regression checks
Experiment Controlled comparison across variants such as prompt, model, retrieval config, tool policy, or agent architecture
Prompt version Stable identifier for the exact prompt/template/instruction used in a rollout
Dataset Versioned collection of examples with schema, split, provenance, and intended use
Example One dataset row containing input, reference output/label, metadata, and optional trace/artifact refs
Split Dataset partition such as train, validation, test, golden, canary, or holdout
Holdout Examples withheld from training and prompt optimization for unbiased evaluation
Canary set Small high-risk eval subset run before broader benchmark or production promotion
Workload Stable logical operation or agent node: tool routing, retrieval, summarization, YAML generation, citation formatting
Lineage Provenance chain from raw event through dataset example, training job, model artifact, rollout, and eval
Dedupe Exact/fuzzy/semantic duplicate removal
Contamination Leakage of benchmark/golden/test content into training, prompt selection, retrieval, or judging
Human-in-the-loop Human review/approval/labeling before promotion
Judge Human, deterministic validator, or model evaluator that scores an output
Reward model Model used to score/rank outputs for filtering, preference learning, or RL-style optimization
Preference pair Candidate outputs for the same input with preference/ranking labels
SFT Supervised fine-tuning on instruction/output examples
PEFT / LoRA Parameter-efficient fine-tuning; LoRA creates a small adapter rather than a full checkpoint
Adapter Fine-tuned parameter delta mounted on a base model
Model promotion Approval and deployment gate for prompt/model/adapter candidates
Training candidate Dataset example that passed redaction, licensing, labeling, and split review

Rollouts, Evals, Benchmarks, and Golden Sets

Rollout is the unit that ties observability to improvement. A rollout records that a specific agent configuration ran on a specific task at a specific time. It should carry:

  • rollout_id;
  • prompt_version;
  • model_provider and model_id;
  • tool policy and retrieval config;
  • dataset/example/task ID when applicable;
  • root trace_id;
  • produced artifact IDs;
  • eval labels and benchmark scores.

Evals attach quality evidence to a rollout. Benchmarks define repeatable collections of evals. Golden sets are the stable high-signal subset used for fast regression checks. Training candidates should only be created after a rollout has enough provenance, redaction, and eval signal to justify reuse.

NVIDIA Data Flywheel / NeMo Mapping

NVIDIA’s Data Flywheel blueprint was deprecated in April 2026. That changes the implementation recommendation: do not build production systems on the archived reference blueprint. It does not invalidate the terminology or architecture pattern. The enduring pattern is production traffic -> workload-tagged logs -> curated datasets -> base/ICL/customized evals -> adapter/model candidates -> human promotion.

In current production planning, treat Data Flywheel as a reference pattern and map it to NeMo Microservices, OTel/OpenInference, a warehouse, and your eval harness.

NVIDIA term Standard term Operational meaning
Data flywheel Rollout/eval/training-candidate loop Production logs become curated datasets, experiments, and model/prompt candidates
Production traffic Raw events, turns, tool calls Real app interactions captured for replay and curation
Log Store Raw vault / raw_events Storage for exact request/response records before curation
workload_id workload, task_family, agent.node_id Stable logical operation key; essential for separating agent nodes
client_id app/deployment/tenant ID Identifies source application or customer context
NeMo Datastore Dataset/artifact registry Stores eval datasets, fine-tuning datasets, and LoRA artifacts
NeMo Customizer Training/fine-tuning job Runs SFT/LoRA customization and creates adapter artifacts
NeMo Evaluator Eval runner/judge Scores candidate outputs or models
Deployment Manager / DMS Runtime orchestration Starts/stops candidate NIM deployments for experiments
NIM Serving runtime Hosts base or adapted model for inference/eval
Base eval Baseline benchmark rollout Unmodified model on held-out examples
ICL eval Few-shot benchmark rollout Same model with retrieved/selected in-context examples
Customized eval Fine-tuned benchmark rollout Adapter/customized model evaluated on held-out examples
Class-aware stratified split Split by workload/tool/class label Keeps minority classes represented in train/eval/holdout
LLM-as-judge Model evaluator Scores outputs; should be calibrated against golden/human/deterministic checks
Tool-calling correctness Tool-call eval metric Exact function plus required argument accuracy; semantic match only for long text fields
LoRA artifact Adapter artifact Fine-tuned parameter delta with lineage to base model and dataset
Model promotion Promotion gate Human/governed approval after eval, safety, cost, and regression review

The major curation risk is weak workload tagging. If an agent chain logs intent classification, retrieval, reranking, generation, tool routing, and YAML formatting under one workload ID, the flywheel will mix incompatible examples. That produces noisy training data and misleading evals. Each agent node needs a stable workload label, and every dataset example should keep lineage back to the rollout, trace, turn, tool call, prompt version, and artifact.

Anthropic-Style Agent Eval Mapping

Anthropic’s “Demystifying evals for AI agents” sharpens the distinction between simple prompt evals and agent evals. Single-turn evals grade a prompt/response. Agent evals give the system a task, tools, and an environment, then grade the resulting state, artifacts, tool behavior, and trajectory. That is the reason our schema keeps rollouts, traces, turns, tool calls, artifacts, and evals separate.

For our domains:

Domain Benchmark construction Scoring
Skill mining Replay similar validated traces and counterexamples Did the skill reduce tool churn, avoid known failure modes, and preserve validation?
Corporate KB needles Use hard but verifiable questions with hidden aliases, old names, owners, and permission edge cases Correct citation, fewer retrieval attempts, lower elapsed time, no fabricated source
YAML generation Build schema fixtures with valid, invalid, ambiguous, and adversarial cases Deterministic schema validator plus semantic/business-rule checks
Finance/source acquisition Require primary-source ledgers, citation anchors, and published artifacts Source correctness, build/search verification, no unsupported claims
Tool routing Stratify by function/tool class and include rare tools Exact function name, required argument accuracy, semantic match only where appropriate
Coding/agent tasks Provide repo/environment fixtures with executable checks Tests, file state, browser/UI checks, resource usage, unnecessary edits

Eval design implications:

  • use golden sets for fast regression;
  • use broader benchmarks for release decisions;
  • use production monitoring for drift and unseen failures;
  • document infrastructure and resource configuration as benchmark variables;
  • calibrate LLM judges against deterministic checks and human labels;
  • track trajectory costs such as extra turns, failed tools, retries, latency, and token spend.

Current NeMo Microservices Impact

The current NeMo Microservices stack changes the production implementation option, not the open observability format. It gives NVIDIA ecosystem services for pieces of the lifecycle:

Service/concept Role Observability mapping
Entity Store / Data Store Entity, dataset, file, and artifact storage dataset IDs, artifact IDs, lineage refs
Data Designer Synthetic data and data design workflows generated examples, curation labels, synthetic provenance
Customizer Fine-tuning/customization training job spans, dataset version, base model, adapter artifact
Evaluator Model/app/agent evaluation EVALUATOR spans, eval rows, benchmark/golden-set results
Deployment Management Candidate deployment orchestration deployment spans, NIM endpoint IDs, resource/runtime metadata
NIM Inference serving LLM spans with model/endpoint/version/latency/cost attributes
Guardrails / Safe Synthesizer Safety and controlled generation GUARDRAIL spans, policy version, rejection/rewrite labels

NeMo Microservices are useful if the organization wants self-hosted or NVIDIA-enterprise customization/evaluation/deployment. They do not replace OpenTelemetry/OpenInference, LangSmith/Phoenix/Arize trace views, Bedrock invocation audit logs, or local Claude/Codex transcript capture.

OpenInference Format and Model Architectures

OpenTelemetry is the transport/correlation layer. OpenInference is the semantic layer that says what each span represents in an AI system.

Required core idea:

{
  "trace_id": "trace_hash",
  "span_id": "span_hash",
  "parent_span_id": "parent_hash",
  "name": "retriever.search",
  "attributes": {
    "openinference.span.kind": "RETRIEVER",
    "session.id": "session_hash",
    "input.value": "sha256:redacted",
    "output.value": "sha256:redacted"
  }
}

Useful span kinds:

Kind Meaning Architecture examples
LLM Model generation Chat/completion, reasoning model, local SLM, NIM, Bedrock, Claude, OpenAI
EMBEDDING Vectorization RAG embedding, semantic memory, clustering, dedupe
RETRIEVER Candidate context fetch BM25, vector DB, SQL, graph query, KB search, web search
RERANKER Candidate reorder/prune Cross-encoder, ColBERT, LLM relevance classifier, NVIDIA reranker
TOOL External action Shell, browser, MCP, API, file edit, database, M365
CHAIN Orchestration Router, prompt builder, parser, formatter, validator
AGENT Agent loop/node Claude/Codex rollout, LangGraph node, Strands agent, A2A delegate
GUARDRAIL Safety/policy Permission, content filter, NeMo Guardrails, policy engine
EVALUATOR Scoring LLM judge, deterministic validator, human label ingestion, benchmark scorer
PROMPT Prompt assembly/versioning System prompt, few-shot selector, template version

Model architecture mappings:

Architecture Typical OTel/OpenInference trace shape
Single-call chatbot rollout -> LLM
RAG rollout -> RETRIEVER -> RERANKER -> PROMPT -> LLM -> EVALUATOR
Tool-calling agent rollout -> AGENT -> repeated LLM / TOOL pairs -> EVALUATOR
Graph agent rollout -> AGENT -> node-level CHAIN or AGENT spans -> nested model/tool spans
Router / MoE / model cascade rollout -> router CHAIN -> selected LLM expert/model spans -> evaluator/cost spans
Fine-tuned SLM rollout -> LLM span with base model, adapter ID, dataset version, prompt version
Data-flywheel candidate eval rollout -> base/ICL/customized LLM spans -> EVALUATOR spans -> promotion decision artifact
Guardrailed system rollout -> pre GUARDRAIL -> model/tool spans -> post GUARDRAIL -> evaluator
Protocol agent UI rollout -> AG-UI protocol spans/events -> agent/model/tool spans
Agent-to-agent delegation rollout -> A2A task span -> delegated agent trace linked by taskId/trace context

The design point is that traces should reflect the architecture. A RAG trace without retrieval/reranking spans hides the actual failure modes. A tool agent trace without tool-call spans is not replayable. A fine-tuned adapter trace without dataset and adapter lineage cannot support model promotion decisions.

Cross-Vendor Ontology Map

Other frontier labs and enterprise AI platforms have similar lifecycle services, but their terms differ. The open standard should absorb their nouns rather than copy any single vendor taxonomy.

Platform Build/runtime terms Trace/eval terms Curation/customization terms Standard mapping
NVIDIA NeMo NIM, Deployment Management, Guardrails Evaluator Data Store, Data Designer, Customizer, LoRA artifact LLM, GUARDRAIL, EVALUATOR, dataset, adapter, deployment
OpenAI Agents SDK, tools traces, trace grading, graders, agent evals datasets, fine-tuning/custom models rollout, TOOL, EVALUATOR, benchmark, training candidate
Anthropic Claude Agent SDK, Claude Code, Managed Agents, Skills OTel, sessions, prompts, tool results, empirical evals skills, prompt/test cases, golden evals session, skill, tool call, rollout, golden set
Google Vertex / Gemini Agent Engine, ADK, Agent Runtime/Registry/Gateway/Identity Gen AI evaluation, metrics, eval datasets Model Garden, tuning/customization deployment, identity, dataset, eval, benchmark
AWS Bedrock AgentCore Runtime, Gateway, Memory, Browser, Code Interpreter, Policy Observability, online evals, CloudWatch eval results Optimization, prompt optimization, model customization trace, tool call, policy/guardrail, eval, promotion loop
Microsoft Foundry Agent Service, external agents, Semantic Kernel, Agent Framework OTel tracing, Application Insights, trace-to-dataset, evals datasets, evaluators, model catalog OTel trace, dataset, eval, external-agent lineage
Mistral Le Chat/agents/API Explorer, Judges Campaigns, Datasets, fine-tuning raw event explorer, judge, annotation campaign, dataset
Databricks Mosaic AI Agent Framework, model serving, Vector Search MLflow traces, scorers, monitoring evaluation datasets, Unity Catalog lineage trace, scorer/eval, retriever, artifact lineage
IBM watsonx Orchestrate agents, tools, ADK, control plane monitoring, governance, pre-deployment evaluation policy/evaluation workflows agent registry, tool calls, guardrails, eval gates
LangSmith LangChain/LangGraph app runs runs, child runs, feedback, experiments datasets trace/span/run, feedback/eval, benchmark
Phoenix/Arize OpenInference instrumentation spans, traces, annotations, experiments datasets/evals open trace/eval semantic layer

The shared lifecycle is:

observe production traces
  -> mine failures and representative examples
  -> create datasets / golden sets
  -> run evals / trace grading / judges
  -> compare variants
  -> optimize prompts, skills, tools, retrieval, or models
  -> promote through governance
  -> monitor drift and regressions

Normalization Rule

Normalize vendor terms into these canonical fields:

Canonical field Vendor aliases
rollout_id run, experiment run, session execution, agent execution, evaluation run
trace_id trace, run tree, Application Insights trace, Phoenix trace, LangSmith root run
span_id child run, step, operation, node, tool event
tool_call_id function call ID, tool use ID, action ID, protocol toolCallId
dataset_example_id example, eval case, row, task, sample
eval_id grader result, judge score, scorer output, feedback, annotation
prompt_version template version, skill version, system prompt hash, instruction version
model_artifact_id adapter, custom model, NIM endpoint, deployment, model version
policy_version guardrail config, policy, safety check, permissions profile

If a platform cannot emit those fields directly, add them at the collector or warehouse. That is the pragmatic difference between a vendor dashboard and a durable self-improvement system.

OpenTelemetry and OpenInference Mapping

OpenTelemetry should carry the transport and correlation contract. The AI semantic layer must be chosen and versioned: OpenInference for Phoenix/Arize semantics, OTel GenAI for gen_ai.* model/agent/tool conventions, and MCP semantic conventions for MCP protocol/tool spans. Vendor-native records should be preserved in raw_events, then converted into spans and curated tables.

Concept OTel / OpenInference / MCP representation Claude / Codex Bedrock / Strands / AgentCore AG-UI / A2A LangSmith / Phoenix
Rollout Root span plus rollout_id, prompt_version, dataset/task refs CLI session segment or desktop task episode AgentCore session or Strands agent execution UI runId or A2A delegated taskId LangSmith experiment run or Phoenix trace with eval metadata
Trace trace_id and root span Session/turn trace emitted by OTel or reconstructed from local logs OTel trace from Strands/AgentCore; Bedrock request joined by requestId Protocol run/task metadata attached to trace LangSmith root run; Phoenix trace
Span span_id, parent, timing, status LLM/tool/file/shell/retrieval operation Model/tool/retrieval span Lifecycle event promoted to span where useful LangSmith child run; Phoenix span
LLM openinference.span.kind=LLM; model/token attrs Assistant/model response step Bedrock invocation span and request metadata Usually app-layer only run_type="llm" or Phoenix LLM span
Tool OpenInference TOOL, OTel GenAI execute-tool span, or MCP method/tool span; input/output refs Bash/MCP/browser/file/function call Strands tool span; Bedrock payload if model requested tool AG-UI tool lifecycle; A2A task/message delegation run_type="tool" or Phoenix TOOL span after translation where needed
Retriever openinference.span.kind=RETRIEVER; document refs Search/read/source tools Bedrock KB or app retriever span Protocol metadata unless app emits retriever span run_type="retriever" or Phoenix RETRIEVER span
Turn turn_id, session/thread metadata, message events User prompt plus assistant/tool sequence App-layer conversation ID AG-UI messages; A2A messages LangSmith thread metadata; Phoenix session metadata
Eval Eval span/event or warehouse row Test/build/human label joined to trace Bedrock/APO/evaluator or external harness Protocol status plus separate scorer LangSmith feedback/eval; Phoenix annotation/eval
Memory event Memory read/write span or warehouse row Local memory/skill/session note if captured AgentCore/Managed Agent memory read/write Protocol state/memory only if exposed Trace metadata or dataset/eval row
Context event Compaction/offload/code-filter span or warehouse row with token counts CLI compaction, shell/code filtering, artifact refs Code Interpreter/browser/tool-output offload, memory context Protocol state snapshot/reduction Trace metadata or warehouse row
Golden set Dataset split and stable example IDs Curated local tasks/examples Bedrock prompt optimization samples or external eval rows E2E protocol replay cases LangSmith/Phoenix dataset split
Benchmark Experiment collection and scorer metadata StateBench/local harness Bedrock eval/APO or external harness Protocol conformance/e2e harness LangSmith/Phoenix experiment

Minimal OTel emission pattern:

with tracer.start_as_current_span("agent.rollout") as rollout:
    rollout.set_attribute("rollout.id", rollout_id)
    rollout.set_attribute("prompt.version", prompt_version)
    rollout.set_attribute("dataset.example_id", example_id)

    with tracer.start_as_current_span("llm.generate") as span:
        span.set_attribute("openinference.span.kind", "LLM")
        span.set_attribute("llm.provider", "bedrock")
        span.set_attribute("llm.model_name", model_id)

    with tracer.start_as_current_span("tool.call") as span:
        span.set_attribute("openinference.span.kind", "TOOL")
        span.set_attribute("tool.name", tool_name)
        span.set_attribute("tool.call_id", tool_call_id)

    with tracer.start_as_current_span("eval.score") as span:
        span.set_attribute("openinference.span.kind", "EVALUATOR")
        span.set_attribute("eval.name", "yaml_schema_validator")
        span.set_attribute("eval.passed", True)

OSS and China/Asia Ecosystem Gap Map

The standard should now track more than frontier labs and hyperscalers. The open-source and China/Asia ecosystems are important, but they should be mapped by contract strength rather than brand presence.

Detailed gap-ingest pages:

Ecosystem Contract level What it can add What must be verified
Langfuse Trace/eval-ready OTel-aligned tracing, LLM calls, tool invocations, retrieval steps, prompt management, datasets, evals, experiments, annotation, self-hosting Export fidelity, retention/cost model, exact tool-call input/output redaction controls
W&B Weave Eval/monitoring-ready LLM traces, online evaluations, experiment tracking, production scoring, model monitoring Raw trace export shape and whether local desktop-agent actions are captured or only instrumented app calls
Braintrust Eval-first trace-ready Trace hierarchy/timeline/conversation views, raw JSON trace view, tool spans, datasets, experiments, scorers, human review Cross-system trace propagation and non-SDK local transcript joins
Helicone Gateway/request-ready Request/response logging, cost/latency, routing, dashboards, alerting, prompt management, removal/redaction controls Multi-step agent/tool trajectory completeness unless paired with app OTel
OpenLIT / OpenLLMetry / Traceloop Instrumentation-ready OTel-based auto-instrumentation for model/framework calls; Phoenix can translate these conventions into OpenInference Semantic drift between OTel GenAI, OpenLLMetry/OpenLIT, and OpenInference; tool-call ID consistency
DeepEval / Ragas / TruLens / Promptfoo Evaluation-ready RAG checks, prompt tests, model/scorer evals, CI gates, regression suites Dataset lineage, scorer calibration, stable IDs joined to traces and artifacts
NVIDIA AI-Q Blueprint Reference architecture Phoenix/Weave-style backends, tracing of agent execution, LLM calls, tool invocations, token usage, and dataset construction from traces Whether the blueprint’s config maps cleanly to your production OTel/OpenInference collectors
AgentScope / Alibaba ecosystem Promising trace bridge AgentScope Studio traces, token usage metadata, tracing_url export to Phoenix, Langfuse, Alibaba CloudMonitor, or any OTLP endpoint Exact emitted span attributes, tool-call payload fields, redaction gates, and production support status
Tencent Cloud ADP Product lifecycle/eval workflow Conversation history, app analytics, Q&A optimization, workflow/multi-agent mode, knowledge base, plugin marketplace, evaluation set export/upload Public raw trace schema, OTLP/OpenInference export, stable tool-call IDs, cost/retention terms
Baidu Qianfan Product lifecycle Agent engine, tools/MCP, model services, enterprise capabilities Public observability/export contract and eval dataset schema
Volcengine ADK / Ark Trace-promising Trace IDs, GenAI attributes, multi-agent tracing, SLI-driven full-chain observability articles Raw event format, OTLP collector behavior, tool-call redaction, English contract availability
Huawei ModelArts / Pangu AI lifecycle/monitoring Data preprocessing, labeling, training, deployment, monitoring metrics, large-model app/agent development Agent-specific trace/tool-call schema and export API
Coze Agent builder lifecycle Workflows, plugins, knowledge, app publishing, bot/agent construction Logs, trace IDs, eval set export, tool-call payloads, warehouse/API export
Moonshot/Kimi / DeepSeek Model/API provider Model access, agent-task capable models, web/tool features, pricing and request APIs Separate agent observability, trace export, and eval-loop contracts

Contract-Strength Checklist

Do not treat an “agent platform” as production-observable until it passes these checks:

Requirement Standard field/table Why it matters
Stable trace identity trace_id, span_id, parent_span_id in agent_spans Enables cross-service causality and flame-chart/timeline debugging
Stable human/task grouping session_id, thread_id, turn_id, rollout_id in turns Prevents a trace tree from losing the user-visible episode
Tool-call pairing tool_call_id, request/result/error/approval in tool_calls Makes local reliability and safety metrics possible
Artifact lineage artifact_id, artifact_type, source refs in artifacts Required for YAML generation, citations, screenshots, diffs, and produced files
Eval conversion dataset_example_id, eval_id, scorer/version fields Turns logs into golden sets, benchmarks, prompt optimization, and model training inputs
Export path JSONL, Parquet, OTLP, OpenInference, LangSmith-compatible run export, or warehouse API Prevents dashboard lock-in
Privacy controls redaction_version, content gates, hashes, byte counts, refs Lets teams separate hot debugging from safe long-term mining
Cost controls sampling, retention, payload/body toggles, archive tier Prevents full-content logs from dominating operating cost

Underdeveloped Standards

Several pieces are still immature across the ecosystem:

  • tool_call_id is not consistently propagated from UI protocol to model call to local executor to eval result.
  • State deltas, memory reads/writes, approvals, and permission decisions are not first-class enough in OTel/OpenInference.
  • Trace-to-dataset conversion is product-specific. LangSmith, Braintrust, Weave, Phoenix/Arize, Langfuse, and eval libraries each expose useful paths, but there is no single open “curated agent example” contract.
  • Chinese cloud platforms often expose agent lifecycle and analytics features before publishing enough raw trace/export detail for cross-vendor standardization.
  • X/YouTube/community material is good for finding failure modes, but should be converted into testable eval labels before it drives architecture decisions.

Source Notes

  • LangSmith OpenTelemetry docs map OTel attributes such as langsmith.trace.name, langsmith.span.kind, langsmith.trace.session_id, langsmith.trace.session_name, langsmith.span.tags, and langsmith.metadata.* into LangSmith fields.
  • LangSmith trace query docs support filtering by project, trace, run type, dataset/example, parent run, error state, root runs, selected fields, root trace filters, tree filters, metadata, tags, feedback, full-text search, and natural-language query.
  • LangSmith dataset docs describe dataset versioning, tagged versions, splits, CSV/JSONL/OpenAI fine-tuning format export, and exporting filtered traces from experiments to datasets.
  • LangSmith few-shot evaluator docs describe using corrected evaluator feedback to populate a corrections dataset and insert examples into evaluator prompts.
  • Phoenix/OpenInference docs describe OpenInference as OTel-compatible semantic conventions and instrumentation for LLM apps, including integrations for Claude Agent SDK, Bedrock, DSPy, LangChain, MCP, Anthropic, Strands, and others.
  • Phoenix translation docs show converting OpenLIT, OpenLLMetry, and OpenTelemetry GenAI attributes into OpenInference format for Phoenix.

Sources

Brandon Sneider | brandon@brandonsneider.com June 2026