← Agent Frameworks 🕐 3 min read
Agent Frameworks

Agent Telemetry Config Verification Harness 2026

Brandon Sneider · June 2026

> **Source credibility: HIGH for config locations and published keys; MEDIUM

See also (wiki): wiki/agent-operations-observability.md · wiki/semantic-observability.md · wiki/mlops-ai-platform-engineering.md

Source credibility: HIGH for config locations and published keys; MEDIUM for end-to-end field claims until collector samples are captured. This page converts the copy/paste telemetry appendix into a test plan. It explicitly separates docs-verified from collector-verified.

Executive Summary

The current observability article now has copy/paste configs, but the next bar is field verification. A config is not “working” because a docs page says it exists. It is working only when a pinned tool version sends the expected events to a collector and the raw payload contains the fields the article claims.

The harness should produce three artifacts per tool:

  1. tool version and effective config path;
  2. raw OTLP/export/native log sample, redacted;
  3. field checklist showing present, absent, gated, or unsupported fields.

Harness Architecture

agent/tool under test
  -> local OTLP collector
  -> file exporter/raw capture
  -> redaction scanner
  -> schema checklist
  -> markdown report + source ledger

Minimum collector:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:4317
      http:
        endpoint: 127.0.0.1:4318

processors:
  batch: {}

exporters:
  debug:
    verbosity: detailed
  file/traces:
    path: /tmp/agent-otel-traces.json
  file/logs:
    path: /tmp/agent-otel-logs.json
  file/metrics:
    path: /tmp/agent-otel-metrics.json

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, file/traces]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, file/logs]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, file/metrics]

Versioned Test Matrix

Harness Local version checked Test action Fields to verify
Claude Code 2.1.150 one prompt, one Bash/tool call, one denied or approved tool call if possible trace_id, span_id, claude_code.interaction, claude_code.llm_request, claude_code.tool, tool_use_id, gen_ai.tool.call.id, prompt gate, tool details gate, tool content event
Claude Agent SDK not sampled one Python query() and one TypeScript query() inside an active app OTel span app root span, propagated TRACEPARENT, child claude_code.interaction, claude_code.llm_request, claude_code.tool, claude_code.hook, session.id, end-user resource attributes
Codex CLI 0.135.0 one interactive run, one codex exec, one MCP/server-related action if practical codex.conversation_starts, codex.api_request, codex.sse_event, codex.user_prompt, codex.tool_decision, codex.tool_result, trace exporter output, metrics exporter output
Codex Desktop / IDE not sampled one desktop/app task and one IDE task whether user-level ~/.codex/config.toml OTel is honored, app-server/local DB paths, tool-event shape
Bedrock invocation logging account dependent one Converse request with payload logging to S3 and CloudWatch request ID, operation, model ID, input/output body/log pointers, token counts, latency, no local tool execution proof
Strands / AgentCore not sampled one Python StrandsTelemetry run and one TypeScript setupTracer() run with a tool call and optional retrieval agent span, event-loop cycle span, model invocation span, token/cache attributes, gen_ai.tool.name, gen_ai.tool.call.id, tool.status, tool result, local JSONL exporter output, Bedrock request join keys
Langfuse/Braintrust/Weave/OpenLIT backend dependent send one OpenAI/Anthropic tool-call trace through each backend trace ID, tool-call display, raw export, dataset/eval conversion path
AG-UI / A2A app dependent emit one run/task with tool-call lifecycle and state delta protocol IDs, trace context propagation, task/run/message/tool IDs, artifact IDs

Verification Vocabulary

Status Meaning
Docs verified Official docs or code reference the key/config/field
Locally version-checked Installed binary reports a version; no ingestion claim yet
Collector verified Raw OTLP/export payload captured and inspected
Field verified Specific expected field appears in raw payload for the pinned version
Gated Field appears only when a content/privacy gate is enabled
Unsupported Tool/version does not emit the field
Product-only Product page describes workflow but no public export schema proves it

Required Report Format

For each tool:

## [Tool] [Version]

**Config path:** ...
**Command:** ...
**Collector:** ...
**Source docs:** ...

| Field | Status | Evidence |
|---|---|---|
| trace_id | field verified | `/tmp/agent-otel-traces.json`, span 1 |
| tool_call_id | gated | present when `OTEL_LOG_TOOL_DETAILS=1` |
| tool output body | gated | present when `OTEL_LOG_TOOL_CONTENT=1`, redacted before publish |

Source Notes

  • Claude Code current docs document enhanced beta traces, content gates, span names, tool IDs, prompt redaction, and managed settings paths.
  • Claude Agent SDK observability docs document W3C trace-context propagation from app spans into the Claude Code child process, nested agent/tool/hook spans, session.id, and end-user resource-attribute guidance.
  • Strands tracing docs document the strands-agents[otel] extra, standard OTel env vars, OTEL_SEMCONV_STABILITY_OPT_IN, StrandsTelemetry, OTLP and console exporters, TypeScript setupTracer, model/tool/retriever spans, and tool-call attributes.
  • OpenAI Codex current docs document CODEX_HOME, user and project config precedence, project-local otel ignore behavior, and OTel log/trace/metrics exporter keys.
  • OpenAI’s Codex safety article documents the intended security use of Codex telemetry: prompts, approval decisions, tool results, MCP usage, and network proxy allow/deny decisions.

Sources

Brandon Sneider | brandon@brandonsneider.com June 2026