← Findings 🕐 13 min read
Findings

How to Get Agent Logs That Actually Matter

Brandon Sneider · June 2026

> **Status: superseded synthesis.** This page is the first-pass narrative on why

Status: superseded synthesis. This page is the first-pass narrative on why agent logs matter. For the current canonical recommendation, use Agent Observability Standardization Playbook. For current copy/paste telemetry configs and schema details, use Agent Observability Format Standard 2026.

Executive Summary

  • There is no single “agent log.” Bedrock invocation logs, CloudTrail, Claude Code/Cowork OTel, Codex OTel, Microsoft Scout session history, AG-UI events, A2A task messages, and Phoenix/OpenInference traces each see a different layer. The useful system is a joined ledger, not a single sink.
  • Bedrock invocation logs are model-call audit logs, not coder-agent replay logs. They can capture request/response data, metadata, model ID, request ID, and account/region context for supported bedrock-runtime calls. They do not reconstruct local shell commands, file edits, browser actions, MCP calls, approvals, or agent reasoning state. AWS also documents a critical current gap: calls through endpoints such as the bedrock-mantle Responses API endpoint are not captured by Bedrock invocation logging.
  • Claude Code has the clearest native coder-agent OpenTelemetry contract. It exports metrics, logs/events, and optional traces, with privacy gates for prompts and tool details/content. Claude Cowork has a separate organization-level OTel path that includes prompt and tool activity when an admin configures an OTLP endpoint.
  • Codex has real OTel configuration, but the Desktop/CLI distinction needs live validation. Official Codex docs expose otel.* config, including otel.log_user_prompt, metrics exporter, and trace exporter. Locally, Codex stores additional operational state in ~/.codex SQLite, session indexes, shell snapshots, skills, and memories. The emitted event schema is less publicly explicit than Claude Code’s.
  • Microsoft Scout should be treated as a desktop operations agent, not merely a coding assistant. Current Microsoft docs describe a preview desktop app that can act on local files, run shell commands, control a browser, access Microsoft 365 data, delegate to subagents, run Heartbeat background tasks, and search past sessions. The open research question is its exportable telemetry contract.
  • The highest-value logging outcome is not a dashboard. The real opportunity is converting high-quality traces into reusable skills, corporate knowledge-base retrieval tasks, regression evals, prompt-optimization datasets, and, when justified, fine-tuning data for narrow domain tasks such as structured YAML generation.

The Logging Stack: What Each Layer Proves

Layer What it captures What it does not prove
Local transcript/session store User prompts, assistant messages, local tool traces, task boundaries, sometimes attachments and file-history Cloud identity, AWS request IDs, retention controls, or whether logs are complete after crashes
OpenTelemetry logs/events Structured user prompt, tool result, decision, timing, status, cost/token, and correlation events Full replay unless content gates and local artifacts are retained
OpenTelemetry traces Span tree for model calls, tool calls, agent steps, handoffs, and custom spans Ground-truth quality or semantic correctness
Cloud model invocation logs Model request/response payload and invocation metadata for supported model endpoints Local agent actions, shell commands, browser actions, approvals, or agent task state
CloudTrail / admin audit IAM principal, API action, account, region, and control-plane/runtime audit context Prompt semantics, model output quality, or local filesystem behavior
Protocol event streams AG-UI user-interface lifecycle, A2A task/status/artifact messages, MCP tool calls End-to-end correctness or whether the model made the right decision
Eval/annotation layer Human or judge labels, failure modes, accepted/rejected outputs, benchmark scores Raw provenance unless linked back to trace/session IDs

The failure mode is assuming one row replaces another. A Bedrock invocation log can tell you that a model call happened. It cannot tell you which local command Claude Code ran afterward. A Claude Code OTel event can tell you that a tool result occurred. It cannot tell you the AWS IAM identity behind a Bedrock call unless you attach or join that metadata. A Phoenix trace can show a beautiful span tree. It still needs labels and validation to become reliable training data.

Platform Notes

Claude Code

Claude Code is the best-documented native coder-agent OTel source today. Official docs describe OpenTelemetry metrics, logs/events, optional traces, and explicit content gates:

  • CLAUDE_CODE_ENABLE_TELEMETRY=1 enables telemetry.
  • OTEL_METRICS_EXPORTER, OTEL_LOGS_EXPORTER, OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_EXPORTER_OTLP_ENDPOINT, and OTEL_EXPORTER_OTLP_HEADERS configure export.
  • OTEL_LOG_USER_PROMPTS, OTEL_LOG_TOOL_DETAILS, and OTEL_LOG_TOOL_CONTENT are sensitive-content opt-ins.

The important operational caveat: Claude Code does not automatically pass OTEL_* variables to subprocesses spawned through Bash, hooks, MCP servers, or language servers. If an instrumented subprocess should export telemetry, configure it directly. Recent docs also describe trace context propagation for subprocess spans through TRACEPARENT.

Local Claude state is valuable but sensitive. On this machine, global ~/.claude/settings.json already points OTLP logs/metrics to http://localhost:4318, while project .claude/settings.json is primarily a permission policy. ~/.claude/projects/<project>/*.jsonl contains transcript-like records; that is useful for mining skills, but it should be treated as a local raw corpus until redacted.

Claude Cowork

Claude Cowork is separate from Claude Code. Cowork OTel is configured at the organization/admin level. Current docs say Cowork events include a prompt.id that links all events triggered by one user prompt, plus standard attributes such as session ID, organization ID, account identifiers, user email, workspace host paths, terminal type, and service metadata.

Cowork exports user prompt events and tool result events when an admin configures an OTLP endpoint. The privacy boundary is sharper than many teams expect: Cowork monitoring docs say prompt content is included, tool inputs can include file paths, URLs, search patterns, and other arguments, and user email appears in attributes. Redaction should happen at the collector or backend before broad access.

Anthropic’s Help Center also notes that Cowork activity is not currently captured in the Compliance API, but shared user account identifiers can correlate Cowork OTel events with longer-term Compliance API records.

Codex CLI and Codex Desktop

Official Codex docs expose a real OTel configuration surface:

  • otel.log_user_prompt opts into exporting raw user prompts with OTel logs.
  • otel.metrics_exporter selects none, statsig, otlp-http, or otlp-grpc.
  • otel.trace_exporter selects none, otlp-http, or otlp-grpc.
  • exporter endpoints, static headers, protocol, and TLS certificate paths can be configured.

Codex config precedence matters. Official docs say Codex loads CLI flags first, then trusted project .codex/config.toml, then profile config, then user ~/.codex/config.toml, then system config, then defaults. Locally, this repo has no project .codex; Codex is using global ~/.codex/config.toml, with /Users/pierre/consulting/stateofai marked trusted.

The Desktop nuance needs live validation before any enterprise claim. A public GitHub issue reports that Codex CLI honored project-scoped OTel config while Codex Desktop/App Server only emitted when OTel config was moved to global ~/.codex/config.toml. A second issue reports that Codex telemetry behavior can vary by entrypoint, with exec and mcp-server not behaving like the interactive CLI in the reporter’s test matrix. Treat these as practical warnings, not final contracts, until reproduced on the installed build.

Local Codex data is more distributed than Claude’s JSONL transcript path. On this machine, ~/.codex/logs_2.sqlite has a logs table keyed by timestamp, thread ID, process UUID, target, level, and log body; ~/.codex/session_index.jsonl has thread IDs and names; shell snapshots, skills, memories, plugin cache, and state SQLite files supply additional context.

Microsoft Scout

Microsoft Scout is currently preview documentation, but it should be included because its shape is exactly the future enterprise agent problem. Microsoft describes Scout as a desktop AI application that acts on files, runs shell commands, controls a browser, and works with Microsoft 365 data. It can:

  • read, write, and search workspace files;
  • use MCP filesystem tools;
  • run shell commands with auto-approve, prompt, and deny tiers;
  • automate a browser with Playwright tools;
  • read console logs and inspect network requests;
  • use m365_* tools and WorkIQ for Microsoft 365 data;
  • delegate to subagents;
  • run Heartbeat background prompts;
  • maintain memories and search past sessions;
  • track Microsoft Information Protection sensitivity labels.

That is an observability-heavy surface. The unresolved research question is not whether Scout performs consequential actions; the docs say it does. The unresolved question is what logs, exports, admin APIs, local session stores, and OTel-like telemetry Microsoft exposes for audit, mining, and replay.

Bedrock, AgentCore, and Strands

Bedrock invocation logging is essential for model-call audit. AWS says invocation logging can collect model input data, model output data, and metadata for supported calls in an account and Region, with CloudWatch Logs and S3 as destinations. It is disabled by default and covers Converse, ConverseStream, InvokeModel, and InvokeModelWithResponseStream.

The nuance that must not be lost: AWS documents that invocation logging only supports calls through the bedrock-runtime endpoint. Calls through other endpoints, including the bedrock-mantle Responses API endpoint, are not currently captured by invocation logging.

For prompt optimization, Bedrock Advanced Prompt Optimization is a separate JSONL format. Each line is one prompt template with fields such as version, templateId, promptTemplate, optional steering criteria, optional custom LLM-as-judge config, optional Lambda evaluator ARN, and evaluationSamples. The current documented limit is 1 to 100 evaluation samples per template. That makes it an optimization harness, not a large-scale fine-tuning dataset.

For fine-tuning/customization, Bedrock uses model-specific JSONL formats. AWS documents prompt/completion style records for non-conversational tasks and message-array formats for conversational tasks. Minimum records vary by model family; current docs include examples ranging from 5 records for some fine-tuning paths, 32 for others, and 100 or 1,000 for some model families. That variance matters for any “how many examples do we need?” claim.

AgentCore and Strands are the AWS-side answer to framework-level traces. Current AgentCore observability docs describe CloudWatch GenAI observability, ADOT/OTel instrumentation, opentelemetry-instrument, automatic instrumentation for Strands and Bedrock calls, and session correlation through OTel baggage such as session.id. Strands itself emits OTel traces for agent runs, with spans for significant actions. That is a different layer from Bedrock invocation logging: AgentCore/Strands traces can explain the agent workflow, while invocation logging audits supported model-runtime calls.

AG-UI, A2A, and Protocol Logs

AG-UI and A2A should not be treated as observability platforms. They are protocol surfaces. AG-UI gives event structure for agent-user interaction: run lifecycle, text message lifecycle, tool-related events, and state mutations. A2A gives agent-to-agent task delegation, task status, and artifact exchange. These events are valuable because they preserve user-interface and delegation boundaries that model-call logs miss.

The right pattern is to preserve protocol-native IDs and attach them to OTel spans and local transcript IDs. A protocol event without model/tool context is too shallow; a model trace without protocol context loses user-facing workflow boundaries.

Phoenix, OpenInference, and ATIF

Phoenix and OpenInference are the bridge between raw agent traces and reusable datasets. OpenInference defines semantic conventions for LLM, embedding, retrieval, tool, chain, and agent-like spans. Phoenix can ingest OTLP traces and render them with OpenInference semantics.

The new key format is ATIF: Agent Trajectory Interchange Format. Phoenix documentation now describes importing ATIF trajectories and converting them into OTel-compatible span trees. That matters because agent frameworks such as Claude Code, OpenHands, Gemini CLI, and Codex can be evaluated through trajectory harnesses that export ATIF-like records. ATIF is closer to the unit we need for agent replay than a single chat-completion span.

What To Mine From Logs

Self-Improving Skills

Do not create skills from clever prompts. Create skills from repeated, successful traces.

Mine for:

  • task family: source acquisition, wiki maintenance, PDF ingest, podcast ingest, code review, benchmark run, YAML generation;
  • trigger pattern: what user prompt or repo state started the task;
  • operator sequence: searches, file reads, edits, validation commands, approvals;
  • failure recovery: what fixed the run after a bad assumption;
  • validation artifact: tests, build output, search hit, scorecard, human review;
  • counterexamples: traces where the same approach failed.

Promote a skill only when several traces show the same trigger, steps, failure modes, and validation pattern.

Needle Corporate KB Questions

Corporate KB mining is not the same as generic RAG. The logs show what people could not find.

Mine for:

  • repeated user questions with high search/tool churn;
  • sessions where the agent eventually found the answer after many retrieval attempts;
  • queries that required knowing a synonym, internal acronym, file path, owner, or old project name;
  • failures where the final answer was wrong because the right source never surfaced;
  • cases where a memory or skill would have shortened the path.

The output should be a KB defect queue: missing alias, missing canonical page, stale page, poor title, missing owner, bad access permissions, no citation anchor, or no answer exists.

Reliability Metrics

Token counts and usage dashboards are not enough. Useful reliability metrics are task-level:

  • pass rate by task family;
  • retries per completed task;
  • tool-call failure rate;
  • approval reject rate;
  • blocked-on-user time;
  • test pass/fail before and after edits;
  • rework rate after human review;
  • hallucinated source rate;
  • invalid structured-output rate;
  • cost per accepted artifact;
  • time to validated completion.

The trace is the evidence; the eval decides whether the evidence was good.

Turning Traces Into Training and Optimization Data

There are four different output formats teams often confuse:

  1. Span-level training pair: one LLM call becomes one prompt/completion or chat-message example.
  2. Trajectory-level example: a full multi-step agent episode becomes one action/observation sequence with terminal outcome.
  3. Prompt-optimization dataset: a prompt template plus evaluation samples and a metric, such as Bedrock Advanced Prompt Optimization JSONL or DSPy/GEPA examples.
  4. Fine-tuning dataset: many curated records in a model-specific SFT, chat, preference, or tool-call format.

For domain-specific YAML generation, do not start with fine-tuning. Start with the cheapest reproducible loop:

  1. Define the YAML schema and validator.
  2. Build 10 to 20 adversarial examples: valid, invalid, incomplete, semantically wrong, and dangerous.
  3. Run a baseline prompt and capture structured failures.
  4. Use 30 to 100 curated examples for prompt optimization or GEPA-style instruction refinement.
  5. Move to 100 to 300 examples for robust edge-case coverage.
  6. Consider fine-tuning only if schema-constrained prompting plus optimization cannot meet reliability, cost, or latency targets.

Bedrock prompt optimization is attractive for this task class because it accepts up to 100 evaluation samples per template and can steer with default, custom LLM-as-judge, or Lambda evaluation. DSPy/GEPA is attractive when the workflow can expose rich textual feedback from validators: YAML parse errors, missing required fields, business-rule violations, and downstream execution failures.

The key is not the example count alone. The key is coverage. Fifty examples that cover every field, variant, and failure mode beat five hundred near-duplicates.

Reference Architecture

Use a collector and a local corpus, not direct export to five SaaS tools:

Claude Code / Claude Cowork / Codex / Scout / Strands / AG-UI / A2A
  -> local or managed OTel collector
  -> redaction and enrichment
  -> trace store (Phoenix/OpenInference/Langfuse/etc.)
  -> local raw transcript vault
  -> eval and annotation queue
  -> skill candidate generator
  -> KB defect queue
  -> prompt optimization dataset
  -> fine-tuning dataset only when justified

Required join keys:

  • agent.session_id
  • agent.task_id
  • trace_id
  • span_id
  • prompt.id
  • tool_call_id
  • protocol.run_id
  • protocol.task_id
  • repo.name
  • git.commit
  • model.provider
  • model.id
  • cloud.request_id
  • aws.account_id
  • aws.region
  • user.account_id or a privacy-preserving hash

Default policy:

  • log structure by default;
  • keep raw content local by default;
  • enable prompt/tool body export only for controlled experiments;
  • redact at the collector before shared storage;
  • store training candidates separately from raw traces;
  • require human or deterministic validation before promoting examples into skills, evals, or fine-tuning data.

The Work Still Needed

This article is the synthesis layer. The deeper research should become four linked artifacts:

  1. Agent trace format crosswalk: exact Claude, Codex, Cowork, Scout, Bedrock, AG-UI, A2A, ATIF, OpenInference, and local JSONL/SQLite shapes. First pass: Agent Trace Format Crosswalk 2026.
  2. Desktop agent observability: Microsoft Scout, Codex Desktop, Claude Cowork, Claude Code Desktop, local storage, admin controls, and OTel/export behavior. First pass: Desktop Agent Observability 2026.
  3. Trace-to-training conversion: Phoenix/OpenInference/ATIF, LangSmith, Langfuse, Weave, AgentOps, ADP, and how multi-step trajectories become curated examples. First local experiment: Redacted ATIF and OpenInference Conversion 2026.
  4. Domain YAML optimization: Bedrock prompt optimization, DSPy/GEPA, lm-evaluation-harness YAML tasks, schema validators, example-count experiments, and when SFT beats prompt optimization. First pass: Domain YAML Generation Optimization 2026.
  5. Format standardization and LangSmith: whether LangSmith applies, where it fits against Phoenix/OpenInference, how to standardize to one warehouse plus multiple semantic views, query patterns, safe raw snippets, and definitions for traces/spans/runs/turns/tool calls. Follow-up: Agent Observability Standardization Playbook and Agent Observability Format Standard 2026.

The practical next experiment is small: take 10 to 20 redacted local Claude/Codex sessions, convert them into an OpenInference/ATIF-style trace record, label success/failure, generate YAML task examples, and compare baseline prompt versus optimized prompt against a deterministic YAML validator.

Sources and Linked Research

Brandon Sneider | brandon@brandonsneider.com June 2026