← Agent Frameworks 🕐 17 min read
Agent Frameworks

Agent Telemetry Pipelines: AWS, Dagster, Hugging Face, OTel, OpenInference, and Training 2026

Brandon Sneider · June 2026

> Primary sources are Hugging Face Hub/Datasets/TRL docs, Dagster pricing/docs,

See also (wiki): wiki/agent-operations-observability.md · wiki/semantic-observability.md · wiki/mlops-ai-platform-engineering.md · wiki/model-risk-management.md

Source credibility: MEDIUM-HIGH. Primary sources are Hugging Face Hub/Datasets/TRL docs, Dagster pricing/docs, AWS CloudWatch/S3/Glue/Athena/Firehose pricing docs, OpenTelemetry/OpenInference docs already cited in the companion pages, and the local redacted Codex conversion experiment. This page is architecture research, not a claim that a full end-to-end production pipeline has been run in this repo.

Executive Summary

Agent logs should not be treated as one thing. They are four different data products:

  1. Raw evidence: native Claude/Codex JSONL, SQLite rows, Bedrock invocation logs, Strands/AgentCore traces, AG-UI/A2A events, local screenshots and file artifacts.
  2. Operational telemetry: OpenTelemetry traces, metrics, and logs with trace_id, span_id, parentage, timing, resource attributes, service version, tenant, and rollout/session IDs.
  3. AI semantics: OpenInference, OTel GenAI, and MCP semantic-convention spans for LLM, tool, retriever, reranker, embedding, chain, agent, evaluator, prompt, guardrail, memory, context, and protocol events. Pin the semantic-convention version and record OTEL_SEMCONV_STABILITY_OPT_IN whenever gen_ai_latest_experimental or other GenAI opt-ins are used.
  4. Learning datasets: curated SFT rows, preference pairs, verifier environments, eval/golden-set cases, skill candidates, KB defect records, and reliability metric aggregates.

The recommended pipeline is:

native agent logs + cloud invocation logs + protocol events
  -> raw vault: S3 or HF private bucket/dataset, partitioned by date/source/tenant
  -> OTel collector / ADOT / app instrumentation
  -> OpenInference / OTel GenAI / MCP semantic spans
  -> bronze/silver/gold tables
  -> review queues and redaction gates
  -> Hugging Face / internal lake datasets in Parquet + JSONL
  -> TRL / Unsloth / LLaMA-Factory / Axolotl / PRIME-RL / AgentCore RL Toolkit
  -> eval, benchmark, and rollout promotion

Use AWS when regulated cloud audit, IAM/KMS, Bedrock joins, private VPC data paths, and long-term enterprise retention dominate. Use Dagster when the hard problem is repeatable, inspectable curation across many sources and review stages. Use Hugging Face when the hard problem is ML-native dataset distribution, trace browsing, mounted storage for Jobs/Spaces, TRL-compatible training data, and collaboration around models/datasets.

What Hugging Face Adds

Hugging Face now matters for agent traces directly, not only for model hosting. The Hub’s agent-trace docs say raw JSONL sessions from Claude Code, Codex, and Pi Agent are natively supported. The documented local directories are:

Agent Local trace directory
Claude Code ~/.claude/projects
Codex ~/.codex/sessions
Pi Agent ~/.pi/agent/sessions

The Hub can show these in a trace viewer after upload to a Dataset or Storage Bucket. This is important because it creates a low-friction route from local agent sessions to an inspectable dataset. It is also risky: the same docs warn that trace files can include prompts, tool inputs, command output, local paths, screenshots, secrets, private code, and personal data. The enterprise pattern is therefore private upload first, redaction second, public release never unless a review gate passes.

Hugging Face Storage Buckets are a new fit for mutable agent artifacts: checkpoints, trace batches, redaction outputs, and intermediate curation results. Buckets are not versioned like model/dataset repos; they are mutable object storage on the Hub. Access patterns now include:

HF access path Best use Agent telemetry use
hf upload to Dataset Versioned/shareable dataset snapshots Curated redacted traces, eval sets, SFT/preference splits
hf buckets sync Continuous batch transfer Sync ~/.codex/sessions or redacted lake extracts into a private bucket
hf-mount Mount bucket/repo as a local filesystem Let training scripts, DuckDB, vLLM, pandas, or shell tools read trace/data files
hf:// with fsspec Python analytics Read/write Parquet from pandas, DuckDB, Dask, Spark-style flows
HF Jobs/Spaces volume mount Managed compute reads bucket data Run curation or training jobs without copying large datasets into the image

Two technical implications matter:

  • Hugging Face Datasets still has local cache behavior. The Hub cache defaults to ~/.cache/huggingface/hub; Datasets converts data into Arrow cache for non-streaming loads. On large trace corpora, use streaming, Parquet, or mounted buckets deliberately instead of accidentally duplicating hundreds of GB into local Arrow caches.
  • HF storage is ML-native and cheaper to browse/share in many workflows, but it is not AWS S3. Current docs say bucket S3 API access is not yet supported, so AWS-native systems still need S3, Firehose, Glue, and Athena unless you build a copy bridge.

Hugging Face Trace and Training Formats

Hugging Face gives three related but distinct format layers.

Layer Recommended format Why
Raw trace storage Native JSONL plus content-addressed artifacts Preserves Claude/Codex/Pi replay shape and lets HF trace viewer work
Analytics storage Parquet with stable schema and partitions Works with DuckDB, pandas, Datasets, Spark-like readers, and cheap column scans
Training storage TRL-supported conversational, prompt-completion, preference, prompt-only, or stepwise-supervision datasets Matches SFT/DPO/GRPO/Reward/PPO/PRM trainers

ShareGPT Is a Training Shape, Not a Trace Standard

ShareGPT-style data is useful but easy to overuse. Its common shape is a conversation array with role-like fields such as from: "human" and from: "gpt" plus text values. That makes it a convenient bridge for multi-turn SFT, especially when older fine-tuning stacks expect ShareGPT-like JSON. It does not preserve enough structure for observability or reliable agent mining:

Format Primary unit Good for Weak for
ShareGPT-style conversation Ordered chat messages Multi-turn SFT, quick chat corpus conversion, compatibility with many fine-tune recipes Trace timing, span parentage, tool-call IDs, retries, retrieval lineage, cost, latency, policy decisions, artifacts
OTel GenAI semantics Timed spans/events/metrics with GenAI attributes Production observability, cost/latency/token metrics, distributed tracing, CloudWatch/OTLP backends Direct model training rows unless converted and curated
AWS AgentCore / CloudWatch GenAI spans OTel/GenAI spans rendered and queried in AWS AWS-native agent operations, IAM/CloudWatch integration, AgentCore/Bedrock joins, searchable spans Cross-vendor training data and local raw replay unless exported/joined
OpenInference OTel-compatible AI semantic spans LLM/tool/retriever/agent/evaluator taxonomy across Phoenix/Arize and compatible pipelines Legal training eligibility, dataset splits, raw transcript completeness
MCP semantic conventions Protocol/tool spans and trace context for MCP calls Tool/resource/prompt server boundaries, cross-process trace propagation Local approval proof, host execution proof, training labels

This changes the pipeline recommendation: ShareGPT should be a gold-layer export, not the canonical warehouse schema. Convert traces into ShareGPT only after redaction, validation, label review, and split assignment. Keep OpenInference/OTel rows next to the ShareGPT export so every training example can be traced back to tool calls, retrievals, validators, costs, and failure modes.

TRL’s dataset-format docs separate format from type. Format is standard versus conversational; type is language modeling, prompt-completion, prompt-only, preference, unpaired preference, tokenized language modeling, or stepwise supervision. That maps cleanly to agent traces:

Agent mining target Trace-to-training conversion
Domain YAML generation SFT prompt-completion rows: task/context/schema constraints -> validated YAML
Better tool-call arguments SFT or preference rows over tool-call JSON with schema validation
Policy or style correction Preference pairs: rejected answer vs accepted answer, with evaluator/human reason
Multi-step reliability Prompt-only GRPO/RLVR environments with programmatic reward functions
Process supervision Stepwise supervision rows from trajectories where each step has outcome labels
KB question discovery Eval/golden-set rows, not immediately training rows

For chat models, TRL chat templates are not a cosmetic concern. The TRL docs call out model-family-specific training templates and assistant-only loss masks. For tool-using traces, the template must preserve tool-call structure and mask only the target assistant/tool-call tokens when training. A bad conversion that flattens tool calls into arbitrary text may improve benchmark-looking loss while damaging tool reliability.

AWS Pipeline

AWS is the default if Bedrock/AgentCore is part of the production path.

Bedrock invocation logs
  -> CloudWatch Logs for hot operations
  -> S3 for durable raw audit
  -> Glue catalog / crawler or explicit schema
  -> Athena / Iceberg / warehouse tables

Agent app / Strands / Claude SDK / Codex / AG-UI / A2A
  -> OTel collector or ADOT
  -> CloudWatch GenAI observability, X-Ray-style trace views, or vendor backend
  -> S3 Parquet trace tables
  -> curated datasets

Prometheus should sit on the metrics branch only:

Agent app / Claude Code / SDK / Strands
  -> OTLP metrics
  -> OTel collector or Prometheus OTLP metrics receiver
  -> Prometheus / managed Prometheus

Bedrock CloudWatch Logs or S3 ModelInvocationLog JSON
  -> parser / Lambda / Firehose transform / batch ETL
  -> derived low-cardinality metrics
  -> Prometheus

Raw prompts, outputs, request IDs, tool bodies, and traces
  -> CloudWatch Logs / S3 / warehouse / trace backend
AWS component Use Hard tradeoff
CloudWatch Logs Hot troubleshooting, account/region audit, log queries Ingestion and retained-log costs rise quickly with full prompts/tool bodies
S3 Long-retention raw vault and Parquet lake You own partitioning, redaction, lifecycle, cataloging, and query ergonomics
Firehose Near-real-time fanout and optional format conversion Charged by ingested GB and optional features; small records can suffer billing granularity
Glue Catalog, crawlers, ETL, data quality DPU/crawler/Data Catalog costs; schema drift has to be managed
Athena Serverless SQL over S3 Charged by scanned data; Parquet, compression, and partitions are mandatory for cost control
CloudWatch Application Signals / Transaction Search Span search and golden metrics Span ingestion/indexing economics can dominate at high trace volume
Bedrock AgentCore Runtime/evals/memory/gateway/observability Native control plane is useful but not a complete raw-data and training-rights ledger
Prometheus / managed Prometheus Low-cardinality counters, gauges, histograms, alerts Metrics only; never store raw prompts, requestId, trace IDs, tool bodies, or full Bedrock invocation JSON as labels

Recommended derived Prometheus metrics:

agent_model_invocations_total{source,provider,model,operation,status}
agent_input_tokens_total{source,provider,model}
agent_output_tokens_total{source,provider,model}
agent_tool_calls_total{tool_name,tool_type,status}
agent_tool_latency_seconds_bucket{tool_name,tool_type,status}
agent_eval_pass_total{task_type,scorer}
bedrock_invocations_total{region,model_id,operation}
bedrock_input_tokens_total{region,model_id,operation}
bedrock_output_tokens_total{region,model_id,operation}
bedrock_invocation_log_delivery_failures_total{destination}

Do not use Bedrock requestId, prompt hashes, user email, file path, trace ID, or tool-call ID as Prometheus labels. Keep those as exemplars or lookup keys in trace/log/warehouse systems. Prometheus is for alerting and rates; the raw ModelInvocationLog record stays in CloudWatch/S3.

AWS cost modeling is necessary but incomplete. In 2026, the biggest savings may come before logging or storage: context engineering. Track and optimize:

context_cost =
  model_input_tokens
  + tool_manifest_tokens
  + retrieval_tokens
  + memory_tokens
  + repeated_tool_output_tokens
  + compaction_summary_tokens

context_savings =
  tokens_before_compaction
  - tokens_after_compaction
  + tokens_filtered_by_code_execution
  + tokens_avoided_by_artifact_refs

Emit context_events for compaction, tool-output clearing, artifact offload, memory read/write, retrieval truncation, and code-execution filtering. A cheap CloudWatch/S3 design can still be expensive if every turn drags stale memory, large tool manifests, and full tool outputs back into the prompt.

Cost posture:

  • Keep hot CloudWatch logs metadata-heavy and short-retention unless compliance requires full content there.
  • Put raw full-content logs in S3 with lifecycle tiers and KMS, then convert to partitioned Parquet.
  • Store prompt/tool bodies as encrypted object refs and hashes in shared trace tables; reveal content only in gated review queues.
  • Query Athena against Parquet partitions, not raw JSON across the whole lake.
  • Add cost labels at ingestion: tenant_id, agent_id, model, provider, workflow, policy_version, prompt_version, rollout_id.

Dagster Pipeline

Dagster is not an observability backend for agent spans. It is useful as the orchestration and lineage system for curation:

raw_trace_batches
  -> redact_native_records
  -> normalize_to_otel_spans
  -> map_to_openinference
  -> build_turns_tool_calls_artifacts
  -> attach_bedrock_costs_and_invocation_ids
  -> run_validators_and_evals
  -> route_review_queues
  -> emit_curated_hf_dataset / s3_parquet_dataset / langsmith_dataset

The asset graph should encode every irreversible decision: redaction policy, schema version, evaluator version, labeler identity, accepted/rejected examples, and training split. That is where Dagster is strong: materialization history, asset metadata, backfills, retries, schedules, and visible lineage.

Dagster+ cost has a specific implication for this use case. The pricing docs say credits measure asset materializations and ops executed, and Solo/Starter pricing charges per credit plus serverless compute if using Dagster+ Serverless. That means a trace curation graph with tiny per-file assets can become expensive and operationally noisy. Prefer coarser partitions:

Bad Dagster asset grain Better asset grain
One asset per trace file One asset per source/date/tenant partition
One op per redaction rule One redaction op that emits rule-level metrics
One materialization per row Batch materialization with row-level counts and sample IDs
Recompute all history for schema changes Backfill only impacted partitions

Self-hosted Dagster avoids Dagster+ credit charges but moves the cost into infrastructure and operations. Dagster+ Serverless removes more operations but charges serverless compute by minute. Dagster+ Hybrid avoids Dagster serverless compute charges for the user code path, but the compute still runs in your environment and appears on your AWS/Kubernetes bill.

AWS vs Dagster vs Hugging Face

Choice Best when Watchouts
AWS-native lake Regulated enterprise, Bedrock/AgentCore, IAM/KMS, private network, CloudWatch/S3/Glue/Athena already standard More plumbing, more FinOps surfaces, weaker ML-native collaboration unless datasets are exported
Dagster on AWS Need repeatable curation, backfills, approval gates, schema lineage, eval and dataset materialization Not a trace UI; asset/ops granularity affects cost; still needs S3/HF/LangSmith/Phoenix destinations
Hugging Face Hub/Buckets Need trace viewing, ML dataset hosting, mounted storage, TRL training, model/dataset collaboration Private data review burden; bucket S3 API gap; cache duplication if used carelessly
LangSmith / Phoenix / Arize / Langfuse / Weave Need app trace debugging, annotation, eval workbench, run-level feedback Should not be the only raw vault or legal training-rights ledger
Local-only vault Sensitive R&D, small team, offline mining Harder fleet reliability metrics, sharing, review workflows, and enterprise audit

The practical combined architecture is:

S3 = durable enterprise raw vault
Dagster = curation/control-plane lineage
OTel collector = live telemetry router
OpenInference or OTel GenAI = AI semantic-convention choice
Phoenix/LangSmith/Arize/Langfuse/Weave = analysis and eval workbenches
Hugging Face = ML-native datasets, mounted buckets, training runs, trace viewer

Semantic-convention choice is a real architecture decision, not a cosmetic attribute rename. OpenInference is strong when Phoenix/Arize-style rendering and LLM/RAG/tool taxonomy are the priority. OTel-community GenAI semantic conventions are the natural fit for AWS/ADOT/CloudWatch/X-Ray and standard OTel backends. If both workbenches matter, dual-emit or translate and record the conversion version.

Canonical Conversion Pipeline

Use a bronze/silver/gold model.

Stage Tables/files Required fields
Bronze raw raw_events, native JSONL/SQLite/CloudWatch/S3 files source, source_version, raw_ref, sha256, ingested_at, tenant, privacy classification
Silver trace agent_spans, turns, tool_calls, messages, artifacts trace_id, span_id, parent_span_id, session_id, turn_id, tool_call_id, timestamps, status
Silver semantic ai_semantic_spans, protocol_events, retrieval_events openinference.span.kind or OTel GenAI gen_ai.*, model/tool/retriever attributes, redacted input/output refs, semantic convention version
Gold reliability eval_results, rollout_metrics, tool_reliability, kb_defects scorer version, benchmark/golden-set IDs, pass/fail, failure mode, latency/cost/token metrics
Gold learning sft_examples, preference_pairs, rl_env_cases, skill_candidates training eligibility, label provenance, validation proof, split, model/prompt policy version

Minimal stable IDs:

trace_id
span_id
parent_span_id
session_id
turn_id
message_id
tool_call_id
artifact_id
raw_ref
dataset_example_id
eval_id
rollout_id
policy_version
prompt_version
model_version
agent_version

Query and Mining Patterns

Goal Query signal Output artifact
Self-improving skills Repeated successful tool/action patterns followed by accepted final answers Skill draft with trigger, steps, permissions, negative cases
Hard KB questions User asks same unresolved question; retrieval spans show low overlap; final answer corrected later KB gap ticket, golden question, retrieval corpus update
Agent reliability Trace/task pass rate by model/tool/prompt/version; retries; human interventions; tool errors Reliability dashboard and release gate
Domain YAML generation YAML artifacts with validator pass/fail, schema errors, repair count SFT examples and eval set by schema/failure mode
Prompt optimization Same task across prompt versions or GEPA/DSPy candidates Prompt candidate leaderboard with cost/reliability
Tool-call training Tool span argument errors and later successful corrected calls Tool-call SFT rows or preference pairs
Cost reduction Node-level token/cost by workflow; simple tasks still using frontier models Distillation/fine-tuning candidate list

Do not train directly from these queries. The query output is a candidate queue. Every learning artifact needs redaction, label provenance, validation, split assignment, and exclusion checks.

Training Dataset Shapes

SFT Prompt-Completion

{
  "id": "yaml_sft_000123",
  "messages": [
    {"role": "system", "content": "Generate only valid policy YAML matching schema v4."},
    {"role": "user", "content": "Create a policy for read-only finance analysts..."},
    {"role": "assistant", "content": "apiVersion: policy/v4\nkind: AccessPolicy\n..."}
  ],
  "metadata": {
    "schema_id": "policy-yaml-v4",
    "validator": "policy_yaml_validator@1.8.2",
    "validation_status": "pass",
    "source_trace_ref": "s3://.../trace_id=...",
    "redaction_policy": "enterprise-agent-traces-v3"
  }
}

ShareGPT-Style SFT Export

Use this only after curation. Preserve provenance in metadata or a sidecar Parquet table because many ShareGPT loaders ignore unknown fields.

{
  "id": "yaml_sharegpt_000123",
  "conversations": [
    {"from": "system", "value": "Generate only valid policy YAML matching schema v4."},
    {"from": "human", "value": "Create a policy for read-only finance analysts..."},
    {"from": "gpt", "value": "apiVersion: policy/v4\nkind: AccessPolicy\n..."}
  ],
  "metadata": {
    "dataset_example_id": "yaml_sft_000123",
    "source_trace_id": "trace_hash",
    "validation_status": "pass",
    "redaction_policy": "enterprise-agent-traces-v3"
  }
}

Preference Pair

{
  "prompt": [
    {"role": "user", "content": "Generate the IAM condition block for..."}
  ],
  "chosen": [
    {"role": "assistant", "content": "Condition:\n  StringEquals:\n    aws:PrincipalTag/department: finance"}
  ],
  "rejected": [
    {"role": "assistant", "content": "Condition:\n  StringEquals:\n    department: finance"}
  ],
  "metadata": {
    "reason": "chosen passes schema and policy simulator; rejected uses invalid condition key",
    "judge": "human+validator",
    "source_tool_call_id": "tool_..."
  }
}

GRPO/RLVR Environment Case

Use GRPO/RLVR only when the reward is genuinely verifiable. Domain YAML is a good fit because schema validators, policy simulators, unit tests, and least-privilege checks can produce repeatable rewards. Open-ended agent helpfulness is not a good first target; the bottleneck is reward design and calibrated evaluation, not the choice between GRPO, DAPO, Dr. GRPO, GSPO, or another GRPO-family variant.

{
  "env_id": "policy_yaml_env_v4",
  "prompt": "Create a least-privilege YAML policy for...",
  "tools": ["schema_validate", "policy_simulate"],
  "reward": {
    "schema_valid": 0.35,
    "policy_simulator_pass": 0.45,
    "least_privilege_score": 0.20
  },
  "metadata": {
    "golden_set_id": "policy_yaml_golden_2026_06",
    "rollout_id": "rollout_...",
    "source_trace_hash": "sha256:..."
  }
}

Treat the environment as a package, not just a JSON row. A strong package owns task generation, reset state, hidden labels, tool access, validators, reward aggregation, trace export, and synthetic-data generation. That same package can serve as eval harness, reward source, and curriculum generator.

Tracing Formats vs Training Formats

Tracing formats and training formats have different jobs.

Question Tracing answer Training answer
What happened? OTel/OpenInference/AWS GenAI span tree Not the primary purpose
What should the model imitate? Candidate signal only SFT/chat/prompt-completion row
What should the model prefer? Eval or human-feedback event Preference pair: prompt, chosen, rejected
What should the policy optimize? Rollout spans plus reward/eval events Prompt-only RL/GRPO/RLVR environment case
What should never be learned? Redaction/safety/policy spans Exclusion list and dataset filter
Why did it fail? Tool/retriever/model/evaluator spans Failure-mode label, not necessarily training text

The hard edge is loss. A trace records many things that should not be predicted by the model: user secrets, retrieved documents, tool observations, evaluator judgments, policy-denial reasons, internal IDs, and sometimes bad intermediate outputs. A training row decides which tokens receive loss and which tokens are context only. That is why a trace-to-training converter must emit both:

  1. a training example in the target library/model format; and
  2. a sidecar provenance row with trace_id, span_id, tool_call_id, validation result, redaction policy, evaluator version, and split.

Training Format Families

Format family Typical fields Good for Libraries / models
Raw text / language modeling text Continued pretraining, domain language adaptation TRL SFTTrainer language modeling, LLaMA-Factory pretraining, Axolotl pretraining, Unsloth continued pretraining
Alpaca / instruction instruction, input, output, optional history Single-turn or simple instruction tuning LLaMA-Factory Alpaca, Axolotl instruction datasets, Unsloth Alpaca-style notebooks
OpenAI / ChatML messages messages: [{role, content}] Chat SFT and modern tokenizer chat templates TRL conversational, Axolotl default keys, LLaMA-Factory OpenAI-format ShareGPT subset
ShareGPT conversations: [{from, value}] Multi-turn chat SFT, legacy/local training recipes, some tool-use rows Axolotl, LLaMA-Factory, Unsloth via standardize_sharegpt
Prompt-completion prompt, completion Format compliance, domain YAML generation, completion-only loss TRL SFT prompt-completion, many hosted fine-tune APIs
Preference / DPO prompt, chosen, rejected Style/policy/tool-arg preference, DPO/ORPO/reward modeling TRL DPO/ORPO-style trainers, LLaMA-Factory preference, Axolotl RLHF
KTO / unpaired preference prompt/completion plus label, or response plus boolean tag Feedback when no paired rejected answer exists TRL unpaired preference/KTO, LLaMA-Factory KTO
Prompt-only RL prompt plus environment/reward config GRPO/RLVR, verifiable rewards, tool-use environments TRL GRPOTrainer, Prime/Verifiers, Axolotl GRPO, AgentCore RL Toolkit
Stepwise supervision prompt plus step labels/scores Process supervision, verifier training, multi-step reliability TRL PRM/stepwise supervision, custom verifier pipelines
Multimodal chat messages plus images/videos/audio or content parts VLM/tool/computer-use training TRL VLM, LLaMA-Factory multimodal, Axolotl multimodal

Model-Family and Architecture Edges

Training format is not only a library choice. It depends on the model’s native chat and tool format.

Edge Why it matters Practical rule
Chat template The tokenizer turns messages into model-specific tokens. Llama, Qwen, Gemma, Phi, DeepSeek, GLM, GPT-OSS, and multimodal models can differ materially. Always serialize with the target model tokenizer’s chat template or a library-vetted template; do not assume ShareGPT text is enough.
Loss masking Some rows should train only assistant/completion tokens, not user prompts, retrieved docs, or tool observations. Record loss_target=assistant_only, completion_only, last_assistant_only, or all_tokens in the dataset manifest.
Tool-call serialization Tool calls may be native JSON, OpenAI-style tool_calls, ShareGPT function_call/observation, XML-ish tags, or model-specific templates. Convert from canonical tool_calls table into the target model’s known tool template; validate by round-tripping through inference.
Reasoning traces Some models expose summaries, hidden reasoning, or <think>-style text; training on the wrong field can teach brittle artifacts or leak policy-sensitive content. Separate assistant_visible, reasoning_summary, and hidden_or_unavailable; train only on approved visible or deliberately curated reasoning data.
Multimodal placeholders VLMs often require the number of image/video/audio placeholders in text to match sidecar media arrays. Keep media refs in artifacts; generate model-specific content parts/placeholders at export time.
Base vs instruct model Base models expect raw text continuation; instruct/chat models expect role-formatted turns. Continued pretraining uses text; instruction tuning uses chat/instruction templates; do not mix without a deliberate curriculum.
MoE vs dense The JSON format may be the same, but sequence length, packing, routing stability, and batch economics differ. Treat architecture as a training-config choice, not a schema change; keep sequence length and packing metadata.
Hosted API vs open weights Hosted APIs may require provider-specific JSONL; open-weight stacks usually accept flexible datasets plus templates. Keep canonical gold tables provider-neutral; emit provider/library-specific exports late.

Translation Pipeline

Use a canonical intermediate representation, not direct trace-to-ShareGPT conversion.

raw trace
  -> canonical span/message/tool/artifact tables
  -> candidate selector
  -> redaction and exclusion filters
  -> label/eval attachment
  -> task-specific training IR
  -> library/model-family export
  -> tokenizer/template smoke test
  -> small overfit test
  -> eval gate

The training IR should look like this before any library-specific export:

{
  "dataset_example_id": "yaml_000123",
  "task_type": "sft_prompt_completion",
  "messages": [
    {"role": "system", "content_ref": "redacted_ref_system"},
    {"role": "user", "content_ref": "redacted_ref_user"},
    {"role": "assistant", "content_ref": "redacted_ref_answer", "loss": true}
  ],
  "tool_calls": [
    {
      "tool_call_id": "tool_abc",
      "name": "schema_validate",
      "arguments_ref": "redacted_ref_args",
      "result_ref": "redacted_ref_result",
      "loss": false
    }
  ],
  "labels": {
    "validation_status": "pass",
    "failure_mode": null,
    "reward": 1.0
  },
  "provenance": {
    "trace_id": "trace_hash",
    "span_ids": ["span_hash_1", "span_hash_2"],
    "redaction_policy": "enterprise-agent-traces-v3",
    "split": "train"
  }
}

Then export late:

Target Export transform
TRL SFT messages or prompt/completion; choose language modeling vs prompt-completion deliberately
TRL DPO prompt, chosen, rejected; prefer explicit prompt
TRL GRPO/RLVR prompt plus reward/verifier environment config
Axolotl OpenAI messages by default, or ShareGPT/Alpaca with field_messages, mappings, chat template, and masking
LLaMA-Factory dataset_info.json mapping for Alpaca, ShareGPT, OpenAI-format subset, preference, KTO, or multimodal
Unsloth ChatML/OpenAI-style messages after standardize_sharegpt if needed; apply model-specific chat template
Hosted fine-tune API Provider JSONL, usually chat or prompt-completion; tool-call support must be checked per model/API version

Hard Edges and Failure Modes

Failure Cause Prevention
Training on user prompts Flattened trace into text with no loss mask Use assistant-only or completion-only masking
Training on tool observations Treating tool result as assistant target Mark tool observations as context/no-loss unless the model is supposed to emit that format
Broken tool calling Exported generic JSON that does not match model native tool template Use model-family chat template and tool-call smoke tests
Lost provenance ShareGPT export drops trace/span/tool IDs Keep sidecar Parquet or metadata manifest keyed by dataset_example_id
Bad DPO pairs Chosen/rejected do not share the same prompt or differ for irrelevant reasons Build pairs from same turn/task and attach reason labels
RL reward leakage Prompt contains the answer or validator output Separate prompt, tool observations, reward function, and hidden labels
Multimodal mismatch Placeholder count differs from media refs Validate content parts/placeholders before training
Incompatible model family Dataset serialized for ChatML but model expects Llama/Qwen/Gemma-specific template Tokenizer/template smoke test before full run
Overfitting tiny examples Domain YAML set is too small or duplicated Deduplicate by semantic hash; keep held-out golden set; run small overfit only as sanity check

Recommendation for Our Corpus

The corpus should standardize on:

  1. Canonical trace tables for evidence: raw events, spans, turns, messages, tool calls, artifacts, evals, and rollouts.
  2. Canonical training IR for curation: task type, role messages, loss mask, labels, rewards, validation proofs, provenance.
  3. Late exports for each training stack: TRL, Axolotl, LLaMA-Factory, Unsloth, hosted APIs, or RL environment runners.

That gives us portability. A YAML-generation example can become TRL prompt-completion today, ShareGPT/OpenAI chat tomorrow, or a GRPO verifier environment later without losing why the example was selected or which trace proved it worked.

For VLM, omni, video/audio, VLA, MoE, and multimodal MoE models, use the separate architecture map: Complex Model Fine-Tuning Formats: VLM, Omni, MoE, and Architecture-Specific Limits 2026. The short rule is: keep media artifacts and model-family processor requirements in the canonical training IR, then export late. A plain ShareGPT or ChatML row is not enough for complex model families.

Recommendation

  1. Standardize on OTel for causality and explicitly choose an AI semantic convention. Use OpenInference for Phoenix/Arize-first workflows, OTel GenAI for AWS/ADOT and standard OTel backends, or a documented translation layer when both are required. Use native raw logs as evidence and curated datasets as learning artifacts.
  2. Use AWS S3 as the regulated raw vault when Bedrock/AgentCore is involved. Keep CloudWatch hot and sampled unless full-content logs are legally required there.
  3. Use Dagster for curation lineage, not for trace storage. Model each redaction, conversion, eval, review, and dataset emission as partitioned assets.
  4. Use Hugging Face for ML-native datasets and trace review after redaction. Upload raw local traces only to private repos/buckets; publish only curated redacted datasets.
  5. Export to multiple consumers. Phoenix/LangSmith/Arize/Langfuse/Weave for debugging and evals; S3/warehouse for audit and analytics; HF datasets for training and benchmark sharing.
  6. Treat training as a separate approval workflow. A trace that is useful for debugging is not automatically legal, safe, or balanced training data.

Open Gaps

  • Hugging Face’s trace viewer creates a useful ecosystem standard for local agent JSONL, but it is not an OTel/OpenInference replacement.
  • HF buckets are promising for mounted ML storage, but lack S3 API compatibility today, so AWS-native lakes still need native S3 paths.
  • Dagster cost behavior pushes teams toward coarse, partitioned assets rather than per-row trace DAGs.
  • Bedrock/CloudWatch costs need workload-specific modeling; the right answer changes if prompts/tool bodies are logged, sampled, compressed, or stored only as encrypted refs.
  • The highest-value future experiment is to implement one verified converter: Claude/Codex/Strands/Bedrock raw records -> OTel/OpenInference spans -> Parquet tables -> HF SFT/preference/eval datasets, with redaction proofs.

Sources