See also (wiki): wiki/agentic-ai-governance.md · wiki/enterprise-slm-specialization.md · wiki/training-architecture.md · wiki/slm-fine-tuning-enterprise.md Cross-references within pillar: slm-data-curation-pipeline-2026.md · feedback-loop-commercial-tooling-2026.md
Source credibility: GitHub repos and official SDK documentation are TIER 1 (primary source, observed directly). Vendor product claims without independent validation are TIER 2. Marketing copy without documentation support is TIER 3.
Why This Note Exists
The three-layer enterprise SLM pipeline is: (1) mine traces → (2) curate/filter → (3) fine-tune. Layers 2 and 3 are documented in slm-data-curation-pipeline-2026.md and enterprise-slm-finetuning-oss-landscape-2026.md. Layer 1 was the gap: how do raw agent execution logs get converted into structured instruction-response pairs that can enter a curation pipeline?
The short answer: every tool below works well at the span level (single LLM call → training pair). None of them solve the trajectory level problem (multi-step agent episode → single training example). That gap is documented at the end of this note.
Infrastructure Layer: OpenTelemetry GenAI Conventions
By early 2026, OTel gen_ai.client spans are stable with attributes gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.response.finish_reasons. gen_ai.agent spans remain experimental. Major frameworks (OpenAI SDK, LangChain, LlamaIndex, AutoGen) ship OTel emitters. Major observability platforms (Datadog, Honeycomb, Grafana) natively map these conventions.
This is the infrastructure layer — it standardizes what gets captured but says nothing about conversion to training pairs. [TIER 1 — opentelemetry.io/docs/specs/semconv/gen-ai/]
1. LangSmith — Best Native Annotation + Export Path
Provenance: Commercial SaaS (LangChain, Inc.), Python SDK MIT-licensed at langchain-ai/langsmith-sdk.
What it captures: Every LangChain/LangGraph runnable execution emits a Run object: run ID, run type (llm, chain, tool, retriever), inputs, outputs, error, timestamps, token counts, and parent/child run hierarchy. Multi-step agent traces stored as parent run with child runs for each tool call and LLM invocation. [TIER 1 — langchain-ai/langsmith-cookbook, primary SDK docs]
Export schema: The JSONL schema for chat-model runs mirrors OpenAI’s fine-tuning format: {"messages": [{"role": "system", ...}, {"role": "user", ...}, {"role": "assistant", ...}]}. The cookbook notebook fine-tuning-examples/export-to-openai/fine-tuning-on-chat-runs.ipynb demonstrates this transform explicitly. [TIER 1 — langchain-ai/langsmith-cookbook repo]
Filtering and annotation before export:
- Feedback scores: Explicit thumbs-up/down, numeric rubric scores (1–5), and correction annotations attach to runs via the feedback API. Filter:
list_runs(filter='and(eq(feedback_key, "correctness"), gte(feedback_score, 0.7))'). - Annotation queues: Runs matching a filter condition (thumbs_down feedback, error category, etc.) route to a directed review interface. Human annotators correct the output and add the corrected pair to a named dataset in one step.
Gap: The Run schema is LangChain-centric. For a LangGraph agent, inputs are the full state dict, not a clean prompt/response pair. Extracting a clean SFT pair from a multi-turn agent trace requires custom extraction logic per agent topology. No native “select which step of the trace becomes the training target” UI — that logic lives in your export script.
Enterprise readiness: SOC 2 Type II available. SSO/RBAC on enterprise tier.
2. AgentOps — Purpose-Built Agent Observability
GitHub: github.com/AgentOps-AI/agentops | Stars: ~5,600 | License: MIT
Integrations: CrewAI, AutoGen/AG2, OpenAI Agents SDK, LangChain, Agno, CamelAI, Google ADK.
What it captures: Session-level telemetry with typed event subtypes: LLMEvent (model, prompt, completion, token counts, cost), ActionEvent (tool name, inputs, outputs), ErrorEvent (exception, stack trace), ToolEvent. All events linked to session ID with timestamps. [TIER 1 — docs.agentops.ai/v1/concepts/sessions]
Export format: REST endpoint /v2/sessions/{session_id}/export returns {session: {...}, stats: {cost, tokens, duration}, events: [{event_type, init_timestamp, end_timestamp, params, returns, ...}]}. Not JSONL training pairs — raw event export. The website claims “fine-tune specialized LLMs up to 25x cheaper on saved completions” as a Pro plan feature ($40/month tier), but the documented API shows raw event export only. [TIER 1 for raw export structure; TIER 3 for fine-tuning claim]
Gap: No native SFT export. No annotation or human-review layer. Conversion to training pairs: filter LLMEvent objects, extract params.messages as input and returns.content as output, format as JSONL — ~15 lines of Python, undocumented, left to the user. Strongest as a collection layer feeding a custom ETL script.
Enterprise readiness: Production-grade for observability. No SOC 2 observed in public docs as of May 2026.
3. W&B Weave + Data Flywheel (NVIDIA)
GitHub: github.com/wandb/weave | Stars: ~1,000 | License: Apache-2.0
What it captures: Weave traces any function decorated with @weave.op(). For LLM calls, records inputs, outputs, latency, token counts, model metadata. Auto-instrumentation patches LangChain, LlamaIndex, and OpenAI SDK. Trace tree mirrors the call stack.
The data flywheel blueprint: wandb/data-flywheel-nvidia formalizes this as an autonomous loop: production logs tagged with workload_id and client_id flow into a Celery task that deduplicates, splits train/eval, and submits to NVIDIA NeMo Customizer for LoRA SFT. Required schema per document: {timestamp, workload_id, client_id, request: <OpenAI ChatCompletion payload>, response: <model response>}. [TIER 1 — data-flywheel-nvidia repo README, directly observed]
Export format: Weave datasets export as pandas DataFrames or JSONL. The data flywheel keeps the full request/response OpenAI ChatCompletion JSON — no lossy conversion — enabling replay, few-shot construction, and fine-tuning without re-extraction.
The closest thing to a full autonomous loop available in OSS. The blueprint is the best production flywheel pattern documented, but it currently targets NVIDIA NeMo Customizer specifically; adapting to other fine-tuning backends requires custom work.
Enterprise readiness: W&B is SOC 2 Type II certified. Self-managed deployment available.
Gap: Requires the application to log the full OpenAI ChatCompletion request/response. If the application uses LangChain’s ChatOpenAI wrapper, the logged object may not match the raw API schema without unwrapping.
4. Arize Phoenix — Best Open-Source Span-to-Dataset Path
GitHub: github.com/Arize-ai/phoenix | Stars: ~9,100 | License: Apache-2.0
The highest-traction open-source option with a native trace-to-training-pair flow. Captures spans from LangChain, LlamaIndex, OpenAI SDK, DSPy, and any OTel-emitting framework.
Dataset feature: Select spans in UI → annotate → export as OpenAI fine-tuning JSONL. The closest to an end-to-end trace-to-training-pair flow in any open-source tool. [TIER 1 — arize.com/docs/phoenix/datasets-and-experiments/how-to-datasets/exporting-datasets, directly observed]
Limitation: Operates at the span level — can export a single LLM call as an SFT pair, but not a multi-step agent trajectory as a single training example without custom assembly.
5. Langfuse — Highest-Star OSS, Best for Data-Sovereign Large-Scale
GitHub: github.com/langfuse/langfuse | Stars: ~27,600 | License: MIT
What it captures: OTel-compatible from any framework. Stores in self-hostable Postgres/Clickhouse backend.
Fine-tuning export: Generations filterable by score, annotated, and exported as CSV/JSON. Blob Storage export (S3/GCS/Azure) for large-scale batch export. OpenAI fine-tuning JSONL documented as a supported export format. [TIER 1 — langfuse.com/docs, export-for-fine-tuning page]
Scale and self-host make Langfuse the dominant choice for data-sovereign enterprises processing >1M traces. Highest star count in the category.
Same limitation as Phoenix: Exports at the generation span level. Multi-turn trajectory assembly is manual.
6. Patronus AI Percival — Evaluation/Monitoring, Not a Data Pipeline
Funding: $40.1M total, $17M Series A led by Notable Capital. [TIER 2 — funding aggregator, corroborated by VentureBeat]
What Percival does: Ingests agent execution traces, classifies failures across 20+ error modes in four categories (reasoning, system execution, planning/coordination, domain-specific). Trace chat interface and trace summarization. Not a training data pipeline. [TIER 1 — VentureBeat, May 2025]
TRAIL benchmark license: Explicitly prohibits use for training systems intended to automate human evaluation. Eval-only. [TIER 1 — HF dataset card]
Gap: No documented trace-to-SFT-pair export path. Patronus is a monitoring/evaluation layer. Correct use: identify which agent steps failed, then route correct steps to a training pipeline via manual integration.
7. LlamaIndex Workflows — Routes Through Third-Party Observability
LlamaIndex Workflows emit OTel-compatible spans via the instrumentation module. No native LlamaIndex export to SFT pairs. The documented paths go through third-party layers:
- Arize Phoenix — native LlamaIndex instrumentation sends spans; Phoenix’s dataset export is the cleanest open-source path.
- Langfuse — via
LlamaIndexCallbackHandler. OpenAI fine-tuning JSONL supported. - MLflow 3 / Azure Databricks — LlamaIndex OTel → MLflow 3 GenAI tracing → pandas DataFrame export. The documented enterprise path for Azure/Databricks shops. [TIER 1 — Microsoft Learn documentation]
Workflow-specific gap: LlamaIndex Workflows emit WorkflowEvent-typed spans. The LLM span may be several levels deep. Extracting a clean prompt/completion pair requires span traversal logic that selects LLMEvent child spans and reconstructs their context. No out-of-the-box tool performs this automatically.
8. Agent Data Protocol (ADP) — The Schema Standard
Paper: arXiv:2510.24702 | GitHub: github.com/neulab/agent-data-protocol | Institution: Carnegie Mellon
Academic framework establishing the formal schema that bridges diverse agent trace formats and SFT pipelines. ADP schema: Trajectory objects containing Action subtypes (api_action, code_action, message_action) and Observation subtypes (text_observation, web_observation). The team unified 13 existing agent training datasets into ADP format and demonstrated ~20% average performance gain over base models via SFT. [TIER 1 — peer-reviewed arXiv paper]
Enterprise relevance: ADP is the emerging standard for what a trace-to-training-pair schema should look like for agent trajectories specifically — not single LLM calls. If an enterprise builds a custom Layer 1 ETL, ADP is the target schema.
Decision Table: Tool by Capture Scenario
| Capture Scenario | Primary Tool | Export Path | Key Gap |
|---|---|---|---|
| LangChain/LangGraph, LangSmith already instrumented | LangSmith | list_runs + annotation queue → OpenAI fine-tuning JSONL |
Custom extraction script required for multi-step trajectory assembly |
| LlamaIndex Workflow, need open-source, self-host | Arize Phoenix | LlamaIndex OTel → Phoenix dataset UI → OpenAI JSONL | Span-level only; trajectory assembly manual |
| Any framework, data-sovereign, large-scale (>1M traces) | Langfuse (self-hosted) | Any OTel → Langfuse → S3/GCS blob export → JSONL | Same span-level limitation |
| W&B shop, NVIDIA infrastructure, autonomous flywheel desired | W&B Weave + data-flywheel-nvidia | Weave traces → Celery → NeMo Customizer LoRA SFT | Blueprint targets NVIDIA NeMo only |
| Raw OTel traces, custom agent (no framework) | Langfuse or Phoenix (OTLP ingest) | OTLP → observability backend → filter + export | Must conform to GenAI semconv stable attributes |
| Agent failure debugging + compliance audit | Patronus Percival | Failure detection only; no SFT export | Not a data pipeline layer |
| Multi-step agent trajectory as single training example | ADP schema + custom ETL | Build against neulab/agent-data-protocol schema | No production tool does this end-to-end |
The Remaining Gap — Trajectory-Level Assembly
Every tool above operates at one of two levels: (a) single LLM span as the unit of the training pair, or (b) raw event log export requiring manual assembly.
The specific unsolved problem: A production LangGraph agent handling a procurement approval might make 12 LLM calls, 8 tool calls, and branch on 3 conditional nodes in a single execution. The correct SFT training unit is not any individual LLM call — it is the full instruction (user task) paired with the complete reasoning trajectory (the sequence of tool calls and decisions) that produced a correctly-rated outcome. No shipping tool:
- Identifies trajectory boundaries automatically
- Selects the outcome-labeled terminal state
- Assembles the multi-turn message sequence constituting the trajectory
- Scores the trajectory against a quality rubric before admitting it to the curation queue
The closest approximation: LangSmith annotation queues (run-level human rating) + custom Python ETL walking the parent/child run tree to reconstruct the trajectory + ADP schema as the output format. That ETL is 200–500 lines of application-specific code every enterprise must write independently.
The gap is not in capture fidelity — OTel coverage of LLM calls is excellent. The gap is in trajectory segmentation: knowing where a semantically meaningful agent episode begins and ends, and what constitutes a correct episode suitable for imitation learning. This is an open research problem with no off-the-shelf enterprise solution as of May 2026.
Sources
- LangSmith documentation (docs.langchain.com/langsmith). TIER 1.
- langchain-ai/langsmith-cookbook (GitHub). TIER 1.
- AgentOps-AI/agentops (github.com/AgentOps-AI/agentops) — ~5,600 stars, MIT. TIER 1.
- AgentOps Sessions documentation (docs.agentops.ai/v1/concepts/sessions). TIER 1.
- wandb/weave (github.com/wandb/weave) — ~1,000 stars, Apache-2.0. TIER 1.
- wandb/data-flywheel-nvidia (GitHub README). TIER 1.
- Arize-ai/phoenix (github.com/Arize-ai/phoenix) — ~9,100 stars, Apache-2.0. TIER 1.
- Phoenix datasets export docs (arize.com/docs/phoenix). TIER 1.
- langfuse/langfuse (github.com/langfuse/langfuse) — ~27,600 stars, MIT. TIER 1.
- Langfuse fine-tuning export (langfuse.com/docs). TIER 1.
- PatronusAI/TRAIL — HuggingFace dataset card. TIER 1.
- Patronus Percival — VentureBeat, May 2025. TIER 2.
- Agent Data Protocol — arXiv:2510.24702, CMU NeuLab. TIER 1.
- neulab/agent-data-protocol (GitHub). TIER 1.
- LlamaIndex OTel tracing — Azure Databricks / Microsoft Learn. TIER 1.
- OpenTelemetry GenAI Semantic Conventions (opentelemetry.io/docs/specs/semconv/gen-ai/). TIER 1.