← Agent Frameworks 🕐 10 min read
Agent Frameworks

RAG From Historical Rollouts: What Research Has Actually Tested

State of AI · August 2026

Historical data is not one thing. A prior record can be used as:

Source status: synthesis of primary papers, public datasets, GitHub repositories, and official project records retrieved 2026-08-02.

Evidence Boundaries: Most RAG research still evaluates a fixed corpus and a fixed test set. A smaller and newer body of work studies historical user logs, prior agent trajectories, feedback adaptation, or skill libraries. Public enterprise evidence is limited by privacy, access, and data-sharing constraints. Reported improvements below are paper claims on named benchmarks or deployments; they are not guarantees for a corporate rollout corpus.

Source files: sources/19-agent-frameworks/rollout-mining-rag-skill-improvement-research-2026-raw.md; related local evidence includes research/19-agent-frameworks/agent-trace-format-crosswalk-2026.md and research/19-agent-frameworks/agent-observability-format-standard-2026.md.

Executive Summary

  • Academic research is separating four questions often collapsed into “RAG efficacy”: did the retriever find evidence, did the model use it, did the answer satisfy the user, and did the system improve after seeing prior interactions?
  • The most direct evidence for learning from historical interaction data comes from LRAT, UNO, ComRAG, Feedback Adaptation for RAG, and a 150,000-interaction production evaluation at Trendyol.
  • The most direct evidence for skill improvement from prior trajectories comes from Retrieval-Augmented LLM Agents, SkillAxe, Skill Coverage, SAGE, Context as a Tool, and AgentTrails.
  • Enterprise-specific benchmarks are becoming more realistic—EnterpriseRAG-Bench, Deep Search, and DQA add heterogeneous sources, conflicts, absent answers, multi-hop reasoning, or replayed support scenarios—but most remain synthetic or small and anonymized.
  • The implication for internal rollout mining is clear: build a temporal evaluation loop that tests retrieval, reuse, and skill changes against held-out workflows. Do not measure success with similarity scores or answer quality alone.

The important distinction: history has four roles

Historical data is not one thing. A prior record can be used as:

Role of history What is retrieved or learned Correct evaluation question
Evidence retrieval Documents, messages, tickets, code, or prior answers Did the system retrieve the evidence needed for this request?
Case or trajectory reuse A prior solution, tool sequence, failure, or decision path Did the prior case reduce work without importing a wrong assumption?
Feedback adaptation A correction, preference, or accepted answer How quickly did the correction affect related future requests?
Skill improvement A reusable procedure, constraint, rule, or policy Did the new skill improve held-out task outcomes and remain valid?

A vector index can support the first two roles. It does not prove the last two. Skill improvement requires a before-and-after test, a temporal split, and a record of when the skill was introduced.

What research says about RAG efficacy

Static evaluation is necessary but insufficient

ARES established a practical metric vocabulary around context relevance, answer faithfulness, and answer relevance, using learned judges calibrated with a small human-labeled set. CUB extends the concern to noisy contexts and finds that methods can look better on simple synthetic data than on naturally occurring samples.

Those metrics belong in an internal rollout system, but only as layers in the scorecard. A retrieved passage can be relevant while the answer is wrong. A correct answer can be produced from model knowledge without the retrieval working. A high judge score can coexist with user dissatisfaction.

The latter point is supported by Whom to Trust?, which studies 150,000 real e-commerce interactions. The authors find that LLM judges track broad user-satisfaction trends but miss important dissatisfaction details. For production evaluation, direct user signals and multiple diagnostic judges are more informative than a single automated score.

Enterprise RAG is a data-relationship problem

EnterpriseRAG-Bench supplies slightly more than 500,000 synthetic enterprise documents across Slack, Gmail, Linear, Drive, HubSpot, Fireflies, GitHub, Jira, and Confluence, with 500 questions spanning project reasoning, conflicts, near-duplicates, metadata constraints, completeness, and information-not-found cases. Its MIT-licensed repository also provides a generation framework for changing source mix and company scale.

Benchmarking Deep Search over Heterogeneous Enterprise Data uses 39,190 enterprise artifacts and tests multi-hop, source-aware reasoning across documents, meetings, Slack, GitHub, and URLs. The paper reports an average score of 32.96 for the strongest agentic RAG methods in its experiment and identifies incomplete evidence retrieval as the main bottleneck.

These benchmarks are useful because corporate knowledge is not a clean document collection. It includes informal discussions, duplicated decisions, stale records, conflicting owners, and answers that do not exist. A rollout corpus adds another complication: the agent’s own search and tool actions become part of the historical evidence.

What research says about learning from historical rollouts

Retrieval can be trained from agent behavior

Learning to Retrieve from Agent Trajectories argues that human-centric retrieval signals such as clicks and dwell time do not fully capture how an agent consumes evidence. It derives supervision from browsing actions, rejected candidates, and post-browse reasoning. The reported result is better evidence recall, task success, and execution efficiency on deep-research benchmarks.

A useful training row is not only query -> clicked document. It can include the workflow state that produced the query, candidates displayed but not opened, candidates later cited or used, the tool action that followed retrieval, the evaluator or human outcome, and whether the result was later corrected.

The caution is off-policy noise. The fact that an agent opened a document does not mean the document helped. The fact that it was not opened does not mean it was irrelevant. Later actions and outcomes are needed to infer utility.

Direct trajectory reuse works only when selection is right

Retrieval-Augmented LLM Agents: Learning to Learn from Experience studies the combination of supervised fine-tuning and retrieval of prior trajectories. Its contribution is a systematic treatment of storage, querying, and trajectory selection, with improved generalization to unseen tasks when retrieval and training are combined.

SWE-ContextBench makes the same question concrete for coding agents. It contains 1,476 tasks from 51 repositories and nine languages, with related task sequences derived from issue and pull-request relationships. Its result is a useful governance rule: compact, correctly selected experience can improve accuracy, runtime, and token cost; irrelevant experience can hurt.

This is why a corporate rollout index should retain negative examples. Wrong-repository matches, obsolete fixes, misleading summaries, and skills that worked only under an old tool version are not noise to delete. They are the test set for deciding when retrieval should abstain.

Logs can improve a system without becoming a giant memory prompt

UNO and its public implementation are unusually relevant to internal usage mining. UNO distills user logs into semi-structured rules and preference pairs, clusters logs by query and feedback semantics, estimates a gap between the model and observed user behavior, and routes future requests through primary or reflective experience modules.

The design says something important about scale. Historical logs should not simply be appended to every future prompt. They should be curated into smaller objects: rules, preferences, representative cases, and reflective corrections, each linked back to raw evidence. The repository is small and benchmark-oriented, so it is a method reference rather than an enterprise component.

ComRAG reaches a related conclusion in industrial community question answering: combine static knowledge with historical QA pairs, but use a memory structure designed for retrieval and growth rather than adding every interaction as another chunk.

Adaptation itself needs a benchmark

Feedback Adaptation for RAG defines two metrics that should become standard in rollout mining:

  • Correction lag: how many requests or how much time passes before a correction changes behavior.
  • Post-feedback performance: whether the correction improves related future requests rather than only repeating the original answer.

This turns “the agent learned from the rollout” into a measurable claim. An internal analogue might be: a reviewer marks a generated migration plan unsafe; within the next 30 days, similar plans should show a lower rate of the same failure without an unacceptable increase in refusals or cost.

What research says about skill improvement

Skill extraction is becoming an observable testing problem

AgentTrails converts chronological agent logs into provenance graphs, aligns repeated executions, and extracts patterns and skill abstractions. The contribution is the representation: tool calls become computational actions, inputs and outputs become data artifacts, and comparisons operate on dataflow rather than transcript similarity.

Skill Coverage translates skill instructions into behavior constraints, checks whether trajectories cover those constraints, and assigns pass/fail outcomes. The paper reports average coverage of only 38.66%-45.51% across evaluated trajectories, then recovers an average 16.0% of failed tasks by strengthening instructions tied to failed constraints.

SkillAxe uses evaluation-guided self-refinement and reports a SpreadsheetBench pass-rate change from 16.0% to 52.0% with 22 skills learned from past trajectories. The result is interesting, but narrow: a skill library can improve when failures are classified and fed back into the skill text. It is not evidence that summarizing every corporate rollout will create reliable skills.

Training on historical trajectories can change the policy

SAGE uses sequential rollouts over related tasks, a skill library, and skill-integrated rewards. On AppWorld, the paper reports 8.9% higher scenario goal completion, 26% fewer interaction steps, and 59% fewer tokens when applied to a model with expert experience. The GitHub implementation is archived, CC BY-NC 4.0, and explicitly released for academic reproducibility, so it is an experimental reference rather than a maintained dependency.

Context as a Tool trains a software-engineering model from complete trajectories with injected context-management actions. It separates stable task semantics, condensed long-term memory, and high-fidelity short-term interaction, reporting a 57.6% solved rate on SWE-Bench Verified. Historical rollouts can therefore train context-management behavior, not only supply old text for retrieval.

The strongest enterprise-specific example is ACRM, a credit-risk model-refreshing system described as deployed at a major fintech institution for three months across six scenarios. The paper combines historical workflow retrieval, safety guardrails, and preference alignment on expert-labeled trajectories, and reports a reduction of the refresh cycle to 1.1 days with 65% fewer iteration rounds. The paper does not release the enterprise corpus, so this is a deployment report, not a reproducible public benchmark.

The most relevant enterprise evidence

Item Data or setting What it demonstrates Limitation
EnterpriseRAG-Bench Slightly over 500,000 synthetic documents; 500 questions; nine source types Heterogeneous enterprise retrieval, conflicts, absence, metadata, completeness Synthetic corpus; not historical agent rollouts
Deep Search benchmark 39,190 synthetic enterprise artifacts; multi-hop answerable/unanswerable queries Retrieval of complete evidence is a main RAG bottleneck Synthetic workflow construction
DQA 150 anonymized IT-support scenarios; replay evaluation Persistent diagnostic state and case aggregation can beat ordinary multi-turn RAG Small, domain-specific evaluation
Trendyol evaluation 150,000 real e-commerce interactions User satisfaction and LLM judges diverge; hybrid evaluation is safer Product QA, not coding-agent usage
ACRM Three-month fintech deployment; six scenarios Historical workflow retrieval can sit inside a governed production loop No public code or raw trajectory data
UNO User-log optimization with MemoryBench and local models Logs can become rules, preferences, and reflective experiences Small research implementation and benchmark setting

The pattern is consistent: enterprise research is strongest when the workflow has a verifiable outcome—ticket resolution, model refresh, accepted answer, or task completion. Pure transcript similarity is a weak proxy for organizational value.

The evaluation loop to build internally

The academic results suggest three separate holdout tracks:

Track Primary measures Required controls
Evidence retrieval Evidence recall/precision, citation validity, abstention, latency, cost Freeze the corpus; split by time and source; include absent-answer and conflict cases
Experience reuse Gain over no-history baseline in success, time, tokens, and tool calls Retrieve only earlier runs; hold out task families, repositories, or teams; keep misleading runs
Skill improvement Skill trigger precision, constraint coverage, failure recovery, stale-skill rate Version skills; record introduction time; test on future tasks; require rollback criteria

For rollout data, every derived object should retain provenance: source run IDs, repository or workspace version, tool/model versions, evaluator result, redaction policy, owner, and validity window. The raw artifact should remain separate from the analytical projection.

The first internal benchmark does not need to be large. A carefully labeled set of successful, failed, misleading, corrected, and no-answer episodes is more valuable than a larger unreviewed transcript pile. Compare no-history, document-only RAG, prior-trajectory retrieval, and skill-assisted variants under the same task and cost budget.

What This Means for Your Organization

The research does not support buying or building “memory” as a single feature. It supports a controlled improvement loop: capture the rollout, label what helped or failed, retrieve earlier evidence or experience, test the next run against a held-out task, and retire stale skills. That loop can use ClickHouse for structured analytics and a hybrid search layer for evidence, while preserving raw trajectories in protected object storage.

For the State of AI corpus, a direct search layer can remain a thin personal-session interface over curated records. The analytical center should be a versioned rollout lake with explicit episode, artifact, evaluation, and skill tables. That gives the organization a way to answer both “what are people doing with GenAI?” and “did a change improve the next workflow?”

For help translating the research into an internal rollout-mining pilot, contact brandon@brandonsneider.com.

Sources

The full provenance record is in the source ledger. Key references:

State of AI | brandon@brandonsneider.com August 2026