HuggingFace Papers
8/10 signal
SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning
agenticreasoningresearch
Summary
This paper introduces SEED (Self-Evolving On-Policy Distillation), a framework for agentic reinforcement learning that addresses the supervision gap from sparse, trajectory-level rewards. SEED uses the policy model itself to analyze its own completed trajectories, generating natural-language "hindsight skills" that capture key insights. These skills are then distilled back into the policy as a dense, token-level signal, which is optimized alongside the primary RL objective. Experiments on text- and vision-based tasks show SEED consistently improves performance, sample efficiency, and generalization.
Problem
Outcome-based reinforcement learning (RL) is a common paradigm for training agentic large language models, but it suffers from a supervision gap. The rewards are sparse, provided only at the end of a trajectory, which offers limited guidance for the intermediate, token-level decisions required for long-horizon tasks. This makes it difficult to optimize the policy effectively based on episode-level outcomes alone.
Method
The proposed method, SEED, is a self-evolving framework. The policy model is first fine-tuned to analyze its own completed on-policy trajectories and generate natural-language hindsight skills, such as reusable workflows or failure-avoidance rules. During RL, SEED re-scores sampled actions with and without the skill-augmented context, converting the resulting probability shift into a dense, token-level distillation signal. This signal is jointly optimized with the standard outcome-based RL objective on a range of text-based and vision-based agentic tasks.
Details
Framework Mechanics:
- SEED converts completed on-policy trajectories into training-time "hindsight skills" expressed in natural language. These skills are designed to capture reusable workflows, decisive observations, or rules for avoiding failure.
- The policy model itself serves as the analyzer that extracts these skills. As the policy improves through RL, its ability to generate insightful skills also evolves, creating a self-improving loop.
- A dense, token-level, on-policy distillation signal is created by calculating the probability shift induced by the hindsight skills on sampled actions. This signal provides granular supervision that is absent in standard outcome-based RL.
- This auxiliary distillation signal is optimized jointly with the primary outcome-based RL objective, ensuring the supervision remains aligned with the current trajectory distribution.
Experimental Findings:
- Experiments were conducted on a variety of text-based and vision-based agentic tasks.
- Across these tasks, SEED was found to consistently improve both performance and sample efficiency compared to baselines.
- The method also exhibited robust generalization to unseen scenarios.
What's new
The primary contribution is a self-evolving framework where the agent's policy model learns to generate its own dense, token-level supervision. By having the policy analyze its own on-policy trajectories to create "hindsight skills" and then distilling them back, the supervision signal co-evolves with the policy, keeping it aligned and relevant.
Conclusion
The authors conclude that SEED effectively bridges the supervision gap between sparse, episode-level outcomes and the token-level decisions required in agentic RL. By converting on-policy experience into dense, evolving supervision through skill distillation, the framework consistently improves performance, sample efficiency, and generalization on complex, long-horizon tasks.
Don't read this site daily. Get it in your inbox.
The daily brief and Sunday deep dive — distilled, scored, and opinionated. For builders only.