HuggingFace Papers
7/10 signal
EgoSteer: A Full-Stack System Towards Steerable Dexterous Manipulation from Egocentric Videos
agenticmodelsresearch
Summary
This paper introduces EgoSteer, a full-stack system for training steerable, dexterous manipulation policies for robots. The system features EgoSmith, a data pipeline that processes 9.6K hours of egocentric human video for pre-training a Vision-Language-Action (VLA) model. The pre-trained model is then fine-tuned on a real robot using teleoperation and human-in-the-loop correction. The resulting EgoSteer policy can execute diverse, free-form language instructions, demonstrating generalization, failure recovery, and few-shot adaptation to complex tasks.
Problem
A defining capability of generalist robot policies is steerability, but this has been largely absent in dexterous-hand systems. The primary bottleneck is the lack of large-scale, language-aligned, and action-accurate demonstration data required for training. This work aims to address this data scarcity and enable the development of steerable, dexterous manipulation agents.
Method
The authors developed a three-part system. First, a data pipeline called EgoSmith was created to curate 9.6K hours of pre-training data from in-the-wild egocentric videos. Second, they built a unified robot stack for teleoperation and human-in-the-loop refinement using DAgger. Finally, they trained EgoSteer, a world-model-enhanced VLA, by pre-training it on the large human video dataset and then post-training it on the real-robot data to ground its manipulation priors.
Details
- The EgoSmith data pipeline curated 9.6K hours of high-quality pre-training data, achieving 9x higher throughput and better accuracy than the previous state-of-the-art.
- The EgoSteer model robustly executes free-form instructions across more than 40 diverse tasks, demonstrating capabilities like failure recovery and dexterity.
- The pre-trained model can be few-shot adapted to complex, long-horizon tasks such as box folding.
- On these long-horizon tasks, the system achieved over 75% success across two different robot embodiments.
What's new
The primary contribution is the full-stack, integrated system for creating steerable dexterous agents. This includes EgoSmith, a novel data pipeline that solves a key data bottleneck by processing large-scale egocentric human videos. The combination of a world-model-enhanced VLA, pre-training on human data, and DAgger-based human-in-the-loop refinement on a real robot represents a state-of-the-art approach to grounding manipulation policies.
Conclusion
The presented full-stack system effectively scales dexterous VLA pre-training from egocentric human videos and enables data-efficient real-robot fine-tuning. The resulting EgoSteer model demonstrates robust, generalizable, and steerable manipulation across a wide range of tasks. By open-sourcing the system, data, and model, the authors aim to facilitate further research in dexterous robot manipulation.
Don't read this site daily. Get it in your inbox.
The daily brief and Sunday deep dive — distilled, scored, and opinionated. For builders only.