HuggingFace Papers
7/10 signal
Xiaomi-Robotics-U0: Unified Embodied Synthesis with World Foundation Model
modelsresearch
Summary
This paper introduces Xiaomi-Robotics-U0, a 38-billion-parameter multimodal autoregressive model for unified embodied synthesis. The model treats embodied generation as an extension of foundation image and video generation, jointly optimizing tasks like text-to-image, embodied scene generation, and embodied video generation. This unified framework adapts a pre-trained world foundation model to robotics, preserving its generalization while enforcing physical constraints. The model achieves state-of-the-art results on several embodied generation tasks and significantly improves the success rate of a policy on real-world manipulation tasks.
Problem
Foundation image and video generation models possess strong generalization but are not directly applicable to embodied AI scenarios. This is due to their failure to enforce multi-view consistency, geometric coherence, and the physical constraints of robot embodiments. Existing methods that adapt these models with limited robot data often sacrifice the rich visual knowledge acquired during large-scale pre-training.
Method
The authors developed Xiaomi-Robotics-U0, a 38B parameter multimodal autoregressive model. Their approach is a unified framework that jointly optimizes five distinct but related tasks: text-to-image generation, image editing, embodied scene generation, embodied transfer, and embodied video generation. By treating embodied tasks as an extension of general generative tasks, the model learns to adapt to physical constraints without losing the capabilities of the base world foundation model.
Details
- The model is the first to support high-quality, multi-view scene generation across multiple different robot embodiments.
- It introduces a novel technique called structured, controllable embodied transfer for fine-grained editing that preserves multi-view consistency and interaction dynamics.
- In human evaluations for embodied scene generation and transfer, Xiaomi-Robotics-U0 outperformed GPT-Image-2.0.
- It ranked first on the World Arena benchmark for embodied video generation.
- When used as a data engine, it significantly improved the performance of a manipulation policy on challenging real-world tasks.
| Policy | Baseline Success Rate (OOD) | Success Rate with U0 Data (OOD) | Improvement |
|---|---|---|---|
| pi_0.5 | 36.9% | 63.2% | +26.3% |
What's new
This work presents a unified framework that jointly optimizes general-purpose generation (text-to-image, video) with specialized embodied tasks (scene generation, transfer). It is the first model demonstrated to produce high-quality, multi-view consistent scenes for multiple robot embodiments. The paper also introduces "structured, controllable embodied transfer," a new method for fine-grained, consistent editing of embodied scenes.
Conclusion
The results demonstrate that foundation world models can successfully serve a dual role as both embodied world models and as scalable data engines for embodied intelligence. By unifying general and embodied synthesis, it is possible to adapt large pre-trained models to robotics, preserving their powerful generalization capabilities while satisfying the unique constraints of physical interaction.
Don't read this site daily. Get it in your inbox.
The daily brief and Sunday deep dive — distilled, scored, and opinionated. For builders only.