HuggingFace Papers
7/10 signal
RxBrain: Embodied Cognition Foundation Model with Joint Language-Visual Reasoning and Imagination
researchmodelsagentic
Summary
This paper introduces Hy-Embodied-RxBrain, an embodied cognition foundation model that integrates language-based reasoning and visual imagination into a single planning sequence. Using a unified multimodal Mixture-of-Transformers architecture, RxBrain represents abstract plan structures with language while grounding them with predicted visual world states. To train and evaluate this, the authors developed an automatic pipeline for generating supervision data from videos and a new benchmark, RxBrain-Bench. Experiments show the model produces coupled text-visual plans and demonstrates promising performance in real-robot control without large-scale action pretraining.
Problem
Embodied agents need to connect high-level task reasoning with the physical world states required to achieve them. Existing vision-language models focus on scene understanding and textual decision-making, while generative world models primarily predict future visual states. This creates a gap, as neither approach unifies abstract planning and visual state grounding into a single, cohesive representation for embodied tasks.
Method
The authors propose RxBrain, a model with a unified multimodal Mixture-of-Transformers architecture that supports language, image, and video understanding and generation. It represents plans as a single sequence where language provides the abstract structure (task decomposition, constraints, logic) and visual imagination grounds the plan by predicting intermediate and final physical states. Training uses a custom pipeline that converts embodied videos into joint text-visual planning supervision data. The model's capabilities are evaluated on a new benchmark, RxBrain-Bench.
Details
- RxBrain is designed to produce plans that couple textual reasoning, world state prediction, and joint subgoal planning within a single sequence.
- The model was trained using an automatic pipeline that converts embodied videos into supervision data by decomposing them into planning steps and aligning them with visual state transitions.
- A new benchmark, RxBrain-Bench, was introduced to specifically evaluate a model's ability to represent embodied plans through joint textual and visual components, rather than through separate understanding or generation tasks.
- Experimental results indicate that RxBrain successfully maintains both embodied understanding and generation abilities.
- The model was extended to generate continuous robot actions, where it showed promising performance on a real robot without requiring large-scale, action-specific pretraining data.
What's new
The primary contribution is a model, RxBrain, that represents embodied plans in a single, unified sequence combining language for abstract reasoning and visual imagination for grounding in physical states. The work also introduces an automatic data pipeline for converting videos to planning supervision and a new evaluation benchmark, RxBrain-Bench.
Conclusion
The results from RxBrain represent an initial step toward creating foundation models for embodied cognition. By jointly modeling language-based reasoning and visual world state prediction in a unified planning sequence, the model demonstrates a promising approach for developing more capable embodied agents, as evidenced by its performance in planning tasks and preliminary real-robot control.
Don't read this site daily. Get it in your inbox.
The daily brief and Sunday deep dive — distilled, scored, and opinionated. For builders only.