HuggingFace Papers Jul 22, 2026 7/10 signal

Masked Visual Actions for Unified World Modeling

researchmodelsagentic

Summary

This paper introduces Masked Visual Actions, a pixel-space control interface that represents robot actions as a partially revealed trajectory within a video. A single pre-trained video model, finetuned on only 15 hours of masked examples, can function as both a forward dynamics model (predicting scene response to robot motion) and an inverse model (synthesizing robot motion for a desired object outcome). The method demonstrates strong visual fidelity and generalization to unseen robot embodiments, outperforming baselines with a LPIPS score of 0.0945 on the DROID dataset. The unified model is applied to policy evaluation, model-based planning, and inverse modeling for robot manipulation.

Problem

Pre-trained video models have rich priors about world physics, but conditioning them on actions for robotics is challenging. Existing action representations like text, motor commands, or keypoints are often sparse, embodiment-specific, or misaligned with the model's visual pre-training. This prevents a single model from easily generalizing across different robots or reasoning in both forward (action to outcome) and inverse (outcome to action) directions.

Method

The authors finetune a pre-trained video model (Wan-Fun-Control 2.2 14B) using a technique called Masked Visual Actions. The model is conditioned on an initial reference frame and a video where only the spatiotemporal trajectory of a specific entity (e.g., a robot arm) is visible, with the rest of the scene masked out. The model is trained to predict the complete, unmasked video. Training data consists of 15 hours of real (DROID) and simulated (Robocasa) videos, with masks generated via segmentation and URDF rendering.

Details

Quantitative Comparison on Controllable Video Generation:

The model was evaluated against baselines on the DROID dataset (seen embodiment) and the BEHAVIOR dataset (unseen bimanual embodiment). Masked Visual Actions outperforms prior work, especially in generalizing to unseen robots.

Method	DROID LPIPS ↓	DROID SSIM ↑	DROID PSNR ↑	BEHAVIOR LPIPS ↓	BEHAVIOR SSIM ↑	BEHAVIOR PSNR ↑
Image-to-video [63]	0.521	0.548	12.42	0.602	0.457	10.22
Wan-move [13]	0.534	0.562	12.99	0.312	0.756	13.17
Ctrl-World [25]	0.362	0.708	18.15	0.196	0.837	18.39
Masked Visual Actions (Ours)	0.0945	0.887	23.74	0.123	0.843	22.90

Ablation on Visual Conditioning Signal:

The dense masking approach was compared to sparser visual conditioning signals (end-effector or skeleton visualization) on DROID, a real-world dataset with an unseen gripper, and the BEHAVIOR dataset.

Method	DROID LPIPS ↓	Real world LPIPS ↓	BEHAVIOR LPIPS ↓
End-effector vis.	0.107	0.183	0.171
Skeleton vis.	0.106	0.169	0.162
Masked Visual Actions	0.0945	0.148	0.123

Qualitatively, models trained on sparse signals (end-effector, skeleton) failed on unseen embodiments, either hallucinating the training robot or distorting the new one.

Robotics Applications:

Planning: Using a Best-of-N planning approach with a Diffusion Policy, the model simulated action candidates which were then ranked by a VLM (Gemini 3.1 Pro). This consistently improved task success rates, which increased with the number of samples (N).
Policy Evaluation: In Robocasa simulation, the success rate of policies evaluated in the video model showed a strong correlation (r=0.982) with ground-truth environment success rates. In real-world evaluations across 4 tasks and 20 demonstrations each, the distribution of task progress in simulation closely matched real execution, though the model showed a consistent positive bias.
Action Extraction (Inverse Modeling): On the CoffeeServeMug task in RoboCasa, the model was prompted with desired object motion to generate a robot video. An inverse dynamics model then extracted actions. This zero-shot application achieved a 90% success rate, outperforming imitation learning baselines like Diffusion Policy, ACT, and SmolVLA.

What's new

The paper introduces Masked Visual Actions, a novel method for representing action as a dense, pixel-aligned, spatiotemporal mask. This approach unifies forward and inverse world modeling within a single video model checkpoint without retraining. By changing which entity's trajectory is revealed (robot vs. object), the same model can either predict outcomes or synthesize the behavior required to achieve them, a capability not present in prior action-conditioning methods.

Limitations

The authors state that the model learns correlations rather than causal relationships, which is a common limitation of generative models. Additionally, the method's performance, including inference speed and expressive power, is inherently constrained by the capabilities of the underlying base video model.

Conclusion

By finetuning a pre-trained video model with a small dataset of masked visual actions, it is possible to efficiently leverage the model's existing priors to synthesize counterfactuals. This single, unified model can function as an action-conditioned forward model to simulate robot actions and as an inverse model to synthesize robot motions that achieve a desired object manipulation. The framework shows strong performance in policy evaluation, planning, and action extraction.

Read the source →

Don't read this site daily. Get it in your inbox.

The daily brief and Sunday deep dive — distilled, scored, and opinionated. For builders only.