HuggingFace Papers
7/10 signal
BadWAM: When World-Action Models Dream Right but Act Wrong
safetyagenticresearch
Summary
This paper demonstrates that the assumed safety of World-Action Models (WAMs), which couple action generation with future world prediction, is fragile. The authors introduce BadWAM, a framework for a new class of adversarial attacks called World-Action Drift Attacks, which use small visual perturbations to break the alignment between a WAM's predictions and its actions. The most direct attack reduces task success rates from 96.5% to 43.1%. The work shows that WAMs can be manipulated to execute harmful actions even while appearing to imagine a plausible, safe future.
Problem
World-Action Models (WAMs) are believed to be robust and safe because their generated actions can be checked against their imagined future states. However, this assumption is largely untested. This work addresses the vulnerability of WAMs to adversarial attacks that specifically target the coupling between action and prediction, seeking to desynchronize what a model imagines from what it actually does.
Method
The authors introduce BadWAM, a framework for modeling and evaluating World-Action Drift Attacks using small visual perturbations. The framework defines two attack types: an 'action-only' attack that prioritizes disrupting the task by driving the model toward failing actions, and an 'imagination-preserving' attack that prioritizes stealth by inducing harmful actions while keeping the model's predicted future close to its original, unattacked prediction. These attacks were evaluated on different WAM variants under closed-loop execution.
Details
Attack Performance:
- The action-only adversarial attack significantly reduces task success rates in closed-loop execution, dropping model performance from 96.5% to 43.1%.
Stealth Attack Vulnerability:
- The imagination-preserving attack demonstrates a WAM-specific vulnerability. It can maintain strong attack performance (inducing harmful actions) while simultaneously reducing the drift in the model's future imagination.
- This shows that a WAM can be manipulated to execute a harmful action while its internal prediction of the future remains plausible and aligned with a safe outcome, making the attack difficult to detect by checking the model's 'imagination'.
What's new
This paper introduces 'World-Action Drift Attacks,' a new class of adversarial attacks specifically designed to exploit the coupling between prediction and action in World-Action Models. It also presents BadWAM, a unified framework for modeling and evaluating these attacks along the dimensions of strength and stealth.
Conclusion
The assumption that coupling action generation with future prediction makes World-Action Models inherently robust and safe is fragile. These models are vulnerable to adversarial attacks that can desynchronize their actions from their imagined futures. This work shows that a WAM can be made to 'dream right but act wrong,' highlighting a critical failure mode for embodied control systems based on this architecture.
Don't read this site daily. Get it in your inbox.
The daily brief and Sunday deep dive — distilled, scored, and opinionated. For builders only.