HuggingFace Papers
8/10 signal
ABot-World-0: Infinite Interactive World Rollout on a Single Desktop GPU
modelsinfrastructureresearch
Summary
The paper introduces ABot-World-0, an action-conditioned video world model that enables real-time, long-horizon interactive world simulation on a single desktop GPU. The system uses a multi-source data infrastructure with agent-driven collection, a progressive bidirectional-to-causal training pipeline featuring a novel 'LongForcing' technique to mitigate autoregressive drift, and a co-designed streaming inference stack. On a single NVIDIA RTX 5090, ABot-World-0 streams 720P video at up to 16 FPS with 1.2s action-to-first-frame latency, using approximately 19 GiB of VRAM.
Problem
Creating interactive world models faces four coupled bottlenecks: acquiring broad, temporally coherent data with reliable action supervision; representing user intent for both camera and character control; preventing generative models from drifting during long autoregressive rollouts; and deploying the full stack at interactive speeds on consumer hardware. Scaling video models alone does not resolve these issues, making most systems unusable as local, persistent world simulators.
Method
The work combines a data infrastructure and a model training pipeline. The data infrastructure, 'WorldExplorer', uses agent-driven collection from AAA games, simulation engines, and internet video, guided by training feedback. The model, ABot-World-0, is trained using a progressive pipeline that first trains a bidirectional action-conditioned teacher, then distills it into a causal student via teacher forcing and ODE distillation. A final 'LongForcing' stage aligns long student self-rollouts with an extended-horizon teacher to improve stability. The system is deployed with an optimized streaming stack for real-time performance.
Details
Performance and Deployment:
- The system achieves real-time interactive streaming on a single NVIDIA RTX 5090 desktop GPU.
- It streams 720P video at up to 16 FPS.
- Action-to-first-frame latency is 1.2 seconds.
- Peak VRAM usage is approximately 19 GiB.
- The deployment stack includes a lightweight VAE decoder, efficient attention, memory-aware scheduling, and low-bit Diffusion Transformer (DiT) inference.
Model and Training Pipeline:
- The model uses raw keyboard inputs as a unified control interface for both scene roaming and third-person character control. An 8-dimensional multi-hot vector represents W/A/S/D and I/J/K/L keys.
- A reference-character memory module provides persistent appearance cues from canonical images to maintain identity consistency in third-person rollouts.
- The training pipeline starts with a bidirectional teacher model, fine-tuned from the Wan2.2 backbone, on an action-video corpus.
- This teacher is distilled into a causal student in three stages: 1) Teacher forcing to adapt the model to causal generation, 2) Causal ODE distillation to enable few-step inference, and 3) LongForcing, which matches the distribution of long student self-rollouts against an extended-horizon teacher to mitigate accumulated drift.
Data Infrastructure:
- The training corpus is sourced from AAA games, simulation engines (Unreal Engine and a 3D Gaussian Splatting backend called ABot-3DGS), and real-world internet videos.
- Data collection is managed by 'WorldExplorer', an agent-driven system that uses training feedback to guide data collection in games and simulations.
- A multi-stage quality filtering pipeline applies 14 deterministic checks across six dimensions (file integrity, visual validity, geometric consistency, game state, action-label alignment, metadata) and VLM-based semantic assessment.
- Data is annotated with action labels (ground-truth for games/simulations, pseudo-labels from pose estimation for internet video), VLM-generated scene descriptions, and structured semantic tags.
Evaluation:
- The model's controllability, visual quality, physical plausibility, and temporal memory were evaluated on the WorldRoamBench benchmark and through extended interactive rollouts.
What's new
The work presents a full-stack system for real-time interactive world modeling on a single desktop GPU. Key contributions include: 1) ABot-World-0, a unified model for scene roaming and character control from raw keyboard inputs. 2) The WorldExplorer data infrastructure with a closed-loop, training-feedback-driven collection process. 3) A progressive bidirectional-to-causal training pipeline featuring LongForcing, a technique to mitigate autoregressive drift by supervising long student rollouts. 4) A demonstration of a co-designed streaming stack achieving 16 FPS at 720P on consumer hardware.
Conclusion
The paper frames interactive world modeling as a full-stack systems challenge rather than a single generative objective. The components—a feedback-driven data infrastructure, a progressive training pipeline with LongForcing, and an optimized streaming stack—work together to create a usable system. ABot-World-0 is presented as a step toward local visual simulators that can be continuously explored, controlled, and improved for interactive creation, agent learning, and embodied AI research.
Don't read this site daily. Get it in your inbox.
The daily brief and Sunday deep dive — distilled, scored, and opinionated. For builders only.