General World Models: First Principles and a Development Roadmap

Overview

How do people gradually learn to ride a bicycle, drive a car, or perform even more complex tasks?

At first, their movements are often uncoordinated. With repeated practice, however, those movements become stable and precise. One important reason is that the brain develops an "internal model" through continuous interaction with the external world, allowing it to anticipate the consequences of an action [5].

In artificial intelligence, models that internally represent the state of the world and learn how it evolves are commonly known as world models. Today, world-model research spans video generation, environment simulation, and robot control. Yet the field still lacks a shared definition of what makes such a model truly general.

Starting from first principles, this article addresses three questions:

1. What makes a world model general?
2. How can it evolve from world generation to autonomous action and world orchestrator?
3. What does this roadmap require, and how far have current systems progressed?

Our central claim is that generality comes from a continuous feedback loop connecting Understanding, Prediction, and Action. Capabilities accumulate along a five-level roadmap, supported by data, architecture, and compute. Existing systems have begun to demonstrate L1 through L3, while important gaps remain at L4 and L5.

From Specialized Models to General World Models

From the perspective of foundation-model development, our goal should not be to build another specialized model for a particular task or setting. It should be to create a general foundation model that can understand the world, predict the future, and take action.

Whether a world model is general therefore depends not on its ability to perform a particular generation or control task, but on three interdependent capabilities:

1. Understanding the world: integrating different sources of information to infer the current state;
2. Predicting the future: reasoning about what may happen next and how different actions could change the outcome;
3. Taking action: influencing a digital or physical environment and using real-world feedback to refine understanding and prediction.

These capabilities cannot remain isolated. Only when Understanding, Prediction, and Action are connected through continuous environmental feedback can a model keep updating its view of the world.

A General World Model is not merely a generator, simulator, or robot policy. It is a general-purpose foundation that can understand the present, predict the future, take action, and continually learn from the results.

To describe how this loop works, we can summarize the information received and processed by the model using four variables. At any time step t:

• xₜ denotes what the model sees, hears, or otherwise observes;
• aₜ denotes an input that may change the world, such as user control or a robot action;
• g denotes a condition, task, or goal;
• sₜ denotes the model's internal estimate of the current world state.

The action aₜ and goal g do not always appear together. A video generator may require a generation condition without executing an action, while an interactive system may continuously respond to user input without an explicit long-term goal.

In the formalization above, Imagination refers to predicting possible future observations. The model uses its history to understand the current state, predict what may happen next, and select an action. That action changes the world and produces a new observation, which updates the model's internal state and begins the next cycle.

Roadmap: How Do General World Models Evolve?

Within this closed loop, world generation, real-time interaction, and physical action become successive stages of increasing autonomy:

Generate the world → Interact with the world → Act in the world → Achieve goals autonomously → Orchestrate the world

L1 — World Generation. The model generates coherent world trajectories and learns about people, objects, motion, and scene dynamics. But looking realistic does not necessarily mean understanding causality or the consequences of action.

L2 — Interactive World. The model receives real-time input, maintains an evolving world state, and changes what happens next. Generation becomes a continuous, responsive process rather than a one-off output.

L3 — Actionable World. A World Action Model (WAM) unifies environment understanding, future prediction, and action generation. It produces executable robot actions and updates its decisions using feedback from the physical world.

L4 — Autonomous World Agent. The model proactively perceives, explores, and plans around long-term goals. It moves from following instructions to making autonomous decisions.

L5 — World Orchestrator. At L5, the model becomes a system-level orchestrator, extending autonomy beyond a single agent. Working toward a shared goal, it organizes multiple robots, digital agents, humans, tools, and resources; develops unified plans; assigns roles and resources; coordinates actions; and continuously adapts its plans based on environmental feedback.

How to Build General World Models?

Turning the five-level roadmap into working systems requires answering three questions: What data should the models be trained on? What architecture can connect the required capabilities? And how can compute support both large-scale training and real-time operation?

These questions point to three core building blocks: data, architecture, and compute.

Data: Building a Data Pyramid from Observation to Action

Internet video offers enormous scale but rarely records the actions that caused visible changes. Robot interaction data directly links environments, actions, and outcomes, but it is costly and scarce. General World Models therefore require a data pyramid that moves progressively from observation toward action: Internet-scale video → Curated domain video → Egocentric human video → Action-labeled human demonstrations → Real robot interaction

For videos without action labels, inverse dynamics, latent-space prediction, and generative modeling can recover action cues and dynamic structure [16,18,19]. Human demonstrations and robot data can then ground the model in executable action. Synthetic data expands coverage of different situations [41], although real-world interaction remains essential for calibration. Failures, corrections, and recovery trajectories also contain valuable learning signals.

Architecture: Enabling Modality Specialization and Collaborative Reasoning

Language, images, video, and robot actions use different representations, yet they must exchange information during understanding, prediction, and action generation.

Mixture-of-Transformers (MoT) assigns modality-specific parameters while retaining shared attention for cross-modal interaction [22]. This allows Understanding, Prediction, and Action to work together within a single model. The design principle is simple: retain specialization where representations differ; enable sharing where reasoning must interact.

Compute: Supporting Scalable Training and Real-Time Feedback Loops

Compute determines both the scale of pretraining and whether a model can remain inside a real-time feedback loop. Interactive generation and robot control must predict and decide before the environment changes. Few-step generation, distillation, efficient attention, quantization, caching, and streaming are therefore part of world-model design—not merely downstream optimizations [69].

From Vidu to Motubrain: Putting L1–L3 into Practice

General World Models span generation, interaction, and physical action, so no single metric can capture their progress. Current evidence comes from video consistency, real-time responsiveness, task success rates, inference latency, and generalization. ShengShu Technology has already put the first three levels of this roadmap into practice.

What Has Vidu Demonstrated?

Released in April 2024, Vidu generates high-quality, temporally coherent video, providing a foundation for learning visual regularities, temporal consistency, and world dynamics [23]. It demonstrates L1 world generation, but remains an offline generator.

What Changes from Vidu to Vidu S1?

Released in July 2026, Vidu S1 lets users intervene through speech and alter what happens next in real time. It supports real-time generation at 540p and up to 42 frames per second [24]. The important shift is not simply faster generation: the generated world can now continuously receive input and respond immediately.

How Do Motus and Motubrain Turn Prediction into Action?

Released and fully open-sourced in December 2025, Motus uses a unified architecture to connect perception, prediction, and action [25]. Motubrain, released in April 2026, further unifies environment understanding, world-state prediction, and action-trajectory generation, moving the model from visual simulation toward physical decision-making [26].

The full Motubrain model runs at approximately 5 Hz—about ten times faster than Motus—while its Video-to-Action mode reaches up to 11 Hz. On RoboTwin 2.0, it achieves average success rates of 95.8% in standard settings and 96.1% under domain randomization. It can also adapt to a new robot embodiment using 50–100 trajectories [26,74].

Together, these systems form a continuous path: Vidu: Generate the world → Vidu S1: Interact with the world → Motus/Motubrain: Understand, predict, and act in the world

What Can Current Systems Do—and Where Are Their Boundaries?

Existing systems show that generation, interaction, and action can evolve along a shared foundation-model path. However, they still operate mainly under externally specified goals and within bounded task and embodiment distributions. They do not yet provide the open-ended exploration, long-term memory, and continual learning required at L4, nor do they demonstrate the L5 capability to serve as a system-level orchestrator for robots, digital agents, humans, tools, and resources.

Outlook: Toward L4 and L5

The central idea of a General World Model is to unify world generation, real-time interaction, and physical action through a closed Understanding–Prediction–Action loop. The five-level roadmap describes how capabilities can progress from Pixel to Interaction, Action, Autonomy, and Organization.

A scalable path is to first learn broad world knowledge in a general foundation model, then use a small amount of real-robot data to align it with specific tasks and embodiments.

Reaching L4 and L5 will require progress in six areas: joint evaluation across understanding, prediction, action, and transfer; causal and physical grounding in contact, force, intervention, and failure; persistent memory that can be continuously updated; online learning from real interaction; efficient deployment under latency and resource constraints; and stronger safety and controllability as autonomy increases.

In the short term, the priority is to strengthen the first three levels while establishing the capabilities needed for goal formation, active exploration, continual learning, and long-term memory. The longer-term objective is a general foundation that can continuously understand the world, predict the consequences of different choices, act under explicit constraints, and learn from feedback.

References

The numbering below follows the original paper. Only works cited directly in this article are included; please refer to the paper for the complete bibliography.

• [5] Daniel M. Wolpert, Zoubin Ghahramani, and Michael I. Jordan. "An Internal Model for Sensorimotor Integration." Science, 269(5232):1880–1882, 1995.
• [16] Bowen Baker, Ilge Akkaya, Peter Zhokhov, et al. "Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos." NeurIPS, 2022.
• [18] Mahmoud Assran et al. "V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning." arXiv:2506.09985, 2025.
• [19] Fan Bao, Shen Nie, Kaiwen Xue, et al. "All Are Worth Words: A ViT Backbone for Diffusion Models." CVPR, 2023.
• [22] Weixin Liang, Lili Yu, Liang Luo, et al. "Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models." arXiv:2411.04996, 2024.
• [23] Fan Bao, Chendong Xiang, Gang Yue, et al. "Vidu: A Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Models." arXiv:2405.04233, 2024.
• [24] Jintao Zhang, Kai Jiang, Jintao Chen, et al. "Vidu S1: A Real-Time Interactive Video Generation Model." arXiv:2607.03118, 2026.
• [25] Hongzhe Bi, Hengkai Tan, Shenghao Xie, et al. "Motus: A Unified Latent Action World Model." CVPR, 2026.
• [26] Motubrain Team, Chendong Xiang, Fan Bao, et al. "Motubrain: An Advanced World Action Model for Robot Control." arXiv:2604.27792, 2026.
• [41] Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, et al. "MimicGen: A Data Generation System for Scalable Robot Learning Using Human Demonstrations." CoRL, 2023.
• [69] Jintao Zhang, Haoxu Wang, Kai Jiang, et al. "TurboDiffusion: Accelerating Video Diffusion Models by 100–200 Times." arXiv:2512.16093, 2025.
• [74] Tianxing Chen, Zanxin Chen, Baijun Chen, et al. "RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation." arXiv:2506.18088, 2025.