Simulating Reality: World Models Rise

Experts explain how they work, what they can do, and what’s still unsettled.

Over the past few years, many of us have gotten a crash course in what we now call artificial intelligence—but really, it has mostly been a crash course in large language models. Increasingly, however, LLMs are no longer the only category of AI drawing high expectations, massive funding rounds, and significant research and product development.

Over the past year, we’ve seen a plethora of new announcements in a category labeled “world models,” and you’ll likely see more movement there in the coming months and years.

Instead of or in addition to working with language, world models aim to lay the groundwork for AI systems that are capable of simulating the physical world, or at least a useful approximation of it.

To examine what’s different and important about this idea, Ars spoke with three expert practitioners working on world models and related technologies: Vincent Sitzmann from MIT, Anastasis Germanidis from Runway, and Ben Mildenhall from World Labs.

From these conversations, we learned that while LLMs-as-a-product started with an interface (chat) and then sought a use case, the big players in world models right now are arguably working in the other direction: They’re starting with specific use cases and applications in robotics, research, and asset generation, but it’s unclear exactly how the interfaces, systems, and tools will ultimately look.

As you’ll soon see, there are many parallels between LLMs and world models in terms of architecture and how people expect them to improve over time. For some, though, they’re seen as a potential answer to the limitations of LLMs, even though work on them predates that contemporary narrative.

“The idea that you’re going to extend the capabilities of LLMs to the point that they’re going to have human-level intelligence is complete nonsense,” former Meta chief AI scientist Yann LeCun told Wired earlier this year. LeCun has made waves with an opinion that some working in AI and LLMs see as contrarian, but he’s actually speaking for a sizable segment of the field.

See also Fei-Fei Li, the computer vision pioneer who co-founded World Labs, one of the new companies working on world models. In a Substack post late last year, she wrote: LeCun and Li’s ventures are built on these ideas, so it’s not surprising they’d say these things. But you’ll also see similar sentiments from some prominent figures still working primarily with LLMs.

“I think we’re in an LLM bubble, and I think the LLM bubble might be bursting next year,” said Clem Delangue, the CEO of Hugging Face—a platform that hosts repositories of LLMs of all stripes.

“But ‘LLM’ is just a subset of AI when it comes to applying AI to biology, chemistry, image, audio, [and] video,” Delangue added, speaking at a conference. “I think we’re at the beginning of it, and we’ll see much more in the next few years.”

Over just the past few months, world models have advanced from a research topic (which they still are, of course) to the basis for new commercial projects and huge funding rounds. A few key examples: These and similar efforts have received substantial funding. World Labs and AMI reportedly raised around $1 billion each in February and March, respectively, and Runway also raised $315 million in February.

Some of the activity around world models is at least in part aimed at ostensibly building the foundations of AGI or superintelligence, but most people working on them are talking about practical applications: training, testing, and driving robots; generating 3D assets for game development and film production; scientific simulation and modeling; and so on.

It’s important to note that “world models” is an umbrella term that is often thrown around without a clear definition, though.

“It’s definitely an overloaded term,” Vincent Sitzmann told me in a lengthy conversation about the research and concepts underlying world models.

Sitzmann is an assistant professor at MIT who has published research on neural rendering, visual computing, and robotics. He leads the Scene Representation Group within MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL).

When asked to give a definition, he simply described a world model as any model that takes in an interaction, “and given that interaction, it enables you to simulate what would happen next in some environment.”

When announcing its GWM-1 family of models in December, Runway defined a world model as “an AI system that builds an internal representation of an environment and uses it to simulate future events within that environment.” Further, “the aim of general world models is to represent and simulate a wide range of situations and interactions—like those encountered in the real world,” Runway added.

I also spoke with Ben Mildenhall, co-founder of World Labs, a former Google computer vision and physics researcher and co-creator of neural radiance fields (NeRF)—a method for constructing navigable 3D scenes from 2D images in a format that is differentiable and therefore useful in machine learning contexts.

“The key things that would distinguish it from an LLM are demonstrating degrees of spatial and—maybe for lack of a better word—continuous understanding,” he said. “A very distinguishing aspect of interacting with an LLM is they are turn-based.” Meaning, users type some text, there’s a pause, and then they get a block of text back. By contrast, he sees a world model as a synchronous, real-time system.

“Something that would define a world model is the degree of freedom that you have in interacting with the spatial world, where you do not have this mediated linear journey of A then B then A then B then A then B,” he said. When a user or agent is utilizing a world model, they are “actually able to interact with it like it is some sort of world and you are taking continuous actions” where “there are parallel things happening at the same time.”

Mildenhall’s co-founder Fei-Fei Li has written that she believes there are three criteria that define a world model: World models “can generate worlds with perceptual, geometrical, and physical consistency,” “are multimodal by design,” and “can output the next states based on input actions.”

The phrase itself is not new; it has long appeared in reinforcement learning and robotics to describe models that predict environment dynamics. What is new is the attempt to scale that idea into general-purpose, generative systems trained on massive visual and multimodal data. The huge funding rounds are relatively new, too.

The truth is that there are numerous approaches and definitions. “Ask Runway, ask us, ask whoever—we’re all gonna give you a little bit of a different term,” Mildenhall said.

Further, it’s important to be aware that “world model” is becoming a branding term as much as a technical one. It’s used as a marketing label that can cover anything from action-conditioned video generation to 3D asset creation to robot policy evaluation. While there is conviction that one foundation model ought to be able to solve for a lot of these things, companies are using the term while solving different technical problems with varying approaches.

“Today, what most people mean when they say ‘world model’ is generating pixels, so, like, generating a realistic video conditional of the actions,” Sitzmann said.

We’ve seen video generation models gain traction over the past couple of years. Runway, for example, built its reputation as a company that makes video models and related tools that are used by filmmakers, advertisers, and others in creative fields. Now that focus has shifted, as the company has made clear its intention to expand beyond those areas to focus on world models like GWM-1 as well, with an eye toward applications in robotics and beyond in the coming years.

That might seem like a jarring lateral pivot, but if you consider how these world models are being developed, it’s a natural next step. In many cases, they’re a direct extension of the work previously done on video models.

“One of the moments we started realizing there was something beyond this short-form video generation task was at the point we released camera controls, so you could direct how a camera moves in the scene,” said Anastasis Germanidis, Runway’s CTO, when I spoke with him about GWM-1 back in December. “It started to feel more almost like a video game and you’re exploring this world and you move around. It feels less that I’m just creating a video and more that I’m exploring an environment that’s continuously being rendered by this model.”

Most video diffusion models generate many frames simultaneously, denoising the sequence in one go.

(Denoising is the core mechanism behind many AI image and video generators. The models begin with an image or frame of static-like random noise, and the denoising process iteratively predicts cleaner versions, guided by patterns learned during training, until the image resolves into a finished output.)

This approach has advantages for coherence and quality because every frame is generated within the context not just of what past frames were but what future ones should be. That’s great for generating a fixed-length video, but it’s not a fit for the open-ended simulation and interactivity that a world model requires. There’s no way for users to meaningfully intervene with input and see an immediate result.

A widely subscribed-to answer for video-based world models is autoregressive diffusion, which is still a denoising process, but the frames are denoised sequentially.

This approach also has some disadvantages. It’s extremely compute-expensive, as each batch of frames requires a full denoising process, and each needs to resolve quickly enough that the user experiences a timely response to their inputs.

Further, the simulation will drift and forget information from early frames over time. “If you’re in a room, you exit the room, you go back to the room, you need to make sure that all the details of the room are preserved inside the context and the memory of the model,” Germanidis said.

These are wrinkles that are still being ironed out, even in the largest, most cutting-edge frontier models.

“Our overall approach is that 3D consistency and that statefulness should emerge from scale and from just predicting 2D pixels,” Germanidis said. “So we should try to, as much as possible, not introduce any specific techniques that tell the model this is a 3D scene and you need to have this 3D representation that you’re using to guide the model.”

This reflects a widely accepted idea in the field called “the bitter lesson,” which originated in a short but influential essay by computer scientist and reinforcement learning pioneer Richard Sutton.

Citing multiple historical examples, Sutton argued that researchers’ intuition often leads them to try to explicitly incorporate prior human knowledge about the world or our own minds when designing AI systems, but that has “proved ultimately counterproductive, and a colossal waste of researchers’ time, when, through Moore’s law, massive computation became available and a means was found to put it to good use.”

When “breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning… the eventual success is tinged with bitterness, and often incompletely digested, because it is success over a favored, human-centric approach,” he wrote.

That does not mean explicit geometry or physics are useless in products or hybrid systems; it means many researchers are wary of building the core learning problem around hand-specified human abstractions if a more scalable objective can learn useful structure from data.

This is a key idea behind much of the current AI Spring. LLMs, for example, are not given robust, explicit prescriptions on grammar, or an overly structured map for interpreting what matters and why about every point of knowledge in their datasets. They are instead given a methodology for working that out themselves, facilitated by massive computational power of increasing scale.

Sitzmann told me he believes this is also a cornerstone of the argument against making explicit 3D data or articulations of the laws of physics the basis for training world models. He argued: For this reason and others, many research groups and companies working on world models today are using a process very similar to what Germanidis described. However, not all world models are about outputting videos to a user.

“There are other kinds of ways in which one could understand it because generating pixels to a degree is maybe not the most important thing,” said Sitzmann. It can depend on the intended use case.

For example, some models are aimed at simulation for robotics or research without generating a video that any human operator sees. Others still involve video to a significant degree but incorporate more explicit 3D information, like data related to positioning, meshes, and so on somewhere in the process.

Other world-model approaches, such as Yann LeCun’s JEPA, try to predict the abstract state of a scene rather than generate every future pixel. The idea is to learn the structure that matters for prediction and planning without spending huge amounts of compute modeling irrelevant visual detail. It’s one of the more prominent alternatives to the generative video-model approach, and it’s something we’ll likely examine in more detail in a later article.

Some video-based models are built to output actual 3D assets or data in addition to 2D video in the form of meshes, Gaussian splats, or something else; that’s the main product focus for World Labs at the moment, though the company’s ambitions for the future are broader.

Speaking to Ars, Li’s co-founder, Mildenhall, described the explicit 3D aspect of the company’s current tools and products as simply a matter of giving users an off-ramp to an export format for now. “We choose to bake out to 3D so that we can go talk to a bunch of people who have workloads today that they can make use of this persistent thing they can share, edit, integrate, render against, and so on,” he explained.

World Labs’ publicly usable systems use video and image data to infer 3D structure and output explicit scene representations using techniques such as neural radiance fields (NeRFs) and Gaussian splats so that users get assets that work with traditional 3D workflows.

Marble, World Labs’ product you can jump into right now, takes in an image or a video, along with an optional text prompt, and creates a space roughly the size of a typical American suburban backyard. You can then fly around in it with a keyboard and mouse or export it as a 3D asset for use elsewhere.

As for why World Labs is using representations like NeRFs and Gaussian splats rather than conventional meshes, Mildenhall said: Producing an asset that can be brought into current workflows gives World Labs’ product the potential for immediate application by game developers, VFX artists, and professionals in other creative and technical disciplines.

However, “one disadvantage of having a 3D representation is clearly that it’s difficult to have things that move in these scenes,” Sitzmann told me.

Once the environment is baked, basically nothing in it is truly interactive in terms of physics or other mechanics—those dynamics must be brought in later with another tool, if at all.

Runway’s approach doesn’t have as obvious a use case for regular users. Its model, which isn’t yet publicly available, functions like a livestream: You input actions as it streams, and the results appear immediately.

Runway’s approach currently harbors an advantage for dynamic simulation, though.

“A lot of the world models of today have focused on kind of navigating this static, lifeless world. And we think that actually simulating the world well and building good world models means being able to generate dynamic worlds where things happen, where you can take actions and see the consequences,” said Germanidis. “That’s the biggest differentiation around our announcement compared to some other world model announcements.”

When I mentioned to Mildenhall that the environments that were generated for me to explore in Marble were completely static, he nodded.

“Very much true right now, right? We output these static Gaussian splats, the end artifact itself is static,” he acknowledged. “We’re definitely working very actively on figuring out how to get fourth dimension into there.”

It’s a spectrum with tradeoffs on both ends. Mildenhall explained: There’s also the fact that the pure frame-based approach is immensely computationally expensive. It remains to be seen whether it will be economically viable in some applications, compared to Marble’s bake-and-forget approach or LeCun’s competing JEPA architecture.

“The key benefit of having a 3D scene is that you need to generate it only once,” Sitzmann said. “And once you’ve generated it, it’s really fast to render it… you don’t need any neural networks anymore.”

At that point, it can render locally on consumer 3D hardware. This is a big plus at a time when cloud compute is at an absolute premium amid exploding demand, and it’s a practical advantage in terms of getting the technology into users’ hands in meaningful ways right now.

On the other hand, if you’re building a model for simulations in robotics, many argue that these explicit 3D representations matter a lot less than they do for content creation or asset generation.

“In order to simulate performing different manipulation tasks, you need to have the robot perform actions and have the interaction with different objects be simulated well. And you need to be able to simulate failure very well as well,” Germanidis said.

But let’s be clear: Whether explicit 3D structure is included in the training process or as an output artifact (or not at all), video is still the foundation for what many of these commercially oriented teams are currently doing, including both Runway and World Labs.

“The underlying technical backbone has a lot in common across most of these approaches… it’s also just diffusion-based stuff,” Mildenhall told me. “It’s not like a fork at the root. It’s really like a branch in the application side.”

Both World Labs and Runway—and many of their competitors—are working on multiple models, and they have their eyes on potential applications in robotics.

That’s because, while the hype might seem to suggest that robotics is poised for a major expansion, some avenues of progress are bottlenecked by a lack of useful data for training robots for complex tasks.

Compare it to self-driving cars: The recent progress we’ve seen on that front is possible because most situations a car encounters fall within clear guardrails, and even when it encounters more out-of-left-field events, the potential responses are relatively limited, and there are specific guidelines to follow for almost every circumstance. Useful training data is abundant, as it can be obtained from sources such as dashcam videos and other sensors, or from human-supervised driving on the exact roads the automated vehicles will ultimately take.

Now take a humanoid robot operating in a home or a workplace as a contrasting example; it is a much more complex machine, with a more sophisticated range of motions and actions in the face of potentially far more varied situations and environments. There is comparatively little useful training data tailored to this purpose. You might have read news stories about AI and robotics companies offering incentives or paying people to wear cameras while doing chores at home; those companies are trying to get training data. However, that approach is unlikely to produce enough data on its own.

World models trained on video could be part of the solution, in that they are a potential fountain of training data, provided it can be demonstrated that their simulations are sufficiently representative of the real-world scenarios the robots will encounter.

“You might want to use a video model to generate a lot of synthetic data of the robot performing different kinds of tasks,” Germanidis said. “That’s very hard to collect in the real world. That will take a lot of time. That might be unsafe.”

The hope is that the world model can safely produce the needed data far sooner than any known alternative approaches.

Physics are key for robotics and other areas of physical AI. Any autonomous physical unit must demonstrate something at least analogous to an understanding of the laws of the physical world. Intuitively, it might seem odd to say that something built primarily from video data could develop anything resembling physical understanding, but many researchers and the companies building products on top of their research believe that video models appear to demonstrate a genuinely useful ability to reflect or predict real-world physics, at least when looking at their inputs and outputs.

“In order to solve this problem of predicting the next frame of a video very well, you need to basically predict so many aspects of the physical world, so many aspects of how objects move, how people move in an environment, physics,” Germanidis told me.

These models are typically not trained on or constrained by explicit physics rules, data, or simulations—remember, the bitter lesson. Rather, proponents of this approach say that there is a useful-enough approximation of understanding in latent space.

When asked to define this term for those outside of the AI world, Sitzmann said: If you looked into your own brain, if you could—which we can’t, but if you could—you would probably not find in your own brain a mesh of your environment. You would find something that is kind of a weird thing that still somehow contains certain information about the 3D scene, but it would not be in a human-readable, human-accessible way. It would not be stored in some format like an explicit 3D scene. And so when people say latent space, that is kind of what they mean.

You don’t tell the model how that 3D representation should look. Instead, kind of like your own brain, you force the model to have a certain input/output behavior—you force it to always predict the next frame of a video, and then if you make the model big enough and you let it figure it out itself, then the model will figure out itself how it wants to represent 3D scenes.

“Latent” simply means not directly visible in the input or output. In this context, it usually refers to hidden internal representations—activations, embeddings, or compressed states—that the model develops as it learns to predict future frames or actions rather than to an explicit human-or-robot-readable output.

Strangely, it’s not explicitly clear to human observers how the model can accurately simulate physical behaviors, but proponents of this approach argue that as long as you can prove the outputs will be useful, it’s not a disqualifying problem that we can’t fully grok the internal workings of the models.

Germanidis described methodologies for testing models like GWM-1 to demonstrate their usefulness for simulation and for training the policies that control robots.

First, he said you can use a model to perform a specific task in the real world and try the same inside a model like GWM-1. If there’s a correlation between the success and failure of a model in a real-world scenario compared to the same model in a simulation, you gain some confidence that you can use the model to test policies. “You can do the same with the synthetic data case,” he said.

Sitzmann described similar scenarios. “The way this will be proven useful is that people will increasingly implement systems that use these models for doing robot tasks in a variety of different ways. Then we have a benchmark, we run the benchmark, there you go,” he said. “This is, by the way, also how it happened with self-driving cars, right? With self-driving cars, what in the end happens is you just show empirically that it works.”

While we’re still in the early days of this field, a growing body of research suggests that video and world models can already be useful for synthetic data generation, policy evaluation, and some robot-planning tasks. There’s enough to justify cautious optimism about these models, though it’s worth noting that their reliability as general-purpose physical simulators remains unproven.

Synthetic data could still teach the wrong lessons if it misrepresents contact forces, surface friction, or any number of other things. Small mismatches can break policies, and closing that gap in wide-ranging scenarios is still not completely solved. Further, real-world situations will inevitably present unexpected variations.

This is one of many reasons why impressive marketing videos from robotics companies showing robots—humanoid or otherwise—performing tasks previously thought impossible can be misleading. Everything from the environments to the tasks to the specific objects being manipulated can be carefully selected to show just what’s possible if all this could be sufficiently generalized—but that generalization is not yet proven.

A useful world model for robotics should improve policy learning or policy evaluation on held-out tasks, in environments it was not tuned for, with failure cases included rather than edited away. It should also be compared against simpler baselines like conventional simulators.

Without that, a demo can show that a model has learned enough structure to be compelling, but not that it is the most reliable or scalable way to build physical intelligence.

While many researchers and investors are nonetheless optimistic about the potential practical applications of world models, there are also unresolved questions about representation—what the interfaces and applications actually look like for users.

“The foundational architecture—there’s transformers in there, right? There’s diffusion stuff. I think that’s almost like less of a question,” Mildenhall told me when I asked about the unknowns for the technology. “It’s really more, how do you control, how do you train and devise the input-output structure of what really matters for your model?”

Sitzmann made the same point in our conversation. “No one needs to be explained how to interact with an LLM,” he said. “You ask them a question, they tell you something. You ask them to code for you, they do something. For world models, maybe not as much.”

Mildenhall called this representation question the real “kicker.” There is likely no one-size-fits-all interface for world models’ various future applications. In terms of human interfaces, some people may interact with what they’re generating via video game controllers and a virtual avatar or the same control scheme they use to play 3D games on a PC. Others will want a Blender– or Unreal-like scene view. And then there are the applications where the interface is for robots, software, or other AI models, not humans. Very little of this is settled.

There’s also the open question of whether there will be an emphasis on highly generalized, horizontal models that apply to a wide range of domains, or if companies and researchers will decide to break them up into specialized models.

Runway announced a trio of models under the GWM-1 banner rather than just one. One handles human user interaction and navigation in real time within what appears to the user as a video stream, another focuses entirely on generating realistic human-like characters for combined video and voice synthesis, and another is trained for policy execution and evaluation for robots.

“We’re big believers in having one big model that solves a lot of tasks versus having a lot of specialized models,” Germanidis told me when discussing GWM-1’s split. “The reason is, first of all, it’s just easier to deploy one model instead of having to kind of educate users that they need to use different models for specific things. But we’ve seen that there is a lot of benefit from the learnings that the model has from one domain that translates to others.”

But practicality and speed-to-market won out for Runway, with a focus on meeting immediate customer needs. Germanidis said the company plans to unify the three models into one generalized model later.

Mildenhall has the same attitude about the value of pursuing a unified model. “Very reasonably, there is one model where you would be running every stream of pixels through it and be getting some kind of foundational knowledge or infrastructure out of that that you can layer things on top of,” he said. “To me, it would feel like we’re doing something too bespoke if we’re seeing a lot of fragmentation and more than a little bit of fine-tuning to have to adapt to those different use cases.”

In fact, Mildenhall went so far as to predict that AI tools like these will help reduce fragmentation within the existing industries doing work with 3D assets and simulation.

Here’s an example: A company like Disney makes Star Wars movies and TV shows, but it is also involved to some degree in working with game studios and publishers to make Star Wars video games. In theory, you might imagine that some 3D assets used in the films could be adapted for use in the games.

But that’s not how it works, for several reasons. Historically, the two use different formats and software and have different levels of detail depending on what type of machine is rendering them and whether the rendering is pre-baked or being done in real time. These assets are enormously expensive to produce, so it would be significant if the differences could be bridged.

“A lot of the things that are issues I think will be solved by more automated tooling that comes with AI,” Mildenhall argued, while also noting that Epic—the company that makes the Unreal Engine and related tooling—has already been working on trying to consolidate much of this.

All of this is to say that much is still up in the air. The investment is pouring in because that’s the only way to see if there’s “a there there,” as the saying goes. And there are promising signs giving researchers and commercial entities reasons to be optimistic.

When I spoke with Sitzmann about competing approaches within the industry and the field—like, for example, between using explicit 3D or physics data in training versus relying on the latents in the video models—he said the latter is what he’s currently most optimistic about as a researcher but that it’s still early days.

“I’m saying it’s a bet. This is not a decided thing,” he clarified.

The notion that companies and researchers are placing bets—some more proverbial than others—doesn’t just apply to the question of explicit 3D or physics in training data. It also applies to figuring out how human users will see and interact with what the models generate. It applies to product decisions about how these tools will fit in current 3D workflows in fields like game development. It even applies to whether the desired output for a specific application is actually video or some other kind of data, or even whether autoregressive diffusion and video models are a key part of building working products in the world of physical AI.

Sitzmann further addressed this topic in a blog post he published after our conversation, writing: People are making these bets because the potential upside of getting it right is huge, both in terms of commercial success and solving problems. Setting aside more speculative applications of world models, such as scientific modeling or healthcare, there’s reason to be hopeful that they will have applications in robotics, manufacturing, and other areas under the physical AI umbrella, even if they are neither the entire nor the final solution.

AI researchers often talk about “AI Springs” and “AI Winters”—periods when progress in the field is either booming or stagnant. The past few years have been an AI Spring, and most people see that in large language models, but the bets being placed right now make it clear that many people in the field believe this AI Spring might not end with LLMs.