Grounding intelligence in the physical world | Runway Summit SF 2026

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Grounding intelligence in the physical world | Runway Summit SF 2026 · Channel: Runway · 25:31 · Watch on YouTube

Topics: AI models, Multimodal models, Robotics and autonomous systems

Summary

The panel explores what world models are and what problems they might solve, especially in robotics and other physical AI applications. Speakers describe different approaches, including video prediction, simulation, multimodal learning, and representations that can be used by tools such as physics engines. They discuss how to evaluate the sim-to-real gap, noting that progress varies by task and that models can exploit weaknesses in simulations. The conversation also considers recent AI capabilities, the importance of precise control and diverse data, and possible obstacles such as latency and whether world models can scale sustainably.

Key points

World models are not a single, clearly defined technology; the term can cover controllable video, 3D systems, coding models, and other approaches.
Potential applications include navigation, predicting the effects of actions, robotics, and autonomous vehicles.
World models can help make real-world data steerable and editable for training and validation.
The sim-to-real gap is task-dependent rather than binary; navigation and locomotion are seen as closer to being solved than manipulation.
Training on video is valuable, but the underlying representation—such as a rendered 3D scene or another structured model—remains an open question.
Combining video, action, language, and code may help build more general world models for physical AI.
Robotics requires capabilities beyond semantic understanding, including low-level precision control and spatial or embodied reasoning.
Reinforcement learning can produce coarse results, but achieving the last mile of precision remains difficult.
Video-based models may face practical limits because generating pixels is expensive and can introduce latency.
World models may be less influential than expected if they do not scale or cannot continually incorporate new data and compute.

Chapters

Start	Chapter	Summary
00:00	What are world models for?	The panel introduces the idea of world models and considers what new capabilities they might enable, particularly for physical AI.
05:45	The sim-to-real gap	The speakers discuss how to measure simulation quality and explain why the gap differs across tasks such as navigation and manipulation.
10:25	Prediction targets and representations	The panel debates whether predicting video is the right objective and what kinds of internal representations might support useful world models.
15:06	Beyond pixels to usable representations	The discussion considers representations that can be rendered and then used in physics simulators, video games, or manufacturing.
17:12	What recent models mean for robotics	The speakers assess demonstrations involving 3D tools and robotics, and outline the capabilities still needed for robotics.
20:49	Code, control, and precision	The panel considers code as a component of world modeling and discusses why precise results remain difficult for reinforcement-learned systems.
22:54	What could go wrong?	The speakers identify possible limitations, including latency, reliance on video, and whether world models can scale sustainably.

Quotes

world models are really good way to make real data to some degree steerable editable — 05:15

Sim to real gap is not a binary thing — 08:53

Video is infinite. — 11:57

I want to live in a house. I don’t want to look at a video of the house. — 16:08

the biggest problem with video based models, world models, is speed. — 23:26

Concepts and entities

world model, sim-to-real gap, physical AI, NVIDIA Cosmos, NeRF, Google DeepMind, Wayve, Runway, Astra

Full transcript (original language)

00:00 Please welcome to the stage Nvidia VP of Cosmos Lab Mingu, Google DeepMind principal research scientist John Baron, Wave research director Alexander Tosev, and Welch Labs founder Steven Welch. >> Hey everyone, thank you for joining us. Um, so today we have Mingu. He’s the vice president of Nvidia’s Cosmos, which is an open platform world model. We’ve got John Baron. He’s a principal research scientist at Google Deep Mind

00:30 and one of the authors of Nerf. And then finally, we have Alex Tersev. He co-authored one of the first papers using deep learning for human pose estimation back in 2013. He co-led Apple’s development of the multimodal LLM MMM1 and recently joined Wave to lead their foundation models for robotics. So excited to talk to everyone today. Um so let’s dive right in. Um this was and I’m Stephen Welch. I run a YouTube channel called Welch Labs. Um, the most popular question from our audience when I told them who the panelists were was, uh, what problems do

01:03 world models solve? So, anyone want to jump in? What problems can we solve with with world models? >> Yeah, it’s a tough one. I don’t think we have any yet, do we? Um, I mean, the the sub question, right, is like what is a world model, which I think is also completely illdefined. Should we should we workshop that? >> Let’s do it. I uh I guess like the default thing is a controllable video model, something that’s making pixels and you kind of interact with it. I gather that runway is doing a lot in this space. Um but I

01:35 think there’s lots of other bets in the space, 3D sorts of things, things like uh world labs, radiance fields, and then there’s coding models. I think these all could be world models. I think it’s an aspirational thing, you know, and uh it’s a thing that we think might exist and we don’t know what it’s going to be yet. >> Yeah. When I was thinking about this question, I thought like if we were sitting here in 2018 or something and I asked you what problem does a language model solve, right? You might say you wouldn’t say this, but you know, next token prediction, right? And who cares,

02:05 right? Um so maybe a deeper version of the question is if we can figure out this world model thing and maybe we are well on the path to doing so. Um what what new problems will become emergent? Like we had no idea that LLMs would be so good at writing code and could automate so much of software >> if we have these incredible world models. What what new problems do you guys see on the frontier? Right. So robotics is obviously a huge one. We could possibly make huge strides in robotics. Is that kind of the main one or are there other steps along the way you guys see from from world model

02:37 adoption and world models getting better? >> Yeah. So maybe I can chime in on these topics. So people live in the world and we want to solve some problems and a lot of problem we want to solve involve some understanding of the world some modeling of the world and because there are different kind of problem we want to solve. So there are different kind of war models right. So you can have a war model for navigation. So help you understand how to avoid obstacle to reach the your destination and it can be

03:08 also you want to predict what happen next if you take certain action right you know if you drop the apple into the bowl what going to happen and also there’s understanding part of it right so and I think a war model become popular because people are we are very good at the digital AI you know AI living in computers and necessarily we think about what happen next, right? And we all dream about having robot to do some task force, right? And also autonomous vehicle, you know, take you

03:41 from destination A to B without, you know, your engagement. So you can sleep all the way. And and to have an AI that u successfully complete task in a physical war, naturally we think that AI need to have some kind of war modeling there, right? So, so I I think that is why war model become so popular uh recently and and because we expect uh the physical AI to solve all kind of different uh tasks. So I think they’re

04:13 going to be also diverse war models. So I don’t think a war model is uh is a single um let me put this way it multiple model together and uh what I’m excited about is uh foundation model we know that uh if you can aggregate data uh represent different patterns together and trend uh puts the this capacity bottleneck and you sometime can train

04:44 the model to generalize to solve of unseen task. Now we are looking at so many different war models, right? If we puts the data for those war model together, will we be able to cover some of the war modeling we didn’t foresee, we didn’t have in the uh in this uh training data and how they can help physical to solve other task. >> Yeah, makes sense. Alex, is it clear how you guys will use world models at Wave? I think in you know at way if we learn we’ve been pioneers in learning for

05:15 autonomy in general for policies and now we’re starting in a robotics effort so world modeling is you know our answer to learning simulation as well >> I think the way how we think about world models at least today given the state of affairs is that it’s a bit challenging to say that we’re modeling the world but I think world models are really good way to make real data to some degree steerable editable >> and useful for both uh training and and validation in terms of synthesizing data as well as creating scenarios that are

05:45 you know quite interesting for the problems it asks right >> yeah absolutely so we’ve talked about simulation a bunch right and I think the big question in my mind certainly is how big is the sim to real gap how big of a deal is it right so let me think of maybe a good way to phrase it right so um if we do close this gap how will we know will it be obvious immediately robots we’re going to way better like how will we know if we’ve closed this sim to gap. I mean, it seems like we’re making progress, but what are what will the signs be? >> What’s the simulator gap? Like the sim

06:16 to real gap? >> Sorry, sim to real audio is a little hard. Um, yeah. How will we know if our simulators are are good enough, right? Like what will we what will we see? >> I I think there’s two gaps there that like the sim tore gap is uh probably solved by scaling video models the way they they currently [clears throat] are. That like that that’s all scaling well. That seems like that’ll just work. Um, I think there’s a bigger kind of simulator gap, which is that like the point of a simul like a physics simulator is to tell you what’s going to happen in the physical world. Yeah. Um, and uh, you

06:48 you could imagine scaling video models such that the pixels they make for you look more like the real world, but that’s not really what you want for most simulators, right? Like if I’m doing, you know, uh, aerodynamic simulation, I don’t want a video of wind going around an airplane. I want the airplane. Um, and I think that that that gap is the interesting thing, right? Like how will we actually get an airplane out of the thing and >> have verifiable physics? Um, that is something that I think is just

07:18 completely out of scope for the uh controllable video model paradigm for world models and is is kind of uh underexplored. >> Right. Right. Yeah. Absolutely. What uh what’s your take, Mu? So I’ve been thinking about how to define we close the gap. >> Yeah. >> And you can see from my earlier comments I’m more like a solving from the solving task perspective. Right. Let’s say I want to solve this physical task and I train two models. One model leverage all

07:51 the real world data and the other model leverage data from let’s say war model or simulator. If these two model uh both achieve similar quality uh in terms of the accuracy and both above the bar of potassium uh kind of threshold then I would say that seem to real gap for this particular task is solved that’s how I will define it and and you know I believe there are many different wall modeling for some of them I think we are

08:22 very close I seem to uh real gap for navigation I think that’s one is more foreseeable we’re going to be solved but synal gap for manipulation uh let’s say you compress some sponge or others in order to or try to push some cable into the the the socket those are harder right so I think there are different world model problem and some of them the real gap we are closer and some we are still far away

08:53 >> right >> yeah totally anything to add Alex >> well I think that I mean I believe the question is a bit misleading. Sim to real gap is not a binary thing, right? There is sim to real gap in various aspects. >> So if you think about robotics, simal gap is closed for locomotion. I think the current state-of-the-art locomotion policies are trained in simulation. So really have to understand exactly what you’re after and um what aspects of the problem you’re trying to solve. If I look at applications of word modeling in particular um training let’s

09:23 say with reinforcement learning um unfortunately area is pretty good at figuring out idiosyncrasies and problems in your word model. So uh it’s pretty challenging to train a strong model work model where Ariel is actually can produce a good policy because it’s going to find weird things and uh so that’s an example where you know we still struggle a lot and it’s kind of counterproductive in a sense that uh you really really need strong work models and I think there is also the question when you talk about simal gap also think about the way how we use these word models and let’s

09:54 say we should do RL are there ways to mitigate this on the learning side where you’re looking at uh learning only on the distributions where you have a strong performance of your world model. Identifying the distribution maybe have world models that even produce uncertainty quantification so that you can utilize it during your learning integrated with real data where you verify against real world unrolls and make sure what you’re learning is the right thing. So I think there is alo question of how you use them and how do you structure your learning problem so that you can actually leverage the strengths and avoid the weaknesses of word modeling.

10:25 >> Right? Makes sense. Okay, let’s keep moving here. Um, so I don’t want to completely rely on comparisons to language models, but I think it’s appropriate if world models are kind of our next frontier. Um, I think with language modeling, what was so shocking is we had this prediction target of next token prediction and all of a sudden you got these amazing emergent properties like in context learning for example. Um do we think that with do you guys think for world models do you think we’ve found the right prediction target is

10:55 predicting the next frame or you know various video-based prediction targets have we found the right target for world modeling um and how do we know right are we seeing those emergent things come from world models that are surprising us um are we predicting the right thing >> it’s so the comparison to language models is so hard because uh language models are named really well right they they model language anguage and you can imagine someone calling them thinking models and that wouldn’t be wrong but it would be confusing and you’d always be

11:26 like is it really thinking or is it just you know it’s the language model describes the the output and the input it’s uh the world model paradigm I think is not a very good one because it doesn’t describe the inputs or the outputs to the system right like I I actually think models that make the world is a good idea that’s a thing we should be doing but that’s not what controllable video generation systems are doing Uh so yeah, not sure how to reconcile that. Um I think that in terms of the target, so that’s kind of like what’s the data, what’s the loss.

11:57 >> Sure, >> I think training on video is a great idea. You should train models that predict uh the next pixel. That’s probably the the best thing we have. We have so many pixels. Video is infinite. Um I think the bigger question is is more like what is the representation under the hood that is making those pixels? Is it a diffusion model? Is it an agent orchestrating a a scene graph and then rendering it? Is it some sort of 3D thing that I think is completely up in the air and I think that which paradigm wins there will unlock different things. >> Yeah, very interesting. Um, so Cosmos is

12:30 obviously highly multimodal, right? Um, do you find so kind of returning to is predicting video enough, right? You guys use all different kinds of prediction targets and inputs. Um have you found that to be a big differentiator if you if you had a video only cosmos model or if you took away most of your multimodality how much would that impact performance you think? Yeah. So it’s all back to our belief that there are many different war models and in the mission of cosmo we want to put them together to build a great foundation uh to enable

13:01 physical developers right and there’s a world understanding part that’s a world generation simulation part and there’s also how you know the the world change when you apply certain action when the agent in the scene you describe apply certain action right so and like John said earlier uh deep learning is all about learning the representation right when you stack uh transformer together uh and to ser uh solve certain u

13:32 training objective uh the next token prediction and you have a uh amazing representation learn uh you can use it for other task right and and for the war model it’s a bit more complicated not like a purely language what like what John described earlier uh it’s in the real world and real world war uh the most uh uh you know available form of data to earth is the videos the internet video huge amount but when you look at the internet videos you have the

14:04 raw pixel but you don’t observe the action taken to cause the change what you observe right so so there’s also some hidden scenes there so my view is more like a this is a challenging problem and we’re going to take whatever objective we need uh to learn a good representation right and it can be sue next token prediction it can sue from the division objective and can also be pursue like a regression predator action so my view is more like a this scene uh

14:35 put together uh and s the information bottleneck to force to learn a good world representation both good for understanding and the simulation task >> right right that makes sense And just going back to the representation comment that you made, John, um, and possibly Alex could weigh in here, too. Um, it seems like the diffusion transformer and auto reggressive variants of it. Um, seem pretty effective right now, like they’re solving a lot of problems, right? And it seems like the recent interviews I’ve done, it kind of seems

15:06 like a lot of focus is turned on to getting the right set of data to really scale, you know? Um, but elaborate more. So, if if it’s not a diffusion transformer, right? What obviously you worked on Nerf John? Um, what do you see as kind of the main alternatives to a diffusion transformer being the way we learn the representation? >> Sure. Uh, I think diffusion and transformers are probably here to stay. Those are very mature technologies. It’s not obvious that they would be displaced by anything. >> Um, but I think you still have a lot of choices around what what they’re doing. >> Oh, I misunderstood. I’m sorry. I

15:37 thought you meant replacing the whole thing. But yeah, tell us more about Yeah, >> but like uh I I think the the the question of using a diffusion transformer to make uh latent that are then decoded into pixels is a very specific choice. >> I I am most excited about the idea of you know diffusion transformers and all that good stuff where um so the problem with the video paradigm right is I can’t reach into the video and grab stuff. I can’t like do anything with this video of a house. I want to live in a house. I don’t want to look at a video of the house. Um uh you can imagine and this is

16:08 a thing that people are starting to build uh systems where the pixels are not some uh neural blob that maps latence into pixels in a way that’s uninterpretable but is just a rendering engine. And you know this is how Nerf works. This is how a variety of work. Um that that I think is underexplored. It’s hard to do because you have to constrain yourself to obey physics. But then the representation you you recover is interoperable with everything else that we have. You could throw that into a physics simulator. You could throw it into a video game, right? >> Um and there like that’s going to be a

16:41 huge unlock for uh all sorts of digital experiences, but also just making stuff physically in the world in like a manufacturing context. >> Absolutely. Okay, great. Um in the interest of time, let’s keep moving here. Um, so there was a lot of noise over the last few weeks about the Astra release, especially using uh OpenAI Astra as a robotics policy kind of out of the box. Um, and then also some noise around Astra being um pretty effective at CAD 3D kind of applications. Um, did

17:12 the Astra release was that kind of an update for anyone about what you thought kind of a general purpose agent could do in terms of either robotics or 3D? >> It certainly is surprising, right? like it’s a very striking set of results. Um I I think of Astra as or all these kind of coding agent sort of models as being basically computer use um via the normal tools that we have for gamedev and movie making and things like that. Um and it’s not just 3D like there’s those clawed

17:42 results of like where it makes videos out of like kind of vector graphics. Uh and I think that that’s cool like that’s a totally reasonable paradigm for doing that. Um my my concern would be that uh I don’t really see a path towards those things kind of transcending the limits of existing like 3D technology like >> uh like we’ve already done the experiments as a society of having like a thousand people work for a decade to make a video game that looks as realistic as possible. That’s like what Grand Theft Auto 6. It’s like like that’s that’s what you get and it still

18:13 doesn’t look like reality. It looks like a video game. Um, and I think that Astra is like, well, you could do that in an hour now. >> Sure. Um, which is really cool, but uh, I think that it’s, you know, making stuff out of these primitives you already have of scene graphs and triangles is kind of there’s a ceiling there. And I think that, uh, it’s interesting to imagine, you know, what gets you past that. I think whatever is past that will still be orchestrated by an agentic LLM who’s kind of >> painting with those those paint brushes, but it’ll be something else, I think. >> Yeah. So, Alex, I know you mentioned

18:43 that hiring is a big focus for you at at Wave right now. Um, why not just use Astra? Do you need a team? I mean, can Astra just be your policy? >> First of all, I don’t want to comment on the latest greatest models of the week. I think we obviously have recency bias and next week there will be something interesting. But really, I think there are two components. Um, in order, let’s say, to solve robotics, you need you need models that have certain capabilities. And occasionally we see a step function in these capabilities and people get really excited and they believe that step function can enable new things. And

19:15 maybe that’s true, maybe that’s not true. Um I think in order to solve robotics, you need the ability to semantically understand the scene and models have been pretty good for this for a number of years now. U you need also the ability to do low-level control, precision control and foundation model, frontier models are not good at it yet. There are a lot of companies and efforts that are trying to tackle this directly because you need dedicated efforts and data for that. Another capability is spatial or embodied reasoning which goes beyond

19:45 semantic reasoning being able to understand spatial relationship motions uh physics in um kind of from perceptual standpoint. Frontier models haven’t been very good at this. Um I think there it’s an active area of research and I think you know we’re going to see some progress and I think astr demonstrating a strong jump in that domain. Another capability is inverse physics. The ability to take real data, convert it into simulation and do some work there and go back. I think um you know the models today seem to be doing better job. So there is a step function there.

20:16 Now put all these ingredients together. Let’s try to see how much we can solve robotics. Most likely we wouldn’t be solving but I think we we see we see progress and it’s important when you see a step function in some of these capabilities to give it a go and see whether where the things are failing, right? So it’s an exciting time and we’re going to see step functions you know >> elsewhere going forward. >> Absolutely. Among you what was your response to Astra? >> Yeah we have been uh watching people share their successful results of using Astra for many tasks I didn’t think uh

20:49 language model was capable to do and uh to be honest um I think they opened a new window to me. So in in the past when I’m thinking about war modeling I mainly concerned about language and video right video uh also action right video and action come together uh those are raw data and language help you understand what happened there or control what you want to happen uh in output and then also about code is an important part of

21:22 war modeling as well right and what show is that if you have a powerful coding agent that can write 3D uh code uh do simulation, it actually do some sort of war modeling and it also allow you to represent your understanding of war in the form of code and the system of foundation model is a powerful model that puts everything together and then the synergy right so now it suggest that actually code is also one important component that we should bring in into

21:52 the uh the war models the more general form a war model to help physical AI developers. So, it’s really shocking uh and but also I think it’s going to u change how many physical AI uh task is solved moving forward. >> Absolutely. Yeah. Yeah. I thought um just the fact that an LLM can do that off the shelf was pretty shocking to me. You’re totally right, Alex. The fine fine grain control seems, you know, a limiting factor, but um yeah, I was it’ll be interesting to see what happens next. >> It’s also not off the shelf, right? I

22:23 mean, like it’s trained. It’s RL for these tasks. >> Oh, sure. Great point. >> Uh like it’s it’s a very intentional like thing you have to try to do, >> right? It wasn’t wasn’t an accident. Yeah. Good >> qualification. The precision thing is is uh related to that because like >> the fact that they’re RLing, you have to RL for these tasks the way they’re set up, right? Because I can’t get gradients to flow through the generation of code and the execution of code to the pixels unlike say video models which you can just do gradient descent. Um, and I think that’s why there’s always they’re always off by a few pixels because like

22:54 it’s just a very like RL is a extremely weak supervisory signal that let you get things coarsely in magic ways, but like the f the last mile is very very hard. >> Totally. Y Okay, we are in between you guys and lunch. So, we’re going to do one more question and then lunchtime. I promise. Um, so taking a contrarian view for a minute. Let’s say we’re all back here in 2030 and world models are have gone poorly. Let’s say it didn’t work out, right? It was a detour. Something else came along. Um, what’s the reason? Why would why would world models, as

23:26 we’re kind of looking forward into the future, scaling up video, we think this is going to we’re going to close the sim to real gap. Why would this not work? >> I I think the biggest problem with video based models, world models, um, is speed. They are very expensive and slow. Even the ones that are real time >> have a latency of 100 milliseconds or so, 200 millonds. Um, that’s a really big problem compared to text where um the the output to a text query can be just a handful of words um and can

23:57 really change your life, right? Like like the the text is a very good medium for this. Uh pixels are very expensive to make and you need a lot of them to do anything with. I think that’s the biggest problem, >> right? Okay. Latency. Got it. Okay. Mingu, Alex, what could go wrong? >> Well, I think one thing I would encourage us is to go beyond video. I think world modeling world is not only um visual data. There is a lot of sensor inputs especially in the physical world behind closed doors in companies that actually capture the world in many different ways and using that data is going to be quite important. I think we

24:29 start with the data that is more plentiful like language and video but we should not forget that we’re actually trying to model the world and there’s a lot more signals to utilize in addition. >> Yeah. Okay. Last word goes to Mingu. What could go wrong? I I don’t think it will go wrong but uh [laughter] so uh because I think physical eye is a problem that the whole community the whole society determined to solve right and to solve physical eye problem you need to have some sort of modeling of

25:00 the world and and what could go wrong is maybe war model doesn’t play uh as important role as we expect today and that could happen if war model doesn’t scale we don’t have a kind a sustainable way of continually ingest new data, new compute to make it better for our task. If that fail, uh, it may not play a big role. That that’s one possible scenario, but I think it’s unlikely. >> Got it. Yep. Well, we got the red light. Thank you for your time today, John, Mingu, and Alex. This was great. Uh,

25:31 looking forward to doing it again. And hopefully it’s not that, you know, path we talked about and world models are crushing it in 2030 and we’re talking about how we’ve onto the next thing. Thank you for your time, guys. Thank you.