Introducing Genie 4

Author: Google for Developers · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Introducing Genie 4 · Channel: Google for Developers · 47:15 · Watch on YouTube

Topics: AI content generation, AI models, Multimodal models, Robotics and autonomous systems, AI safety

Summary

Google DeepMind introduces Genie 4, a real-time interactive world model that builds on Genie 3 with more photorealistic environments, generated audio, and new ways to steer scenes. The discussion explains how world models simulate grounded environments and how Gemini can work alongside Genie to interpret user actions, guide dialogue, and influence what happens next. Demonstrations include first-person exploration, conversations with characters, creative scene changes, and robot-task simulations. The team also discusses potential uses in robotics, education, and filmmaking, along with latency, safety, evaluation, and plans to offer Genie 4 through Project Genie and an API.

Key points

World models represent the state of an environment and predict how it changes when objects interact or a user takes an action.
Genie developed from 2D platformer environments in Genie 1 to 3D scenes in Genie 2 and real-time interactive worlds in Genie 3.
Genie 4 adds synchronized audio and video generation, including ambient sound and experimental character dialogue.
The model’s environments are more photorealistic, building on the Omni foundation, though the team says there is still room to improve.
Users can steer scenes through first-person controls, camera movement, and text; Gemini can help interpret actions and shape scene responses.
Potential applications include generating robotics training scenarios, supporting robot action planning, education, and creative pre-visualization.
The team describes a trade-off among frame rate, input latency, responsiveness, and the cost of running the model.
Combining generated pixels with code could help preserve world state and consistency while retaining the expressiveness of generated environments.
Genie 4 is planned for Project Genie and an API; outputs will include SynthID, and the rollout will use trusted testers to gather feedback on safety and edge cases.
Evaluation remains an evolving research challenge, and the team hopes game-playing agents will help assess Genie-generated worlds.

Chapters

Start	Chapter	Summary
00:00	Introducing Genie 4 and defining world models	The hosts introduce Genie 4 and explain how world models simulate grounded environments and the effects of actions.
03:13	How Genie developed from 2D to real-time worlds	The team traces the progression from Genie 1’s 2D environments through Genie 2’s 3D scenes to Genie 3’s interactive worlds.
05:19	Audio and realism in Genie 4	A music-scene demo highlights Genie 4’s synchronized audio, improved realism, and the role of its Omni foundation.
09:37	Real-time steering and first-person interaction	The speakers discuss pausing and steering scenes, then demonstrate first-person navigation and interactions with characters.
14:27	Gemini’s role and potential applications	The team explains how Gemini can interpret actions and guide responses, with possible uses in education and robotics.
17:02	Creative scene changes and robotics simulations	Demos show transforming a filmed set and generating a simulated robot task, illustrating creative and training possibilities.
24:01	What has improved and what remains challenging	The discussion covers photorealism, freeform controls, audio, consistency over time, and the need for low latency.
27:12	World models, code, frame rate, and latency	The speakers consider combining generated worlds with code and explain the trade-offs among frame rate, compute, and input latency.
31:27	Maintaining world state and exploring open-ended experiences	The team discusses how code could preserve elements of a world while generated scenes retain their expressive flexibility.
34:04	Availability, SynthID, and real-time safety	Genie 4 is planned for Project Genie and an API, with SynthID outputs and safety measures developed for real-time use.
38:15	Steering scenes and serving different users	The team discusses using Gemini to monitor or steer scenes and the need to tailor deployments to different user needs.
41:21	Testing prompts and evaluating Genie 4	A restaurant-kitchen demo explores scene dynamics and dialogue before the speakers discuss evolving evaluations and automated testing.
46:09	Agents and closing remarks	The team describes using game-playing agents to evaluate generated worlds and closes by inviting developers to try Genie 4.

Quotes

It needs to understand what is the state of the world and what happens when things interact in that world — 01:37

the Genie 4 model predicts both the video and audio together from the same model. — 08:28

it’s the shift from exploration to simulation almost. — 24:32

we want it to be really low latency so that it’s really reactive — 26:40

we actually want those agents to be able to eval Genie worlds for us. — 46:09

Concepts and entities

Genie 4, Genie 3, Genie 2, Genie 1, Gemini, Gemini Omni, Project Genie, SynthID, Argon, SIMA, Google DeepMind, Logan Kilpatrick, Raia, Diego, Hang

Full transcript (original language)

00:00 If we wanted to put all the TPUs for one, just for Logan, - Just for me. - could we run it at sixty FPS? I actually don’t know the answer to that. - Good question. Sixty FPS, we never tried. I actually do not know. - Hey everyone, how’s it going? My name is Logan Kilpatrick. Welcome back to Release Notes. Today, I’m here with Raia, Diego, and Hang, and we’re talking about Genie 4.

00:31 Maybe we can just start with the news. The new model is here, coming through our products and API for the first time, which is exciting. Someone want to kick it off and talk about high level what’s going on with Genie 4? - We’ve been cooking up Genie 4 for quite a while. Genie 3 was released back in January, the beginning of the year. So, we’re ready for Genie 4. This has new capabilities that we will talk about, audio, Gemini in the loop to control the generation, the world model, and amazing photorealism

01:03 that comes from this model being based on Omni Flash. Yeah. - That’s awesome. For someone who’s just like, first time, they’ve never seen Genie, they don’t know what our world models are, maybe we can just do the TLDR of what it is, why we’re excited about this space, and then let’s see some cool demos that will hopefully bring whatever that description is to life. So. - We should start out by defining what world models are, because that term’s been used actually quite a lot. You can think about the language models, Gemini, as a thinking model.

01:37 It deals with knowledge and language, versus a world model is about understanding a grounded environment, a world, and how do interactions happen within that world. So it means that the world model needs to have an internal state. It needs to understand what is the state of the world and what happens when things interact in that world, when things collide or they move, or when we make an action,

02:09 how is that going to change the world? And so understanding the physics of that, even if it’s something that the world has never seen before, even if it’s fantastical, then, it’s still about understanding what are the likely ways that the physics of that environment are going to act when we take an action. - I love that. And maybe actually one more quick comment and then we’ll see some demos. Maybe a little bit of the historical context as well would be helpful,

02:39 which is this is not our— this is Genie 4. There’s been a bunch of other world model releases that we’ve done all the way back from, I think maybe just like a I don’t know, was Genie 1 just purely internal that we showed some demos of? And I don’t know if anybody actually used externally Genie 1, but then with Genie 2, Genie 3, I don’t know if that’s true, but maybe the history of the project as well and how it got started and where we are, I think would be helpful context. - Yeah, I’ve been watching the development of Genie for a long time. DeepMind started out thinking about game environments

03:13 because we wanted to understand how agents work in a game environment. And we knew that we could train better agents if we had richer simulations, richer environments? And so we started to think about, could we make a world model to just endlessly generate new simulations, new environments to train AGI in? And so Genie 1 was a high-risk research project that we didn’t honestly

03:43 think would work as well as it did, but it gave us that proof of life, proof of concept that just showed that we could generate a 2D environment. So, we trained it on 2D platformer games, showed that you could run right, run left, and jump, and be able to have it change how it looked. Genie 2 expanded that a year later into 3D environments, going from 2D to 3D, still very much looking like gaming environments rather than realistic worlds,

04:15 and it was not quite real-time, not quite interactive. So, then we really pushed, and Genie 3 was the first model released earlier this year. - In January. - In January, where we were able to show a new level of realism in the environments that we could create, that they were rich, they were 3D, and it was real-time and interactive. So, that’s when we really started to declare success in this world model. - I feel like Genie 3 was this inflection point in the research

04:46 where it became a little bit more useful for people to actually try out, but very clearly still in the frontier research, early. And we spoke about this also when we released Genie 3. And now I think with Genie 4, we’re taking a step forward in the utility range, but we’re still in the research range very much. - Yeah. - So this is still a model that will occasionally make mistakes, but it actually starts to showcase new capabilities that are actually incredibly useful. And so we’re pushing along that spectrum across different research dimensions. - And starting from the Genie 3, you begin to see the model is able to absorb

05:19 and reflect on a greater variety of the worlds and begin to be real-time responsive, and people can actually try to control characters in that world. And Genie 4, we’re just making further leaps on those fronts. - I love that. I’ve got a million questions, but let’s see some demos. Diego, I think you’re in the driver’s seat for this. - I figured one interesting way to approach the demos here is to maybe showcase Genie 3 as a baseline for one world. Okay, cool. And then we’ll showcase a difference for Genie 4, and then we’ll go

05:49 through a series of different demos, and then we can pause and talk as well. So, the first baseline that I wanted to show was this world. This is actually an image, so this is a starting frame. This is a Genie 3 generation. It’s just a short one just for the purpose of showcasing it. From the definition of these characters, Aside from the ones that are captured in the initial frame, they’re actually not that realistic. So, you can see more of a gamified look. And it’s also very importantly a silent world, which is something that we will get to in just a second.

06:20 So that’s just the starting point. We can use that as a reference of where we were in Genie 3. I think, like Raia was saying, the leap between the Genie 1, 2, 3 was pretty significant across different generations. But now we’re seeing even a further push in the realism side. So with that, let’s go to that example and see how that looks. So I’m going to start with the world muted for now, and then we’ll introduce audio. And there’s a lot happening in this interface. We’ll get into the details in just a second. And then here I’m going to prompt, the scene comes to life.

06:54 Let’s see if–okay. You can tell that the band is now playing. And if I unmute here… - (Vocalizing) - So, I think, a few things to highlight here. The audio piece is the obvious one, which is, I’ll lower the volume for now, but it’s audio is a little bit like in the V02 to V03– Let me pause this–

07:25 transition days where it’s the obvious thing that you didn’t know you were missing. And then once you go here, it’s very hard to go back because it is just this immersive layer and also a simulation layer that’s pretty incredible. The other thing that we can highlight as well here in this demo is the realism. So, I was highlighting how in the previous example, let me just mute this for now, the actual definition of the characters was much more gamified looking. - Yeah. - And here you can tell, actually, if I get the camera really close,

07:56 in this case I’m just controlling the camera, the details are quite impressive. And one thing behind the scenes, which we can talk about, is a lot of this is actually built on the foundation of Omni as well, which is providing a lot of that realism behind the scenes. And also, there’s a few pieces here within realism, right? There’s obviously the character definition, but then there’s just the hand motion. There’s so many different layers to it that are very subtle, and it’s not fully perfect, but it is definitely a big leap forward. - Yeah, no, this is awesome. - Audio modality is a net new modality

08:28 that’s being added in this Genie 4 model. Previously in three, you can only navigate in a scene and only see the visual output. So here the Genie 4 model predicts both the video and audio together from the same model. - That’s awesome. And without giving the secret sauce for this, is it like it’s just like great new data that has audio, video together or something like that that’s enabling this? Or is it like we’re— is it like the— rebasing onto Omni and then building off of that? I’m curious, as much as we want to say anything

09:01 technically about what’s happening or what’s different for Genie 4, what’s actually making this possible? - I think building on top of the Gemini Omni as the newest foundation model on video certainly is providing a big lift. On the other hand, in terms of data, real-world data is just multimodality in nature. When you have the video data, it’s always paired with audio, it’s always in sync. And that is the world we want to model. You want to model only world from one single perspective, but then audio modality, video modality are essentially just the same components of the same world.

09:37 - Yeah, no, I love that. And Diego, you actually you mentioned there’s like a bunch of stuff going on in this demo. I don’t know if any of the other stuff we’re looking at visually on the right side or anywhere else is worth it. The pausing and resuming is, I don’t know if that’s another new thing or that’s just for this research demo. - Maybe we can talk about a few things here because there’s a lot that, and we’ll showcase this across a few demos. The first is the pause interaction. This is mostly just for research purposes. I think we found that, because you’re now prompting in real time,

10:07 which is a new unlock that we can talk about as well, you’re interleaving text captions in real time and you’re also just steering the model in certain ways. But the problem is the stream is still running. And so it’s very hard to type as the stream is running. And you also want to make sure that you’re like hitting the frame at the right time. So pausing it just helps us in the development process. Whether or not it’s something that we expose to users as a capability or as an experience, I think it’s to be seen. We’ll probably test this and see what feedback we get. There’s been a lot of different points of feedback on

10:38 the right way to steer the model with text. Maybe it’s you queue up things ahead of time, maybe it’s you interleave these text captions during the way, or maybe there’s something like procedural actions or other things that we can think about. We’ll talk about the Gemini story as well, which is what’s happening on the right in the next example. - Cool. No, this is awesome. Anything else on this demo we want to look at? Otherwise, I’m ready for more examples. - Okay, let’s do another example. So one thing we leaned on in Genie 3 a lot was this idea of a third-person character.

11:08 You’ll remember when we did that demo of the little dragon-like toy that we would walk around, and then we would have different sort of examples where you would take a picture of something and animate it into real life. Yeah. And that was a very fun demo. I think for this iteration, we’re actually focused on two interaction paradigms much more than the third-person character, which is one is egocentric views. So first-person views and actually being able to navigate with keyboard and that, or with KBM and being able to actually steer the world in real time from that perspective.

11:39 And then the second piece is actually camera controls, which is what we showed in the previous demo. So let me now show an egocentric demo, which is just a first-person view. And see how this goes. - This is an interesting example. - Very interesting. This is a good highlight that Genie 4 really focused on the photorealism a lot compared to Genie 3. And here you see egocentric hands making coffee. - Beautiful coffee. Yeah, yeah. Shockingly consistent as we walk, which is cool. - We were talking about audio before.

12:10 As you may imagine, there’s a big part of audio. We’re showing the music example. There’s obviously environment audio. But a big part of audio and simulation is also dialogue with people in the world. This is one area that I think we’re starting to do some pretty active research on. It’s a very complex area, and I think it still has a lot of work to be done to be fully perfect and polished, but it does expose entirely new capabilities. So, let’s see how this goes, and then we can try it out. So, I’m going to unmute this. Let me drop the coffee.

12:45 - This guy’s not nearly weirded out enough about us. - Oh, sorry about that. - No worries at all. Accidents happen. - Okay. So, you saw in that case, let me just mute this. It’s an interesting example, right? Because there was a few things there, and actually maybe let’s do this. I think this is a good example. Maybe, Hang, you can jump on. - It’s a good time to also take a pause to look at how the audio modality previously we mentioned in the previous example is only about ambient music and having audio to pair with video.

13:19 In this case, we’re also emphasizing a lot on dialogue. - Yeah. - You can have dialogue with a character in the scene, in the world. Previously in Genie 3, the control dimension you mostly have is around controlling the main character, either in the first-person view or first-person view, to navigate and walking around the scene. But here, a lot of emphasis is being put on how can we begin to interact with the scene? How can we, right, in the examples of picking up a coffee cup, serve a coffee cup, and there you see people in the scene responding to the actions of the character in real time

13:55 and simultaneously also having dialogue between the characters. - It’s crazy. - Yeah, I think, the part that is very mind-blowing is that everything is done in real time. And so there’s so much that could go wrong and sometimes goes wrong just in terms of the artifacts in the dialogue and making sure that everything just feels like a cohesive dialogue and conversation, but then also that the dialogue is in, I would say, coherent or around the actual context of the world, if that makes sense. And it’s a pretty fascinating research and experience problem.

14:27 - On the two lines we mentioned on the education and the robotics, so here the dialogue piece is driven by Gemini. It’s not only driven by the Genie 4 world model alone, but also driven by the Gemini. There’s a Gemini model here in the background thinking about how to respond to user’s action. And later you can imagine this is going to be able to be connected to more instructional use cases where the Gemini essentially has the world knowledge, has a lot of reasoning capabilities. And then

14:57 that’s one of the enablers how world model can serve for instructional and educational use cases. And the second piece on the robotics here, many of the emphasis we see here is on the first-person view where you see from the first-person angle how an action is accomplished, how different tasks are done. So, hopefully those domain would transfer to robotics domain later, and then we can use the same technology to maybe make robotics better. - Yeah, very interesting. And so, just something that’s interesting

15:27 maybe for folks who are not familiar with world models, but are familiar with a lot of the stuff that’s happening right now with text language models, it’s almost like this is a world model harness around, am I framing this right, around Genie 4 that’s allowing us to do this sort of intersperse the audio and it’s Gemini somehow in that mix as well. Is it a bad or— reasonable mental model of this? - One piece that is net new here is that to what Hang was mentioning,

16:00 is that you have Gemini working hand in hand with the diffusion model. And so all of a sudden you have almost a different layer of intelligence that’s also provided by Gemini at the core, looking at the world and basically saying, okay, what happened in the previous frame? What did the user intend to do? And then what’s supposed to happen next to actually be able to meet that? - And this is just raw Gemini, just video image understanding as you would get out of a normal— - Yeah. So in this case, we’re using Vanilla 3.8 Flash, mostly because we want to make sure that things run in real time as well.

16:32 But we have done experiments also with Argon, for example, and seeing how far we can push, for example, in robotics examples, which we can talk about as well. Nice. - Here, Genie 4 is certainly a multimodal world model on its own, but it’s also enriched, and many of the intelligences are provided by the underlying Gemini layer. - Yeah. Do we have any Live plus Genie 4 examples? Is that what it’s using, the new live model and you can stream a video and then have the video stream be used?

17:02 Or is it actually just baseline 3.8 Flash taking in images? - This is just baseline 3.8 Flash, but you’re talking about the audio model that could be plugged in A2A and— - A2A, but I think for A2A you can also share a video. - Yeah. - A video stream. - It’s an interesting experiment. In this case, we’re just doing 3.8 Flash as a tech station. - We’ll do it in AI Studio. Perfect. - We’ll do it in AI Studio. Yeah. - I love it. It’s already all hooked up. - We just took a picture here of the set. We wanted to turn the set into maybe something a little bit more creative

17:32 and fantastical to showcase that a big part of this is world simulation, but you can also be able to simulate, like we were talking about earlier, maybe things that are harder to recreate in reality. So, let’s see. So here I am on set. People are moving around. Let’s see. - In case there has ever been any illusion, we have nice cameras. So, very nice cameras. We don’t have that lounge though. - Yeah, of course it’s— oh, there it is. Okay, so we got a dragon there. Let’s see what the audio says.

18:03 - Looks like framing and lighting are locked on this end. - Okay, so we got some audio there. And then let’s see, the dragon blows fire. - I really like the ability to pause stuff. - Yeah. - I feel like for interactive stuff, I feel like it’s very helpful to be honest. So you’ve got your “sample size one” to add this to Project Genie in the API. - Yeah. So, this is a fun little different type of example. I think one, so for example, in the past we spoke about trying to understand how people would use this for filmmaking, for example,

18:35 or for just creation and different creativity purposes. The thing that has become clear is that there is a phase in pre-vis or in the ideation phase or in the storyboard phase that benefits a lot from this idea of rough input or rough outputs that you can get rapidly and iterate on. So this idea for let me move from this very coarse idea to this very rich interactive medium that I can see and iterate on and say put in a dragon, throw fire, change the camera angle. And so I think even just that as an interim output is really valuable.

19:09 And to that point, you could pause the scene, you could move around, and so on. That could be really interesting. - Yeah, it’s fascinating. We had talked before about robotics and education as two places that there’s excitement from an application perspective. Why, if folks are not deep in robotics, why does this matter for robotics, Raia? I’m curious, what’s the application use case? - Yeah, it’s not immediately clear, but think about how humans move through the world. We really we have a world model in our heads essentially that just predicts a little bit forward, and that’s what allows us

19:42 to move fluidly, to be predicting when we’re going to make contact with things and what’s going to happen for every move that we make. Having that model that plays things forward in our head, we’re not generating pixels, but we’re still, we are constantly running that forward. And we think that if robots have a world model as well, that they’re going to be able to make use of that to do all of those short-term and some slightly longer-term predictions, and just be able to do a much better job of being dexterous, being agile,

20:15 being a little bit more at that level of biological movement that we would like to see in a robot. - Yeah. And that’s for like, the idea is it’s like, actually onboard real, as I see a humanoid robot walking around the office and it is actually doing planning… …on, maybe not on device, but in a real live deployment? Or is it more like you want to get good training data for those types of or could be both or many different things?

20:48 - I mean, we started out thinking just about training data and oh, if we had this world model and we can prompt it to look like our robotic setups then we can use it to generate a lot of training data. - Yeah. - I think that we can go far beyond that if we’re actually directly using the world model, using Genie to produce the actions for the robot. Then, you’re actually producing the actions, but the model knows what’s going to happen next. It can make those predictions accurately and knows that physics

21:20 of interactions and motions and what happens picking up a heavy cup versus a light cup and what happens when you drop things, what happens when you drop the cappuccino. It really could be a significant unlock in terms of generalizable robotic movement and skills. - Interesting. - Research is always research. You never really know until you’ve really done it. So, it’s very promising, but I’d say there’s still work to go. - I love that. - Maybe a good segue to maybe show– this is a recorded demo that our robotics partners did.

21:53 - Nice. - Which could be an interesting proof point of this is actually building— let me just pull up an example here of Scrabble with captions. And then I’ll highlight before I play what’s happening behind the scenes. So in this case, I was talking about how we were using 3.8 Flash. In this case, they actually use Argon as the text station. So, this is not necessarily running in real time, but the actual orchestration of Argon and Genie 4 is producing interleaved actions for the robot to perform things to complete a certain task. So, what you can see at the top are actually captions

22:26 that are drafted by Argon directly based on the user intent of like you should fulfill X task. And in this case, I think it’s arrange the Scrabble letters to actually spell. - And we’re seeing a simulation through Genie? - Exactly. All of this is rendered through Genie. - Okay. - So, maybe one analogy here is Argon is providing the instructions and then Genie is generating the actions and the pixels. - Interesting. For that example, again, it’s this is more of like a, at least in that one

22:58 particularly, it’s like a more training data type example where like we want to be able to visualize robots doing specific tasks and we can just do it better, faster, cheaper than actually–if folks haven’t seen, I just for the first time went over to the robotics lab and like it is, there’s like, it takes time to get robots set up to actually do a task. And then they’re, it’s almost like a movie scene. They’re like, okay. And stop this thing and now we have to do this. And so the idea that you could just scale up simulation somewhere behind

23:28 the scenes and be able to capture lots of data, I think seems super interesting. - It’s this path through which you can start to really push on this idea of spatial thinking and spatial reasoning in infinite permutations. You could just let this run with Argon and Genie infinitely effectively and then just have all these variants of different checkerboards and see how far you can push it. - Yeah, We’ve seen robotics, we’ve seen audio, we’ve seen new steerability controls. If we were to summarize the places where Genie 4 really shines versus previous models,

24:01 it’s those three things? - I think realism is a big one. We’re getting some early feedback on this as well. One thing that jumped out with Genie 3 is that it was a pretty massive leap from Genie 2. But one point of feedback that people would bring up is that the people generated in the scene were actually not— they were either static or they maybe didn’t actually behave as intended. And you could see from the jazz example that it was much more realistic. There’s still, I think, some ways to go there. And I think that’s the exciting

24:32 part of research is that we can continue to push in that direction. But it’s a pretty meaningful step forward, again, building on that Omni Foundation. The idea of freeform controls is super interesting. Because one piece that jumped out from Genie 3 as well is that it was this pathway through which people could imagine anything and explore it. But there was also only navigation, at least in the way that we released it. And now that we’re actually pushing on the quality of these ideas of freeform controls, you can actually then have pretty much anything that you can do in the world. And so it’s the shift from exploration to simulation almost.

25:06 The last piece, and probably the most obvious one, is audio. Audio just going back to the earlier point is the obvious thing that we should get to at some point that we didn’t know that we, the impact of not having it. And I think it’s a pretty interesting unlock from a simulation perspective, from a dialogue perspective. And I think we’ll see a lot of progress there over time. - I feel like for Veo, that was the thing that moved the needle. I feel like Veo 2, no audio, Veo 3, I think with audio was just like, I think it made all the difference. Yeah, I’m curious where the things are, obviously, continuing to scale the model, but I’m curious where are the bottlenecks?

25:38 I feel like it’s pretty obvious and my mind is mostly in mainline Gemini world, but it’s scale compute, scale data, throw some new algorithmic innovation in there. And I think generally the recipe works. We keep getting better models. Argon hopefully is a great example of that. Future versions after that are a great example. Is it roughly the same stuff on the world model side where it’s just we get better base models? Actually, the caption stuff that we are seeing is maybe a great example of this

26:08 cross-capability transfer, improving one thing? We have state-of-the-art now image understanding models. I’m sure it’s helpful in generating these captions and scenes and stuff like that. So is it? - I think that having the model be able to generate a given environment more consistently for as long as possible. Originally, there were problems with things would just blur out after running for a little while. Now we get these very long, consistent, crisp, realistic scenes over time.

26:40 But there’s still a big frontier to push. And although compute and scale is nice, we want this to be fast. - Yeah. - We want it to be really low latency so that it’s really reactive so that you can really engage with these worlds. So that’s a challenge. How do you keep on bringing in new capabilities, but actually train a model that’s small enough and efficient? And then, of course, there’s the future. I think that we all know how good coding has gotten. The best coding models can just generate you

27:12 a 3D scene or a game that you can then interact with. So I guess the question is, do we still need Genie if we can just generate a world with code and have it be interactive? And I think that I would say yes, we still want to have this, but there’s really interesting ways in which— generative coding is going to come together with generative pixels to be able to really bring something to life and have it be potentially more efficient to run in terms of the compute required

27:44 and be very long-lasting. - Yeah. - Could even support multiple players at the same time, things like this. So, I think there’s a lot more to come. - Yeah. No, this is super exciting. Well, actually, a couple of really quick follow-ups on this. For example, this speed bit, I think, is the thing that I think people maybe is look at AI videos and it’s like what’s the discerning characteristic of an AI video? When I use world models, sometimes it’s like the discerning characteristic is almost like the FPS.

28:14 And I’m curious, is this an is this a production limitation where we could actually run the models if you just had infinite, or maybe not infinite TPUs, but if you put all your TPUs for one person, could you just run Genie 4 at sixty FPS and it’s really incredibly snappy, feels like real time, and we just trade that off with scale and availability for more users? Or is it like actually there’s still technical breakthroughs that we need to make from an infrastructure or from a research perspective to make it so that you can get this very fast sixty FPS,

28:49 whatever the video games run at, experience from a world model? - Yeah, there are a lot of trade-offs, we have been it’s certainly not linear. And I think one of the moments while we were developing this model was the point where the model was able to first time run live, was able to first time run real time. There we had to do a lot of exploration on the space on how do we trade off different configurations, how do we optimize different components,

29:19 and eventually get the model to run real time still in a cost-effective way. So it’s not always clear and trivial to scale, to having scale as an answer. - Yeah. - But Hang, If we wanted to put all the TPUs for one, just for Logan, - Just for me. - could we run it at sixty FPS? I actually don’t know the answer to that. - Good question. Sixty FPS, we never tried. I actually do not know. - We’re happy if we get to, what, twenty-five to thirty, twenty-four.

29:51 - Is it like same thing with pixels for image quality where at a certain FPS it’s like I’m just I just, sixty just is a random number that came to my head because I think things run at sixty FPS or whatever, or maybe videos are sixty FPS, but is there a specific value at which, are we just choosing twenty-four because it’s where we are at today, or that’s mostly noticeable by humans and after that they can’t discern the difference or something? - It’s more the latency, I think, that you see in terms of when you take an action, how long that takes, right?

30:23 So that’s a different, there’s a couple of different dimensions to looking at that. That’s–how snappy it is. One of them is the FPS, one of them is the latency. - So playback and the interactivity are also like two slightly different components where if you want to get the fluent playback, you could generate more, let’s say frames, and throw more compute at it to generate frames faster. And then you get a higher frame rate when you play back the video. So that would of course give you higher FPS playback.

30:54 To the point of input latency, the reason we also choose twenty-four is that’s the latency where we can interrupt the generation. That, at certain breaking point, we have to take a user interaction here, in this case from a laptop, into the generation and close that loop. So, there’s a connection between what’s the frequency that you allow the external input signals to flow in and also generate sufficient frames in the next immediate window to stream back to user without making the whole experience laggy or choppy.

31:27 - Got it. Well, that makes sense. And Raia, one more question on this interplay between code and world models is, and maybe this is, I’m sure my intuition is wrong, and so this won’t reveal any technical details, but is the intuition that you walk around and one of the constraints I think for Genie 3, as I remember it, is there’s a context window constraint, which is after thirty seconds or sixty seconds, whatever it is, we would run out of tokens in context and so

31:58 you would lose whatever the parts of the world that you had explored, which previously you could look back at and see, you know, a house was there or something like that. And so is the idea that you could like look away, use code to generate a house and then replace it almost in simulation or something like that? Or is that completely a wrong random intuition? Or how, and maybe we can’t talk about it, but like how would those two things actually come together? - If you’re bringing code together with a world model, then I think it does allow

32:28 you to have infinite consistency in a way, because then you’re able to remember the state of the world, which isn’t always what you want though, interestingly. The power of a world model is that it’s so expressive. - Yeah. - Right? There’s a dragon that comes through and flies around, or everything turns into watercolor and you’re all of a sudden swimming. And it’s that expressiveness of a concept brought to life in an amazing way.

33:00 And so we talk about this with the team because using code, using 3D, there’s a lot of ways to potentially bring in structure and consistency over time. - Yeah. - But we don’t want to lose as well the freedom of just having a purely generated world that you can interact with. - Yeah. - But yes, the idea is that you could use code or some way to have that as your backend, your underlying structure, and then that can stay fixed. That state can be easily maintained so that you can go wander the hillsides,

33:32 come back, and find the house is the same as you left it. - Yeah. - I mean, there’s an interesting product question behind that on the point of more exploratory/open-ended world models, which is: this is something we’re exploring with labs. What is the future of these interactive experiences that are almost open-ended and adaptive, and you don’t know what you’re going to explore, what you’re going to encounter? I think there’s a question behind that, which is: that open-endedness, Is it represented by code? Is it represented by pixels or something else? But besides that, I think there’s just this open thesis of

34:04 if you create something that is genuinely open-ended that no one knows exactly where the boundaries are what is the experience or the end experience that people create with that? And I think that’s just a pretty fascinating product direction to explore, maybe with the API as well. - I’m super excited for Genie 4. Where will it actually be available? - So it will be available still in Project Genie. So we’ll continue to build out that surface, and there’s some interesting and exciting updates there. And then it will also be available in the API. I think that’s part of the feedback that we got from the previous release that we want to expose different ways of people

34:35 building with the model in novel ways that maybe we haven’t thought about. So, yeah, we’re excited about that. - Nice. I’m excited too. One of the interesting things that the science team has been pushing for a long time is SynthID, which allows us to watermark all the stuff that— all the different models that we’re creating across Gemini, the GenMedia models, et cetera. I assume this is supported in Genie as well. And so I’m curious, yeah, any interesting difficulties of actually doing that in a real-time model? - Outputs will have SynthID.

35:06 And then also just more broadly, maybe taking a step back, it’s worth maybe touching briefly on the complexity of safety running in real time. Because I think there’s a question on what– an uncharted question, I would say, of what happens when photorealism starts to meet real time. And then you can simulate anything in the world. And obviously we want that to be used for good, but there’s also a lot of important things that we need to think about along the way. And so, we have, with the team, with the safety teams, we’ve started to build almost pioneering systems of how these

35:36 should be filtered and, or how edge cases should be filtered in real time or how we should set the right guardrails, but also at the same time not compromising on the real-time aspect. So it’s this interesting balance of how do you ensure that you keep the quality with the bars around real-time and latency and all these things, but also make sure that you’re doing it responsibly? - Yeah. I’m always having to remind myself on this safety point for world models that actually a lot of these, Raia, you mentioned this educational use case, but there’s also this sort of,

36:06 which is related, which is this almost role-play scenario type of situation. And these are explicitly scenarios you actually don’t want to put humans in. And this is like, how do you, as a first responder, what happens in these disaster scenarios or something like that? I can imagine people using this for, but you probably have to generate stuff that is, relevant and realistic of a disaster scenario, which is probably things that would normally you’d block in a model output. And so there is this really interesting, I can imagine the tension of you want to enable

36:39 these very positive use cases for people to keep humans out of harm’s way. And at the same time, yeah, need to make a model that’s safe, that can’t do bad things. And so I’m curious, yeah, I don’t know if there’s anything more on this or if that’s just - I think it’s the similar approach in general as we take for Omni and how a model is used differently when you just want to have it being used by the general public for general things versus by a creator or somebody with a specific purpose in mind.

37:11 It just, it’s a different context and there’s lots of valid reasons for a filmmaker to want to prompt a model in a specific way or an educator to be able to create a realistic scene of a battlefield, you know? Yeah. Something like that. So, yeah, it’s funny; these questions felt like real stumbling blocks a couple of years ago and now it feels like oh, we’ve got some answers here. We can address these by simply looking at who is using it and how are they using it?

37:43 But certainly the first time that we put something out, we really want to make sure that we are able to have the guardrails and keep the content safe for people who are going to be using this. - Yeah. - This is also one of the reasons why we start with Trusted Testers, by the way. That’s part of the plan behind the gradual rollout is to get early feedback on the model, but also to make sure that we’re doing that in the right way and figuring out what are the edge cases that we can actually start to get feedback on. - I’m curious about of course, you can’t usually be steering the model in real time, but we can actually do that.

38:15 So is that part of the safety approach to steer it rather than simply terminating an episode? - Yeah. - It’s a good question. I think that a part of, or benefit of having Gemini in the loop and maybe Hang, you can expand on this as well, is you are able to actually almost steer the model with text. Those captions could be steered in certain ways, but I think it’s also very scenario dependent. I don’t know if we’ve seen specific examples where maybe Gemini is the path to steer

38:45 us towards a specific or away from a specific scenario. - I think with more intelligence layers being added here with Gemini being one, we have more just tools to deal with safety. One way is of course we can try to understand what’s being generated and see whether it’s safe. Cut it off when the time is right. Or now, the Gemini intelligence layer essentially also plays the role of monitoring the scene, observing what’s happening in the scene, and driving the underlying logic of the scene.

39:17 So, certainly that’s a possibility where the scene can be steered away just from the Gemini level. - Yeah, very interesting. And Raia, I think actually this different user persona thread is very interesting. I feel like the breadth of Google’s products sometimes I think works against us in certain scenarios, but I think this is actually a great example where you want to be able to meet these really diverse set of people with really capable models in a different way. And so I’m curious, is that front and center as we think?

39:47 And obviously for this initial, I feel like the deployment services for Genie has continued to expand over time. And so I’m curious as we think about next iterations, increasing the distribution and the deployment services. Maybe into Flow or something for movie creators or video creators in the future and other things like that? - Yeah, it’s never a single model and it’s never a single deployment. To your point, Google is big. We have a lot of customers that come to us from very different types of spaces.

40:18 And so it is often necessary to really provide something that is more, you know, that is customized for that need. A director using Genie might have a very different need in terms of it’s all about the camera control that you want to have be very accurate, have very good control. But for another user, it’s just going to be something else. So we do as much as we can with a single model, and then we figure out what more is needed. - I love that. - But I can’t wait to see what developers do with the API.

40:51 - I think it’s going to be fun. There’s all these examples in my head of like, we do something cool with our models internally, we’re super jazzed. And then we’re like, we see external people doing stuff and we’re even more jazzed. And so it really is like a, it’s a “1+1 = 3” scenario, I feel like in almost all cases, which is exciting. Yeah. So it’ll be cool to see what people build. One of the fun things is always hearing about the cool, interesting prompts that folks, as you all probably use the models more than anybody.

41:21 And so do you have any good go-to prompts when you’re testing a new rev of Genie? - I think one of the favorite ones that we were testing in the earlier iterations of the model was a photorealistic restaurant kitchen, and maybe with everyone busy, cooking. Let’s try this out and see. - Why do you like this? What’s the spark? - I think it points to a few things. The first is I feel like it really highlights the dynamics of a busy environment, of a lot of things happening in sequence.

41:54 So for example, here you can tell that there’s a lot of people there’s a chef over there, there’s someone chopping onions over here, and there’s a lot of interplay dynamics between them. But then my other favorite part of this is you can also do fun things. So for example, let’s say I grab the cucumbers. - So if you got to get an audio thing for this. - Yes. Here, I’ll just turn the audio on. - And can you just talk to it? - Yeah. Let’s see. Let’s go.

42:25 Let’s go to the chef. - Grab cucumbers there. Hey, Chef, what are you cooking? - Maybe let’s go… let’s go closer to the chef. - He’s cooking. He’s like, I’m not talking to you right now. - Fresh salmon with herbs and garlic for the evening service. - Oh, there you go. Okay. - Nice. Thank you, Chef.

42:57 - Yes, Chef. - Just pan-searing some fresh salmon with herbs and garlic for the evening service. - Could we ask these– is it possible to ask them to increase the sense of— this is the most zen kitchen that I’ve been in, in a long time. We got to get these guys moving and chopping. - Let’s see the sprinkler system. - Just pan searing some fresh salmon, herbs, and garlic for the evening service.

43:28 - Are these guys twins? - Yeah, that’s one of your guys. - If you turn around, there will be three more. - Exactly. And I turn this on. - Oh, the sprinkler system is on. - The sprinkler system is on, yeah. - Excellent. - Okay. Wow. Interesting sprinkler system. - I feel like this is like a little bit of one of those chef watering stations, if you know what I mean. To spray down into pots. - Yeah. - That’s awesome. - We’ve been trying to set fire to this kitchen a lot of times. - Yeah, we had this fun, to the point on the kitchen being mellow,

44:02 we had this fun example where we were really trying to push on the Gemini intelligence point on just something that seems very simple, but it’s actually not super trivial, which is you want a sequence of events, very specific events to react. To your point. Let’s say if a fire starts, the sprinklers start, and then everyone starts doing a sequence of actions to mitigate that. So you want that to be happening in the world. - See, I usually go non-photorealistic. I usually go like, I like bumper cars where every bumper car is a different animal or something like that,

44:33 or where the bumper cars are vegetables or something, just to try to push it to do something arbitrary and where you get some of the dynamics. And it always does a racing game well. So bumper cars is a variation there. - Yeah. - But I haven’t done as much with the photorealistic. That’s amazing, the detail. - Yeah. - That is a dirty— - that is a dirty— that is also a lot of burners. Never been in a professional restaurant. So I don’t know. I’ve never been in a restaurant behind before. So I’ll add this to my bucket list and find out

45:03 whether or not there’s really this many burners inside of it. - You can go in Genie 4. But yeah. Seemed too real. - No, this is awesome. Actually, this is just one other follow-up is, is it easier or harder? There’s so much talk about the eval space for mainline Gemini models or just mainline LLMs right now, just exponential difficulty improvement or increase rather. And to build a new benchmark, it used to take fifty hours and now it’s 500 or 5,000 or something like that.

45:35 I’m curious. Is it easy to hill climb or know whether we’re hill climbing model quality for world models, or is it so subtle in the like, the burners are weirdly configured and that’s not the way that kitchens really look or something like that makes it difficult? - Certainly, the evals are never great, never ready, and we keep evolving the evals as we evolve the build of the model. Always we discover new problems, new aspects we want the model to be better.

46:09 So as all research goes, it’s really a lot of the challenges, also surprises would arise. So EVAL is never ready, I guess. Yeah, that’s also one of the reasons why we are about to release this. - Yeah. - One of the other projects that we have in frontier AI within Google DeepMind is agents that can play games. So this was a SIMA release about the same time. And so we have agents that can play

46:42 in these worlds and explore them automatically. So we actually want those agents to be able to eval Genie worlds for us. And that would be helpful. - That’s super interesting. Raia, Diego, Hang, thank you so much for sitting down. This was a ton of fun to see all the live demos. I’m excited for more folks to try the model. Excited to see what people, I know it’s too cliché, everyone says excited to see what you build, but I am excited to see what people build with Genie 4 and the API. It’s going to be awesome. So thank you all three and the rest of the teams for making this possible.

47:15 And thanks everyone for tuning into this episode of Release Notes. We’ll see you in the next one.