AI Breakthroughs Everyone Missed

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: AI Breakthroughs Everyone Missed · Channel: Matt Wolfe · 24:57 · Watch on YouTube

Topics: AI content generation, AI models, Multimodal models

Summary

Matt Wolfe highlights lesser-covered AI and technology projects, including expressive text-to-speech, audio-driven video generation, and several approaches to creating interactive world models. He compares systems that use memory or game engines to maintain consistency, while noting limitations such as low resolution, hardware requirements, and limited availability. The video also covers tools for reconstructing and animating 3D scenes, plus a non-AI browser tool for generating fictional regions. Wolfe says he wants to make more videos exploring under-the-radar research and interesting technology.

Key points

Fish Audio’s Drama 3 lets users direct speech performances with written instructions, sound effects, and multiple speakers; it is open-weight but requires substantial hardware to run.
ByteDance’s Omni Human 2.0 creates videos of a person or character speaking, singing, or acting from an image and audio, with clips of up to five minutes.
Tencent’s World Crafter uses camera-position-based memory to improve consistency when revisiting parts of a generated world.
ByteDance showed an unreleased world model that can generate new camera paths through scenes and respond to prompts, but it has no public release date.
Matrix Game 4.0 pairs a conventional game engine with an AI video model that reskins its rendered scenes, keeping gameplay state in the engine.
Nvidia’s Stream Render H3 uses color-coded maps as input for a video model to render racing scenes, but its demonstrations require data-center GPUs.
Odyssey connected a simulated fruit fly brain to a camera moving through its Odyssey 3 world model; the presenter notes there is no paper or code explaining the setup.
Preferred Networks’ Telescope generates detailed 3D meshes from photos, while Gaussi Animate adds motion to soft materials around character skeletons.
World Gacha is a non-AI browser tool that procedurally generates fictional regions and can export scenes in several formats.

Chapters

Start	Chapter	Summary
00:00	Under-the-radar AI news and Drama 3	Wolfe introduces the video’s focus on less-covered projects and demonstrates Fish Audio’s expressive text-to-speech model.
04:09	Audio-driven video and world models	He covers ByteDance’s Omni Human 2.0 and Tencent’s World Crafter, discussing their capabilities and limitations.
09:51	AI agents and game-world consistency	After a sponsored segment about OpenAI Dots, Wolfe explains how Matrix Game uses a game engine and AI video model together.
14:02	Nvidia’s real-time rendering approach	He describes Stream Render H3’s use of color-coded maps and contrasts its hardware demands with Matrix Game.
16:38	A simulated fly in a world model	Wolfe discusses Odyssey’s fruit-fly-brain demonstration and shows how users can interact with Odyssey 3.
18:11	3D reconstruction and character animation	He introduces Telescope for photo-based 3D meshes and Gaussi Animate for animating soft materials around character skeletons.
21:19	World Gacha and closing thoughts	Wolfe demonstrates a non-AI procedural world-generation tool and explains why he plans to feature more overlooked projects.

Quotes

some of the cooler things that I’ve come across recently that haven’t made the bigger end of week news videos. — 00:31

it lets you write the direction the way you’d like explain it to an actor. — 01:01

An ordinary game engine runs the game. — 11:25

Every pixel is a flat color that means road or grass or barrier or rival car. — 14:33

the theme of my channel is just cool stuff that I come across online — 23:24

Concepts and entities

Fish Audio, Drama 3, Artificial Analysis, Hugging Face, ByteDance, OmniHuman 2.0, world models, Tencent, World Crafter, Matrix-Game 4.0, game engine, NVIDIA, Stream Render H3, Claude Code, Odyssey 3, Eon Systems, Telescope, Preferred Networks, Gaussian splatting, Gaussi Animate, World Gacha, Matt Wolfe

Full transcript (original language)

00:00 So every week I like to make an end of week AI news breakdown video where I share everything that happened in the world of AI for the week. Usually those videos are very focused on some of the bigger labs. They’re the ones that get most of the news. OpenAI, Anthropic, Microsoft, Google, Nvidia, companies like that. However, there’s often some really cool news that flies under the radar. There’s a lot of research going on and smaller tools that not as many people, including myself, are talking about right now. So in this video, I

00:31 want to share some of the cooler things that I’ve come across recently that haven’t made the bigger end of week news videos. Let me know in the comments if you like this style of video and I’ll probably make them more often and share some stuff that’s not from the big labs. But let’s start here with Fish Audio. Fish Audio released a model called Drama 3 and it’s a text-to-speech model where you direct the performance in plain English instead of picking from like a menu of emotional presets. So like most voice tools, they’ll give you a

01:01 drop-down for happy, sad, angry, things like that. Drama 3, however, it lets you write the direction the way you’d like explain it to an actor. You put it in square brackets right in the script. So for example, here’s some spoken text with no direction. >> I’m fine, really. Go on without me. >> And now here’s one where there’s some direction written in, barely holding it together, forcing a smile. >> I’m fine, really. Go on without me. >> And you can even pick specific words. Like right here, they specifically tell

01:33 it to whisper go on without me. >> I’m fine, really. Go on without me. >> So you can like sculpt an effect to specific words. You can drop in sound effects, things like a small laugh, shaky breath, and you can put them exactly where you want them. >> I’m fine, really. Go on without me. >> You can even have multiple speakers with multiple voices. >> You were there that night. >> I was home, asleep. >> Then why is your coat still wet?

02:03 >> It rained. >> They even licensed Uncle Roger’s voice for the demo page, which is kind of hilarious. >> Hiya, nephew. Why you put ketchup in egg fried rice? Fried rice need soy sauce, need MSG, need wok hei, not tomato sauce. Uncle Roger cannot watch this. >> If you’ve ever watched like any of the Uncle Roger videos, it’s actually pretty spot on. If we take a quick peek at artificial analysis here, we can actually see that Fish Audio Drama 3 here scores the highest on

02:35 pronunciation. So, it’s really good at pronouncing words, even scoring higher than 11 Labs and SpaceX AI’s text-to-speech model, which is what that I actually demoed in last week’s news video. However, when looking at the arena leaderboard preference, which you know, sort of did a blind taste test where people listen to a bunch of different models and picked their favorites without knowing what model they were picking. Well, it didn’t quite do as well. So, you know, it reads hard words better than almost anything else, but a lot of people still prefer other

03:07 voices. It also appears to be open weight available on Hugging Face. It is a 36 billion parameter model, so you’re going to need some pretty decent hardware in order to run it. People have reported that you need roughly a 46 GB VRAM card, or you can get away with 32 GB with a community quantized 4-bit version, but neither of those are going to run on like most laptops. A couple other quick notes that I found about this one. So, the license is free for commercial use, but only if you’re

03:38 making under half a million a year, which is probably a non-issue for most people. The API is about $10 per million characters right now, but that is a launch discount that jumps to $50 after November 12th. And Fish audio takes do vary, so plan on generating a few and keeping the best one. And if you are going to use the API, double check that you set the model to drama three because it often will quietly fall back to older models and you’ll still get billed even if you’re using the older models. So,

04:09 make sure you check that. But if you head on over to fish.audio/dramat3, click try drama three for free, you can play around with it right now. And if you make audiobooks or game dialogue or anything with characters, probably worth trying out. All right, while we’re talking about voices, once you have that voice performance, you might want like a face to go with it. Well, ByteDance recently released Omni Human 2.0. Now, this model will take a single image and an audio track and then it’ll generate a video of the person talking or singing

04:41 or acting it out. Now, this is the 2.0 version and the biggest difference is that the previous version topped out at just 60 seconds. This one will allow you to go up to five minutes in one continuous shot. So, you don’t have to like stitch a bunch of short clips together and hope that you know, you don’t see the seams or hear the seams. So, here’s some examples. >> The kingdoms called it a legend. Tonight, I make it real.

05:13 >> And it even works with singing. >> Now, double [music] time, follow the flow. Every expression ready to go. Voice in the feeling, feeling emotion. >> Wait, did you catch that? >> You can also add in text prompts to control camera moves or add things that weren’t in the original photo. Now, it generates in 720p resolution and it doesn’t just have to be humans, it can work with like cartoon characters and animals and things like that as well. Now, if I’m being totally honest, I still think these lip syncing models have a long way to go. When I watch a

05:43 video with like an AI lip sync attached to it, it still just like isn’t working for me. We’re getting closer and closer. I mean, like we’re way better than we were a couple years ago, but it still has a lot to be desired. I don’t know, what do you think? Let me know in the comments. As far as pricing goes, the price on this one dropped from 12 cents a second in the last model down to 7 cents per second in this model. So, if you are making a full 5-minute video, it’s still going to cost you about 21 bucks. Now, a few other quick things to

06:13 note about this one. Everything I’m showing you is ByteDance’s own demo reel, and some early viewers did notice like background shifting around in their clip. Again, it’s only 720p. You’re not going to get 1080p or 4K out of this. And because this is only one photo plus any audio, there’s some pretty obvious potential deepfake concerns with stuff like this. Although, if you saw what I just saw, I don’t think we have to worry about too many people being fooled just yet. This one is available now through

06:45 Byte Plus and through their API. All right, now let’s talk about one of my more favorite topics recently. I’ve been absolutely loving these. Let’s talk about world models. So, these are AI models that generate a video world that you can move around in. And Tencent’s Arc Lab released one called World Crafter recently. And this one starts from just a single image or a single line of text. Now, one of the big issues that we’ve had with world models is that if you’re inside a world model and you

07:16 look away from something and then turn back, a lot of times it’s like gone or changed. The model only remembers the last few seconds of what it showed you, and if you move around the room, everything kind of goes wonky. Now, World Crafter’s fix is a memory that you look up by camera position. So, when you point the camera somewhere, it pulls the past views that match where you’re looking and feeds those back to the generator. So, when you turn back to the original restaurant, it remembers what it saw the first time. So, they prove it

07:48 with this little trick here. They take the video the model generated and rebuild a 3D point cloud from it. If the world were inconsistent, that 3D reconstruction would come out as just like mush. Instead, you get clean geometry and the camera path lines back up when it revisits a spot. It handles moving things, too, like a character going out of frame and then later actually coming back into the frame, like we can see with this horse video here, where the camera takes the horse out of frame, and then when the camera

08:19 pivots back around to the right, you can see that the horse is still there even though it was moving. Now, on their own benchmark of 145 uh different scenes, they say that it beats the other models they tested on revisit consistency and camera control. But, just keep in mind that that is their benchmark. So, you know, take that with a grain of salt. The code for this is available on GitHub right now, and there’s even a version on Hugging Face that you can go and play around with right now. In my own testing, though, telling it what to

08:50 generate, it’s kind of not the fastest thing in the world, but eventually it gets there. ByteDance also showed off their own version of a world model recently at an event in Beijing. My buddy Tim from Theoretically Media, he posted some footage that somebody sort of took during the event. So, it’s like this bootlegged camera version. Now, the wildest demo they shared was they took a scene from Stephen Chow’s movie King of Comedy and generated a brand new camera

09:20 path through it. It’s like reshooting a shot that was filmed decades ago. You can now get new angles on that shot. You can also type a description to get a world spawn a giant spoon and then climb it and walk through a portal from a room into outer space. Now, there is a catch, and that’s that we can’t use it yet. It doesn’t even have a public name yet, and there’s no release date, and the only footage I’ve actually seen of it was from people who recorded it at this event, as, you know, Tim from

09:51 Theoretically Media shared here. So, yeah, we’ll keep an eye on that one, but it looks pretty impressive. Real quick, before we get back to the video, I wanted to show you guys how I’ve been using this new feature, Dots from OpenAI. Now, I’ve talked about this in a recent AI news video, and since then, I’ve dug into it even more. Basically, Dots are these always-on AI agents that you can assign ongoing responsibilities to, and they keep on working on them even when you’re not using ChatGPT anymore. So, for example, I’ve been using my Dot to help me with my YouTube analytics and updating them

10:22 automatically every time I post. I connected it to my YouTube channel, and it pulls views and engagement data to find patterns across my titles and topics and angles and post times. And I also have a shared space with my team, where my Dot keeps all this organized and creates new subpages with weekly reports and video ideas based on what’s working. What’s cool is that each Dot has its own cloud computer and can use ChatGPT work and Codex to get things done. It’s powered by GPT-6 Astra and learns my preferences, so it can proactively suggest things without me

10:54 having to ask. So, check it out in the link in the description, and thanks so much to OpenAI for sponsoring this portion of today’s video. All right, so we already saw two world models trying to fix the memory problem. Well, Sky Works Matrix Game 4.0 takes a completely different approach. This one stops asking the AI to remember the world at all. With a normal world model, the AI is doing everything. It’s drawing the frame, but it’s also the physics engine, the game rules, and the memory. When something leaves the screen, the only

11:25 record it ever existed is in the pixels the model already drew, and that’s kind of why things tend to vanish. Matrix Game splits that job into two. An ordinary game engine runs the game. It knows where every enemy is, what happens when you shoot something, what’s behind you. It draws a cheap, blocky, low-poly version of that scene. Then the video model’s only job is to repaint that blocky frame so it looks finished. So we can see in some of these models down here, on the right is what the game

11:56 engine actually draws. On the left is what you see. It’s the same frame in the same moment, but the engine handles the gameplay and the AI handles the look that’s skinned over the gameplay. Since this game engine is keeping track of everything, the world’s not going to forget a pyramid behind you is still in the engine’s map whether the AI is drawing it back in or not. Here’s another example of a car game. You can see the blocky, early rendered version and then AI putting its own graphics

12:28 over the top. So the game engine only has to remember this and the AI does that. There’s also another piece. If you ask for something the game rules don’t cover, a coding agent writes new rules into the engine and updates the prompt the video model uses. So the world can change mid-game without the AI having to invent new physics on the fly. Down here we can see three games, a medieval castle shooter, an ancient Egypt shooter, and city racing. These long play sessions can run up to about a minute and a half without any sort of

12:59 cuts. Now there are some catches. Each game has its own model and each one is about 5 billion parameters each, trained just for that scene. So it’s not really a general make me any game model and the output is 480 by 864. So it’s not anywhere near HD. Now the project page does talk about running on a single RTX 5090, but the model card does say there’s no guaranteed frame rate, consumer GPU memory fit, or exact scene

13:30 reconstruction. So, it may not totally fit into your GPU’s memory, and fast camera moves and long sessions still can cause some drift, and the actual training code isn’t included. But, you can get the weights and the inference code under Apache 2.0 license, so you can use them commercially. Now, there is a hugging face space listed here for hugging apps Matrix game 4.0. So, you should be able to play around with it here. They do have some examples that are pre-cached in if you want to check out some of those examples. Now,

14:02 personally, I think this engine plus like AI paint job or AI reskin is a pretty realistic path to AI games, at least more so than like asking a model to do everything. Turns out Nvidia thinks so, too. Next up, let’s talk about something that came out of Nvidia called Stream Render H3. This is actually a pretty similar concept to what we just looked at. It uses kind of that same split. Code runs the world, and then a video model draws it in real time. The difference with this one is

14:33 how little the game actually hands the AI. Matrix game gives the model a blocky 3D render. Nvidia gives it a color-coded map. Every pixel is a flat color that means road or grass or barrier or rival car. There’s no lighting, no textures. The video model actually draws everything you see. So, here’s the game engine’s own view, the color map that it actually sends, and then the final frame that, you know, the user sees. Now, because the map carries no like looks at

15:05 all, you can actually restyle the whole game by swapping one reference image. We can see here that this is all the exact same drive, but when we fast forward a little bit, they can swap out the scene for anywhere they want and put the exact same drive in any different scene. It’s the same exact input what we see in the middle just rendered out in different locations and different settings during the day and things like that. So, one thing to notice here, this is kind of one of the hard parts for this model, is that the color map paints all of the

15:35 rival cars the same exact color. So, the model has to actually remember on its own which car is which and make sure to keep the same paint job across the various cars. So, a fun fact about this model is they actually handed Claude code the job of building a faster video decoder for this. They gave it a speed target and then they had a working version running in the pipeline in about 3 hours. Now, this does actually require some serious hardware. Each rendering step takes about 218 milliseconds.

16:06 However, that’s on eight GB200 GPUs. These are like the chips that they put into data centers from Nvidia. So, this isn’t something you’re going to be running on your gaming PC, at least not for a while. Also, it’s only been trained on racing so far. That’s why all of the demos are like race cars. So, you can put this one kind of next to Matrix game which we just looked at and you get a decent picture of the trade-off. Matrix game aims for one consumer GPU at like lower resolution where Nvidia’s

16:38 going for photoreal but out of rack of data center hardware. Okay, this one’s a little weird but I want to show it off cuz it’s also kind of cool. So, Odyssey, the world model company that launched Odyssey 3 not too long ago posted a clip saying they put a fruit fly brain inside their world model and now it flies around imagined worlds by itself. So, this fly brain wasn’t actually created by Odyssey. It’s a simulation built by a company called Eon systems from a complete map of a fruit fly’s brain,

17:11 about 140,000 neurons and 50 million connections. Odyssey swapped out the virtual body that brain was hooked up to and connected it to a camera moving through an Odyssey 3 world. I don’t know. I thought this one was fun, but there’s like no paper, no code, no explanation of how the two are connected. And people in the comments did note that the fly never actually tries to land anywhere, which, you know, if it was a real fly, it probably would. But it’s brought up a lot of questions about, you know, simulation theory.

17:41 Like, does this fly believe it’s real and in a real world, but it’s actually in a simulation? And if so, what are the implications of route simulation theory for, you know, humans and stuff? We don’t need to go down that rabbit hole, though. While you can’t actually play with this fly flying around inside of Odyssey 3, you can actually mess around with Odyssey 3 itself. Just head over to experience.odyssey.systems, give it a prompt of what you want it to create, and you can actually interact in

18:11 this world. I am steering this fox right now. I’m telling it to jump. I’m telling it which direction to go. I’m changing camera directions. All of this is me actually controlling this. Pretty cool. All right, moving into 3D. Preferred Networks, which is based in Japan, just released Telescope, which turns a set of photos of a place into a detailed 3D mesh. Like, from a single object all the way up to city blocks. Now, the usual way to get a high-quality mesh from photos is to spend a long time fitting a

18:42 model to that one specific scene. Telescope actually skips all that. It’s a generative model, so it’s learned what surfaces generally look like, and it fills in gaps where you didn’t get many photos. It also handles shiny and glass surfaces, which normally wreck photo-based 3D. So, the name is actually the method itself. It builds the whole scene at low detail first, then zooms in on areas that need more detail, and refines them, like a telescope. The colors here show which level each

19:13 surface came from. So, if we look here, we have a Gaussian wrapping, which is sort of the standard method. And if you look at this, it looks pretty good, but it’s lacking a lot of detail. Now, this is telescope. And there’s so much more detail here. So, here’s the original Gaussian wrapping, and here is telescope. Like, it’s a pretty big difference. Here’s another example of Gaussian wrapping. And when I move this over, you can see their version and all of the details pop into focus. Now,

19:43 there is a little bit of a catch. You do need a lot of photos to achieve this. You can see this one took 256. This one also took 256 right here. This one actually only took 56, which isn’t too bad. And it can only rebuild what’s inside the area your camera actually moved around. This is one we don’t have any sort of real access to. It does say the code is coming soon, but it looks like a pretty promising leap from what we got out of Gaussian splatting. Here’s another pretty cool one called Gaussi Animate. And this one’s actually used

20:15 for animating 3D characters. It’s designed for rigging up characters that are soft on the outside, but rigid on the inside. So, people in loose clothes, animals, or garments just on their own. So, a normal animation skeleton gives you control, but a loose shirt or like a dog’s belly doesn’t move like bones. So, this adds on a whole bunch of other floating nodes on top. These additional floating nodes sort of capture all the jiggle and flow and the additional physics that come along with the, you

20:48 know, the cloth or the material on top of the bones. So, we can see here that it finds the sort of shape, but then once it finds the shape, it actually puts a skeleton in the middle to figure out which parts need to actually be rigid. Once it figures out the points that need to be rigid, then it figures out what points need to sort of flow outside of the bones. So, here’s some other examples like a clothed human. You have the skeleton, you’ve got the bones, you’ve got the skinning, and then everything combined where you can see the the dress sort of flow. Now, you do

21:19 need a captured motion sequence of the character first. It’s not going to just work from like a single image, but the code for this one is available on GitHub and available for research use right now. All right. Now, this is the last one and it’s actually not AI, but I just thought it was really cool, so I wanted to share it anyway. This one’s called World Gacha. I think I’m pronouncing that right. And it’s actually a free browser tool from a Japanese developer that actually creates random fictional regions that has mountains and rivers

21:50 and coasts and towns and railways and rice paddies and and almost anything you can imagine and it draws it as like this manga style line art. Well, this one’s actually not in that manga style line art. This is a randomly generated city, but you can see if I add it with the lines, you kind of start to get that look. You could also put it on lines and tone and you get like a black and white version. You could put it on this like clay mode and this is again a randomly generated seed. I could re-roll the dice and it’s going to build me a whole new

22:20 town or city or whatever. So, now I can zoom in on this city that it just created. We can also change it to downtown. It’s going to build us a new city that looks like a downtown area or skyscrapers. It’ll build us a new city and these are again all randomly generated every single time and you can zoom in to different views. So, let’s say we want like a top view, we can look straight down on this city like this. Maybe you want an eye level view of Central Street and it actually brings us into the city where we can look around

22:51 and wander around in this city. Now, imagine using something like this and then re-skinning it with like one of the AI models we were looking at earlier. You can build a pretty primitive looking city and then use AI image or video models to reskin it into something really realistic and maybe even build scenes for like films and animations and stuff. Again, this is not actually an AI model. There’s no outside data. It’s just random numbers and math generating all of this. And if you do like a specific scene, you can actually export it as a PNG or an SVG or even a 3D

23:24 object file. So, you could theoretically create a world and then pull it into Unreal Engine and build a game inside of it or something like that. I know I usually focus on AI stuff, but really, you know, the theme of my channel is just cool stuff that I come across online that I want to share and sometimes it’s not always AI. But that’s what I got for you today. Again, I make a news video every single Friday where I break down all of the news that came out in the AI world each week, but it’s usually the biggest stories with the most impact on the most amount of

23:54 people. I’m sort of trying to filter out the signal from the noise of everything that’s going on and help you understand what might actually change what you do on a day-to-day basis. And in doing that, sometimes I miss some of these smaller research projects that don’t get a ton of love and not a lot of people talk about. And I find all of this stuff really cool and really fascinating. And if you go back like 3 years on my channel, this is the type of stuff I used to talk about a lot more frequently until all of the big labs and big companies started having new launches

24:25 every single week and I kind of shifted more focus to that. But I want to get back to my roots a little bit and make these videos a little bit more often. So, if you liked videos like this where I sort of dig and find some more underground things that are happening in the world of like AI and just cool tech research, let me know. I’ll make sure to make more videos like this. I’m planning on making at least one of these a month because it’s just really fun for me to dig in and find some of this stuff. But again, that’s what I got for you. If you like videos like this and you also want to be super looped in and informed on

24:57 all the latest AI news. I’m going to keep up with all of that for you. I put out a video every Friday with all of the news. I also put out a newsletter twice a week. You can subscribe for free over at futuretools.io. But, thanks so much for hanging out with me, nerding out with me today. There’s a lot to nerd out on. I really, really appreciate you, and hopefully I’ll see you in the next one. Bye-bye.