Matrix-Game 4.0 Is CRAZY: AI Paints a Playable Game From Plain Boxes

Author: Prompt Engineer 48 · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Matrix-Game 4.0 Is CRAZY: AI Paints a Playable Game From Plain Boxes · Channel: Prompt Engineer 48 · 19:42 · Watch on YouTube

Topics: AI agents, Software engineering with AI, AI content generation, Model inference and optimization, AI models

Summary

Matrix-Game 4.0 combines a code-based game engine, a multimodal coding agent, and a video model to render playable worlds in real time. The engine owns game state and rules, while the video model turns simple proxy graphics into detailed visuals; the coding agent connects the two and can modify game mechanics. The authors report 480p gameplay at 60 FPS with 112 ms latency on an RTX 5090, but the narrator did not independently test the system and notes that these figures depend on specific hardware and settings. The project is promising, though it currently supports a limited set of games and needs independent testing and broader evidence of generality.

Key points

The design assigns distinct roles: the game engine is the referee, the coding agent is the script writer and translator, and the video model is the painter.
Game rules and state—including health, inventory, collisions, and quest progress—are enforced by code rather than inferred from generated video.
The video model renders from low-poly proxy frames, dynamic captions, reference frames, and spatial memory.
Training uses paired low-poly and high-fidelity clips captured in Unreal Engine 5, with separate rendering models fine-tuned for each game.
The authors report 480p gameplay at 60 FPS and 112 ms latency on one RTX 5090; the narrator did not reproduce these results.
The report’s comparisons with other systems are not controlled head-to-head tests, and the systems use different resolutions and latency measurements.
The project page shows castle, desert, racing, and air-combat footage, but released weights are available for castle, Egypt, and city racing.
Because mechanics live in code, a coding agent can add modes and rules such as shields, checkpoints, collision penalties, and pursuit objectives.
The current system is a lightweight-game preview, not a replacement for large studio pipelines; the authors list lower inference costs, higher fidelity, multiplayer, and improved visual control as future goals.
The narrator’s RTX 4060 with 8 GB of memory could not run the reported setup, which uses about 18.2 GB of peak memory.

Chapters

Start	Chapter	Summary
00:00	The Matrix-Game 4.0 claim	The narrator introduces the system’s real-time rendering claim and explains that the video will assess the project materials rather than present an independent test.
01:02	Why separate the game from the video model?	The video outlines the limits of pixel-only interactive world models and introduces an engine, coding agent, and renderer with separate responsibilities.
04:08	Training data and dynamic captions	The narrator describes paired Unreal Engine footage, per-game fine-tuning, and rule-based captions that track objects and their changing states.
06:43	Model architecture, memory, and training	This section explains the video model’s inputs, spatial and temporal memory, and the training stages used to make generation causal and fast.
09:49	Speed results and comparisons	The report’s successive inference optimizations lead to its 112 ms latency figure, followed by a comparison with other systems and caveats about the measurements.
12:28	Results, gameplay footage, and limitations	The narrator reviews held-out clips and longer sessions, noting that the visuals look coherent while emphasizing that the examples are curated and not independently tested.
15:01	Changing game rules and accessing the project	Examples show the coding agent adding game modes and mechanics, then the video covers the repository, available weights, licensing, and hardware requirements.
17:37	Limits and the narrator’s verdict	The narrator summarizes the system’s current constraints and argues that its main contribution is a division of labor between code, the engine, and neural rendering.

Quotes

It is the same game, the same frame. — 00:00

The engine is the referee, the coding agent is the script writer and translator, the video model is the painter. — 02:03

Code owns the rules, the video model owns the look. — 04:08

Everything you saw today is the author’s footage, and I have labeled it that way. — 17:05

It is the division of labor, code for rules, engine for state, video model for pixels. — 18:39

Concepts and entities

Matrix-Game 4.0, Skywork AI, Unreal Engine 5, RunPod, Matrix-Game, RTX 5090, Unreal Engine 5, dynamic captions

Full transcript (original language)

00:00 Look at the left half of this frame. Real stone walls, palm trees, smoke, and textured character. Now, look at the right half. Gray checkerboard floor, flat colored boxes, a character made of blocks. It is the same game, the same frame. And according to the authors, a video model painted the left side from the right side in real time. This is Matrix Game 4.0 from Skywork AI. The headline claim is a playable game world at 480p, 60 frames per second on one RTX 5090 with 112 milliseconds from key

00:31 press to the first frame that reacts. That is the paper’s own number. I have not reproduced it, and I will tell you exactly why later in the video. What I can do is go through the project page, the repo, and the technical report with you line by line and explain what is actually new here because the interesting part is not that a video model draws pretty pictures. The interesting part is what they refuse to let the video model do. >> [music] >> Let’s start with the problem. A lot of the recent interactive world models, the

01:02 Genie style ones, work like this. You press a key, the model predicts the next chunk of pixels, and the pixels become the world. There is no game underneath. The video is the game. The report lists four things that go wrong with that. One, nothing corrects the model from outside, so the world state only exists implicitly inside generated pixels. Once a chest leaves the screen, nobody is keeping track of it. Two, small errors pile up. Each new chunk is conditioned on the last one, so a slightly wrong wall becomes a very wrong wall a minute

01:33 later. Three, memory is just old frames, so the cost grows with how long you play. And four, the mechanics stay thin. You can walk around, but health, ammunition, quest progress, a shield that breaks, those are hard to enforce when the rules have to be guessed from pictures. The authors say something I like here. They argue that today’s video generators are not really world models. Much of the apparent intelligence comes from the text encoder or the language model upstream and the diffusion part acts more like a renderer. So, their answer is let it be a renderer, only a

02:03 renderer. So, here’s the design, three parts, a game engine, a multimodal coding agent, and a video generation model. The report gives each one a job title and honestly the titles are the clearest way to remember it. The engine is the referee, the coding agent is the script writer and translator, the video model is the painter. The referee first, the game engine keeps the real estate of the world. Health, inventory, gravity, collisions, quest progress, all of it lives in code and is enforced in code. It does not draw anything fancy. It

02:34 draws what the paper calls a low-poly proxy boxes for buildings, a cylinder with a sphere for a tree, a figure made of blocks for a character, black outlines, no texture, no shadow, black sky, gray checkerboard ground. And here’s why that matters. If an enemy walks out of view, the engine still knows where it is. If you shoot an oil barrel, the engine decides whether the shot hits and whether it explodes. The video model never has to remember or decide any of that. It just gets handed a picture of the consequences. Now, the

03:04 script writer, the coding agent works in two directions. Downstream it translates the engine says, “There is a box here with this object ID.” And the agent turns that into words, like an old thatched cottage weathered by sea winds with a dark red wooden door. Upstream it writes, “You tell it make monster swarm in from all directions.” And it edits the scene configuration and the game code including where they spawn, how they move and attack, and what counts as winning. If you leave details open, it improvises. The paper’s example, a player opens a door that has nothing

03:35 behind it yet. The agent imagines a storage room with a locked iron chest in the corner, registers it in the engine, and from that moment the engine owns that chest. It can be opened or not, by the rules, not by the vibes. Then the painter, the video model takes the proxy frame, the current text description, a few reference frames, and a first frame, and turns them into materials, lighting, and atmosphere. The whole loop is player input, engine updates the world, proxy and control signals come out, the video model renders, and the player sees the result.

04:08 Code owns the rules, the video model owns the look. How do you teach a model to do that? With paired data. The team captured everything in Unreal Engine 5, and every clip is recorded twice from the same camera at the same frame index. As the low poly structure render, once as the full high fidelity picture. So, the two videos line up pixel by pixel. Table one in the report lists these scenes: Medieval Castle, a third-person shooter, 17,848 training clips. Ancient Egypt desert, also a shooter, 40,296.

04:41 Desert canyon air combat, 28,720. And a city sample, megacity for driving, 27,470. Total, 100 14,334 clips, 1,712 hours. They also hold out whole capture sessions that never touch training, three sessions each for castle, desert, and city, and entire capture batches for air combat. So, the clips you see in the results were not seen in training. Worth saying clearly, because it is the first thing I would check. One more design choice, they

05:11 train one rendering model per game, not one giant model for every world. A castle shooter model, a desert shooter model, a city driving model, each fine-tuned on its own data. Keep that in mind when you think about how general this is. The second ingredient is what they call dynamic captions. In a game, the picture changes with every action you turn the camera and different objects are visible. You trigger something and an object changes state. A single fixed prompt cannot follow that. So, every object in the scene has an ID linked to a description. Each frame, the system

05:42 reads which objects are visible, their states, and their distances from the engine, and fills the template. No vision language model involved. The paper stresses this rule-based no VLM. Look at the example from the city driving clip. Frame four lists parked minivans four and 10 m away, pedestrian standing 30 m off. By frame 53, the pedestrians are walking and getting closer. And a dark gray building comes into view. By frame 155, a trash bag and a street lamp enter. The text changes with the world. The proxy and the

06:12 caption split the work. The proxy supplies structure, where things are, how they move. The caption supplies meaning, what each thing is and what it is doing. Together, they pin the model down from both sides. And you can watch this on the project page. There’s a prompt panel for a 40-second pyramid run, 12 segments, one every 80 frames. It shows a scene summary with the player’s state, enemies, and landmarks around, and the events in that segment written by the harness. Player action, enemy action, hit, explosion, enemy defeated, things entering and leaving

06:43 view. All color-coded. Now, the painter itself. It starts from 12.2, the 5 billion parameter model in its fun control variant, and turns it into a video-to-video model. Four things condition each chunk. The low-poly proxy concatenated with the latent being generated, the dynamic caption, spatial memory reference frames, and the first frame which connects the new video to what is already on screen. The numbers, if you like architecture, 30 transformer blocks with 3,072, 24 attention heads,

07:14 one latent frame is four video frames and at 480 by 864, that is 405 tokens per latent frame. The text encoder is UMT5XXL, and the captions, keys, and values are cached and only recomputed when the caption changes. The part that makes it a game rather than a movie is the cache. Each new chunk attends to three kinds of memory, up to three spatial memory frames retrieved by camera pose, the reference frames, and a sliding window of the latest 20 chunks. After a chunk is generated, the model runs once more

07:45 on the finished result with a control input removed and writes that clean version into the cache. So, later chunks look back at clean history, not at their own noisy drafts. That special memory is how it copes with going back to a place. Candidates are picked from frames at least 40 frames earlier, keeping ones whose view overlaps, and choosing the closest by camera pose. It is not a perfect memory, and the engine is what really holds the layout, but it keeps the look consistent when you return. Training happens in two big phases.

08:15 First, supervised fine-tuning. The model learns as a bidirectional video-to-video model on 81-frame windows at 704 by 1280, using 64 GPUs, a learning rate of 10 to the minus five, 4,000 steps. To support guidance and generation without a first frame, they randomly drop the first frame 30% of the time, the prompt 20%, and the proxy control 20%. Then comes the part that makes real-time possible. A bidirectional model sees the whole clip at once. A game cannot do that because the future has not happened yet, so they distill it into a causal

08:47 generator following a method called causal forcing plus plus in three stages. Stage one, teacher-forced autoregressive diffusion. Each latent frame gets its own noise level and can only look at clean frames before it. 20,000 steps. That gives a causal teacher. Stage two, causal consistency distillation. A frozen teacher, an online student, and an average copy of the student working on a grid of 48 noise levels. After 5,000 steps, the student can make a chunk in four steps.

09:18 Stage three, distribution matching, the DMD family. Now, the student learns from its own rollouts and ends up denoising each chunk in two steps, and the very first generated frame in four. One practical detail I liked, the deployed system uses a tiny decoder. So, during this stage, the student sees the tiny decoder version of the control 80% of the time. They train for the setting they will actually deploy. Even a two-step model is not automatically fast. Short chunks mean the model is called very often, and with only one denoising step, each call does

09:49 so little arithmetic that the GPU spends much of its time waiting for the CPU to launch kernels. So, the second half of this paper is basically a speed story. Table two is my favorite part. It is cumulative. Every row adds one thing. Start the causal 1D it with four steps and the full 1 decoder. 4.5 frames per second, 1,110 ms of latency. Swap in the tiny decoder TAEHV 6.5 FPS, 867 ms. Add a custom rotary position kernel and cache text keys, 15.4 FPS and 551 ms.

10:23 Next, 8-bit weights plus a CUDA graph which records the whole transformer once and replays it in a single launch, 24.2 FPS, 466 ms. Then one-step distillation, 51.2 FPS, 376 ms. Then asynchronous input and output, 57.7 FPS, 367. Then the big latency cliff, fused Triton kernels take it to 65.8 FPS, and latency drops from 367 to 121 ms. A ring buffer for the key value cache, so the newest

10:55 frame overwrites the oldest slot instead of copying the whole window, gives 73.7 FPS, and split key value attention, which cuts the window into seven ranges and merges the partial results, lands at 83.4 FPS and 112 ms. Overall, 19 times faster and 9.9 times lower latency from 1,110 ms down to 112. At one step, it denoises a four-frame chunk in 48 ms with about 18.2 GB of peak memory, text encoder, transformer, and decoder all on

11:27 the same card. Two steps gives roughly 57 FPS. And there are smaller card numbers with fixed prompts and one step about 67 FPS on an RTX 5080 and 58 on a 5070 Ti using around 12.8 and 10.9 GB. The repo news says those optimizations landed on October 16th. Notice that fixed prompts condition though, the dynamic caption path costs something. How does it stack up? Table three compares four recent systems that also render code defined worlds with video

11:58 models, code world model streaming 10.2 seconds on 1 H 200 with peak memory above 100 GB stream render H3 517 milliseconds, but on 8 GB 200 GPUs. Agent Garden 878 milliseconds on 1 H 100, Magpie 1.55 seconds on 1 H 100, and Matrix game 4.0 112 milliseconds on 1 RTX 5090. A single consumer GPU with 32 GB, the paper calls that about 1/5 of the next system and the only one running

12:28 on a consumer card. Two cautions from me. First, these are numbers the authors compiled from each project’s own reports and the table’s own caption explains the way each latency is computed, so they are not a controlled head-to-head. Second, resolutions differ. Matrix game is at 480p, some of the others are at 768p still. The order of magnitude gap is real enough to be worth noticing. Now, the part you came for, results. The project page has a gallery, one held out clip per game, 15 seconds each. Castle,

12:58 pyramid, racing, and air combat. Left of the divider is the generated frame, right as the engine’s low poly proxy. Watch what the model adds that the proxy does not contain. In the castle, flat blocks become stone walls with weathering, sparks and smoke show up when something is hit. In the desert, the boxes become sandstone buildings, thatched roofs, and palm trees. In racing, a city of gray prisms becomes facades, glass reflections on the road, and the character keeps the same look from the first chunk to the last. The report runs 40-second rollouts, 961

13:30 frames at 24 FPS that start from the real first frame and then go fully auto aggressive. Figure 7 shows them at 1:11, 21, and 31 seconds. The layout of the proxy holds, the camera follows the same path, and the look stays stable. That is the claim, and the frames in the figure do support it. The page also has long play sessions, continuous sessions from 54 seconds to a minute 39, shown with the game HUD and the key presses, and with game audio. Here is the castle one, then the pyramid. As you watch, pay

14:00 attention to the edges. What happens when the camera turns quickly, and what happens when a lot of things explode. I will be honest about what I see. These are curated clips on the author’s own page from the games they trained on. They look impressive, the structure follows the proxy, and I do not see the usual melting you get from long unconditional rollouts. But a curated page is a curated page. The real test is hands-on a keyboard, and I have not done that. Here is the racing one. Notice how the road, the buildings, and the traffic

14:30 stay put while the car moves fast. That is the engine’s proxy doing its job. A pure video model would have to invent a consistent street in real time with nothing holding the street in place. And air combat, the fourth scene, it is in the gallery and in the report’s data table, but notice it is not among the three games that have released weights. The release has castle, Egypt, and city racing. I will come back to that. Now, the feature that I think sells the whole idea. Because the rules live in code, the coding agent can change them, and

15:01 the video model just keeps rendering whatever the engine outputs. Figure 8 shows three modes built that way. In the shooter, the request was add a shield that breaks at its durability limit. The agent added health and ammunition control, shield durability, a shotgun with a higher fire rate, a game over condition at zero health, and simple sound effects. In the driving game, add road checkpoints and collision penalties. Give a time trial with checkpoints, penalties, and drifting. In the same driving game, turn into a pursuit mode, the player has to

15:32 intercept a target vehicle and stay close for 5 seconds. The report says the engine evaluates every rule and the video model renders each frame from the engine output. A shooter where everyone fights alone could become a team battle, a time trial could become cops and robbers. I like this because it changes who can prototype describing a rule in a sentence is much cheaper than hand-building a new art pass for it. The authors also say plainly that this is a lightweight engine and lightweight games. It is a preview of a workflow, not a replacement for a big studio

16:04 pipeline. Can you run it? Let’s look at the actual repo, Matrix game on GitHub from Skywork AI with the V4 folder holding inference, a browser demo, and the game source code. Weights and game assets for Medieval Castle, Ancient Egypt, and City Racing are on Hugging Face. At the time I looked, the repo had about 2,700 stars and 286 forks. One licensing note, because the repo is slightly confusing, the V4 readme says Apache 2.0 and points to the V4 license file. The older

16:34 versions 1, 2, and 3 stay under the MIT license in the repo root. Third-party components and assets keep their own licenses, so check before you ship anything. The readme lists single GPU inference paths for a 100, RTX 5090, RTX 5080, and RTX 5070 TI. The 5090 path includes FP8 linear layers, fused kernels, CUDA graphs, and an optional Sage attention back end. Offline rendering and an online browser demo are both there. So, did I test it? No. My laptop here has an RTX 4060 with 8 GB of

17:05 memory, and the report puts peak usage at 18.2 GB before you count anything else. It will not fit. I could have rented a 5090 on RunPod, but a fair test needs the optimized path, the weights, and the game assets configured properly, and I did not want to hand you a half-baked result. Everything you saw today is the author’s footage, and I have labeled it that way. If you do want to try the optimized path, you need the hardware the readme names. Otherwise, treat the report’s speed table as what it is, a vendor measurement carefully described on a single 5090 with a

17:37 setting I can read and you can check. What are the limits? The authors list them themselves, so let’s start there. In the near-term, they want to reduce inference cost and hardware requirements with the goal of a technical preview on Steam. They say they still want better visual fidelity, a refinement stage for higher resolution, multiplayer with consistent views across players, and more precise control over generated visuals. From my reading, here are the real constraints. The output is 480p. It is one model per game, so a new world means new pair data and a new fine-tune.

18:08 The data came from Unreal Engine 5 assets, so new scenes cost real capture work. The released games are three, and air combat is not among them. And the speed numbers come with conditions. 83 FPS is one denoising step on a 5090 steady state. 60 FPS gameplay with 112 milliseconds is their evaluated configuration. The 67 and 58 on smaller cards use fixed prompts. None of that is hidden. It is all in the report, but it is worth reading the footnotes before you repeat a headline. So, what is my take? I think the last

18:39 thing I did here is not the speed record. It is the division of labor, code for rules, engine for state, video model for pixels. Each part does the thing it is actually good at, and the video model is never asked to be a physics engine. If that holds up in the wild, it also says something about where generative games might go first. Not as a single model dreaming an entire game, but as a neural renderer sitting on top of a cheap engine with a coding agent that rewrites the rules when you ask. That is a much more believable path to something people can build on. Is it

19:10 finished? Obviously not. It is a technical report with open code weights for three games and some very nice footage. What I would want to see next is independent testing, other people’s scenes, and what happens when the proxy is rough or wrong. If you’re looking for cheap and reliable GPUs, I use RunPod, and the link is in the description. If you want me to rent a 5090 and try this properly, tell me in the comments, and tell me which of the three games you would want to see first: Castle, Egypt, or Racing. The links to the project page, the report, the repo,

19:42 and Hugging Face are in the description. Thanks for watching, and I will see you in the next one.