I Ran Agent Garten's Neural Renderer on an H100 (and Built My Own Code World)

Author: Prompt Engineer 48 · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: I Ran Agent Garten’s Neural Renderer on an H100 (and Built My Own Code World) · Channel: Prompt Engineer 48 · 15:01 · Watch on YouTube

Topics: AI agents, Model inference and optimization, AI models, Multimodal models

Summary

Agent Garden separates a program-defined world, which stores its state and rules, from a shared neural renderer that turns geometry into visual frames for agents. The video reviews the project’s hide-and-seek results and renderer, then explains which parts are publicly available and which are not. The creator runs the released renderer on an H100 using a small custom Python world, and tests an agent loop that learns through written playbooks. The renderer worked at the reported speed, but the creator’s small experiment showed only a weak learning effect, leaving the project’s central claim about rapid agent improvement unverified from the outside.

Key points

Agent Garden keeps world state and rules in executable code while a learned renderer generates the pixels agents see.
The same renderer is used across different worlds and visual styles; changing the code can create new environments without new 3D assets.
The released renderer is based on a 7-billion-parameter student model, and the report claims about 36.5 frames per second at 480 by 832 on an H100 with the cache full.
In the project’s hide-and-seek setup, pretrained language-model agents act by writing short Python programs and improve between rounds by adding lessons to playbooks.
The creator notes that comparing the project’s rounds with OpenAI’s training episodes is not a like-for-like comparison.
The repository includes renderer training code and streaming inference, but the checkpoint, real-time rendering engine, code worlds, and agent practice loop were not released at the time of recording.
The creator built a 12-by-12 courtyard and ran its depth and surface-normal outputs through the released renderer on a rented H100; a small red hider was rendered inconsistently depending on the reference image.
In the creator’s agent test, playbooks produced a small gain on six held-out layouts, but results were noisy and varied between layouts.
The creator concludes that the renderer is usable and the speed claim held in their test, while the claim that agents improve in just a few rounds remains promising but unverified independently.

Chapters

Start	Chapter	Summary
00:01	What Agent Garden does	The video introduces the code-defined worlds, shared neural renderer, hide-and-seek demonstration, and project gallery.
03:07	The renderer and agent learning loop	The creator explains the design trade-off, rendering pipeline, training approach, and how agents learn through playbooks.
06:13	Results, caveats, and what is released	The video reviews results from several worlds, qualifies the comparisons, and checks the repository’s release status and limitations.
08:17	Building and rendering a custom world	The creator describes a small Python courtyard, runs its geometry through the renderer on an H100, and compares visual references.
11:24	Testing an agent with playbooks	The creator tests a round-based agent loop, reports noisy results on new layouts, and assesses what the experiment demonstrates.
13:26	Conclusions and cost	The video summarizes what worked, what remains unverified, and the resources needed to try the renderer.

Quotes

code keeps the world, the renderer shows it. — 00:32

The training idea is called adversarial forcing. — 04:38

generating a world is the easy part. The harder question is whether the world is worth an agent’s time. — 08:17

So that part is promising but unverified from the outside. — 13:58

Concepts and entities

Agent Garden, Mirus Lab, neural renderer, adversarial forcing, NVIDIA Cosmos 3 Nano, WAN 2.2, H100, Runpod, Claude Sonnet, Claude Haiku

Full transcript (original language)

00:01 All right, in this video we will see Agent Garden, a research project from Mirus Lab. Right now, every tile on the screen is plain geometry, just shapes that a program exports. And now this is what their neural renderer turns that geometry into in real time. 20 tiles all playing at the same time. And mind you, there are no textures and no game engine art behind any of them. I went through the report, the blog, and the repo. Then I built a small version of it on my own laptop and I ran the released renderer on a rented H100. So let’s go ahead and

00:32 see what is real, what is not released yet, and what happened when I tried it. This is the project page and the headline is code keeps the world, the renderer shows it. So the objects, the rules, and the goals of a world live inside an executable program and a learned neural renderer draws what the agent actually sees. Here is the clip that explains it best. The top row is what the program exports. The plane view, the surface normals, and the depth. The bottom row is the same game rendered three ways. A haunted night town, a dark cave, and a Minecraft

01:02 village. The game did not change, only the look changed. And their first big test is hideand-seek. The same game Open AI used back in 2019. A hider, a seeker, a few boxes, and a ramp. But this time, the agents are language model agents that look at pictures and they learn by writing notes. They also made a 2 minute 40cond film. It opens in 2019 with OpenAI’s hideandsek where the policies read object states and sensor readings and not images. Then it jumps to today’s

01:33 version, an agent in a richl looking world writing a little Python to act. In the film, round one shows four agents playing in parallel with a task file that holds the rules, the actions, and the budget. And the caption on the later round says, “By round six, the seeker can no longer get in.” It closes on the framework, a code world on one side and a neural renderer on the other. Then the same game rendered in four different visual styles and a robot arm with a pipette to show that changing the code gives you new tasks and new environments.

02:05 The gallery on the page has 18 worlds and every clip is split the same way. Geometry on the left, rendered on the right. This first one is an ant tunnel and you can see the soil, the roots and the small stones appear out of plain brown tubes. Then a robot boxing match from a firstperson view and a cannon on a ship deck with the explosion. A fighter jet on a carrier deck. A catapult shooting a ball toward a basketball hoop. Two robot arms playing chess and a hand cooking in a walk with a flame and the vegetables built from a set of simple blocks. A crane next to a

02:37 truck. A rider on a horse going down a dirt path. A robot arm lifting the lid off a pan. A pipet dropping liquid into a flask. And two race cars in the rain. Each one from its own camera. A roller coaster. A lunar rover. A goalkeeper on a football pitch. A torch next to a campfire in a forest. A wingsuit flight over mountains. And a third person walk through a snowy valley. Now, why build it this way? The blog starts from a trade-off. A simulator or a game engine gives you explicit state.

03:07 You can inspect it, edit it, and test it. But every new look needs 3D assets and a rendering pipeline. So, new worlds are expensive. A video world model is the opposite. The pictures are rich, but the state lives inside the generated frames and the model’s memory. You cannot ask it where box number three is, and you cannot edit one object and leave the rest alone. Agent garden takes one half of each. A program holds the state and the rules. A learned renderer shared across all the worlds produces the pixels. So a new world is just new code

03:38 and there is no new asset work for it. And mind you, it is the same renderer for every world in the gallery. The end tunnel, the chessboard and the cooking pen all go through the same weights and only the geometry and the reference picture change. Here is the loop from the report. The agent sends an action. The engine updates the scene and exports depth and surface normals. The renderer takes those plus a reference image, a text prompt, and its own visual history and produces the next frames. Those

04:08 frames go back to the agent. The renderer starts from the Nvidia Cosmos 3 nano video model. The release student is 7 billion parameters in BF-16. It generates in blocks of four latent frames, four dnoising steps per block, and the video goes through the WAN 2.2 autoenccoder. The speed claim is in table two. One block of 16 frames takes about 439 milliseconds on one H100. That works out to 36.5 frames per second at 480 by A32 with the cache full. And it

04:38 matters because the agent waits for the result of each action before it picks the next one. The training idea is called adversarial forcing. In simple words, the model is trained on its own rollouts and not on perfect history. So errors do not pile up and a discriminator that has seen real videos keeps the textures from decaying. In their 30-cond rollouts, the older method gets repetitive patterns and theirs keeps the detail. There is one more detail I found neat. They replay the roll out in a second pass with gradients and they check that the replay matches the original roll out bit for bit. So, a

05:10 loss on later frames can teach the model how to encode earlier ones without keeping the whole roll out in memory. Now, the agent side in hideand-seek, there is one hider and one seeker five games per round. And the agent only sees the rendered firstperson frames. No coordinates, no hidden state. It acts by writing short Python programs and after each round it reviews its games and updates a playbook. And the agents themselves are not trained. They are pre-trained models and they improve only through what they write down between rounds. So the same loop could run with

05:41 any model that can read an image. The playbook is a folder of small markdown files, one lesson per file. The report shows names like check shelter, object control, and ramp landing. And one line from the RAM file reads, “A ramp route has three separate milestones.” And here’s the headline number. Openai selfplay needed about 25 million games before hiders built shelters and about 100 million before seekers used ramps. The visual agents did the first one in four rounds and the second in 10. You can see the three clips on the page. The

06:13 hider moves a panel to rebuild its cover. The seeker carries a ramp to the wall and climbs over. And in the third clip, the seeker moves the ramp closer after a short climb and tries again. Now, one honest thing, rounds against episodes is not a like for like comparison, and the report says so itself. A round is five games where a model reads what happened and writes notes. An episode is one run of a train policy. So, it is a statement about how

06:44 fast this loop learns and not a fair race. They also try four more worlds, four rounds each, and every card on the page puts round one next to round four. A corgi companion, a one-lane bridge, two sheep dogs hurting sheep, and a wheel loader in a quarry. The corgi engagement score goes from 13 to 19. On the bridge, two cars have to swap ends, and the time until both arrive falls from 71 seconds to 41. The loader reaches 90.9 out of 100 in round four

07:15 with 31 seconds to spare. Look at hurting though. The numbers on the page are 60, then 90.1, then 87.6, then 88.3. It jumps in round two and then wobbles and the blog does not discuss it. So treat these as a trend with noise, not a clean curve. Now let’s go to the repo because this is where you need to be careful. As I record this, it has 153 stars, three forks, eight commits, no releases, and an Apache 2.0 license. The readme has a checklist

07:46 checked, training code for the renderer, streaming inference, the report, and the blog. And the checkpoint on hugging phase not checked, the realtime rendering engine, and the code worlds with the agent practice loop. So, the hide-and-seek experiment itself, you cannot run today. On hugging face, the model card lists the student weights and the frozen text tower, and you need the one 2.2 VA separately. One thing to watch, the model’s license field says OpenMDW 1.1, while the repo code is Apache 2.0. The authors list the limits

08:17 themselves. Depth and normals do not capture everything. A spinning round object looks identical in a depth map. Dense geometry is redundant. And the line I keep thinking about is generating a world is the easy part. The harder question is whether the world is worth an agent’s time. So I wanted to try the idea myself. Their code worlds are not out. So I wrote a small one. It is a 12x 12 courtyard in plain python. Outer walls, one or two inner barriers, two pushable panels, a red hider, and a blue seeker

08:49 that walks a 36 cell loop around the edge. The world file has a small ray caster. For every frame, it exports metric depth and camera space normals. the two things the renderer expects. There are no textures and no assets. On the left is the untextured view, then the normals, then the depth at 832x480. The score is simple. Out of 40 steps, how many steps can the seeker not see you? Walls and panels block the line of sight. On the six layouts I held out, standing still gives about 41%. A search

09:20 that knows the seeker’s path ahead of time gets about 95. That is the gap an agent has to close. Then the real renderer. My laptop has an RTX 4060 with 8 GB and the student checkpoint alone is 7 billion parameters. So it was never going to fit. I rented one H180 GB on Runpod at 399 an hour and the whole session was about 34 minutes. Setup was clone the repo, install Torch 2.9 or later, download about 30 GB of weights from

09:51 hugging face and write one script that connects the readme streaming loop to the conditioner. The weights loaded in 3.7 seconds in the script. The loop is the one from the readme. Encode the first frame as an anchor. Then for every block of four latent frames, run the four sigas den noiseise and commit the clean block to the cache. The depth and normal latence for the next block can come from whatever your engine did in the meantime. The renderer wants one clean first frame so it knows what things look like. I made that with an image model from my untextured view.

10:22 Then I fed it 81 frames of my depth and normals. And you can see it. The walls, the two barriers, and the crates are where my code put them. Now in a stone courtyard in a small town, and the blue seeker walks its lap. But look at the red ball, the hider. It is there in the first second, and then the renderer turns it into a crate. Timing from my log, the first run is slow between 6 and8 seconds per block because the kernels compile and the CUDA

10:52 graphs get captured. After that with the cache full a block of 16 frames takes 43 seconds that is 37.4 frames per second and the report says 36.5. So that number held on my run. Now the same depth and normals with a Minecraft reference picture. And this time the hider stays a red ball all the way. So the stone run lost it and the Minecraft run kept it from the exact same geometry. A small sphere is a tiny blob in a depth map. And the paper itself says that color and identity are not fully captured by geometry. The depth and normal latence

11:24 are only 8 by 14 cells. In my first test, I tried a Halloween night and that one was the weakest because the picture I generated had a different camera angle than my geometry and the renderer was caught between the two. So the lesson is simple. The reference frame has to agree with the geometry. The last part is the evolving loop. I wrote the same round structure. red task.md play write playbook.mmd archive the agent is claude called from

11:55 the command line and it only gets a picture of the arena never coordinates in my version the agent sees one picture of the starting position and it has to commit all 40 letters at once up down left right or stay there is no correction in the middle of a game the score only arrives at the end so whatever it learns has to come from the nodes it writes between rounds my first runs used the small Claude Haiko model and it just stood still. Its playbook said don’t commit a full 40step plan until the mapping is confirmed.

12:25 That is a careful agent but it learns nothing. So I switched to Claude Sonet and that is what the numbers from here on are from then four rounds four games each. The round means were 49.4 then 60.6 then 37.5 then 50. So it is not a straight line. It is noisy but look at what the playbook says. By round four up is screen up. Left is screen left. The seeker laps the edge in about 40 steps. Never walk more than four steps and tucked behind a panel early. For a fair check, I held out six new layouts the

12:56 agent had never seen. Without playbooks, the mean was 47.5. With the round four playbooks, it was 52.1. Standing still gets 40.8, and the search gets 95. Three of the six went up, one stayed the same, and two went down. So, that is a small gain I would not bet on, and one layout carries most of it. But the agent did write down real facts about the world, the direction mapping, the seeker’s lab, and what block sight from nothing but pictures in its own results. This is the best layout, 72.5.

13:26 The hider walks two steps up, three right, and two down, and then stays from step 13 on. It is hidden for all but one step. And this is that same trajectory, depth, and normals out of my code world through the real agent garden renderer. So, what do I think? The idea of splitting state from pixels is clean and the speed claim held when I measured it myself. The renderer is genuinely usable. It took me an afternoon to wire

13:58 my own world into it. What I could not check is the main claim that agents evolve in a few rounds. The code worlds and the practice loop are not released. My small version showed only a weak effect and their numbers come from their own runs. So that part is promising but unverified from the outside. If you work on agents, the part to copy is the round structure, the task file, the parallel games, the playbook, and a fresh start from the archive. If you work on video models, the part to read is the exact replay and the

14:29 discriminator trick. And if you just want to see it, the project page has all of it. And what did it cost me? a bit over $2 of H100 time, six generated reference pictures, and about 90 cloud calls. If you want to try the renderer yourself, you need a GPU with 80 GB. So, rent one for an hour. So, in this video, we have seen what Agent Garten is, what is released, and what is not, and how I fed my own code world into their renderer. If you’re interested, you can go ahead and try this out on your own system as well, and let me know how it goes. The links are in the description.

15:01 I’ll see you in another one. Check out the next video.