AI Now Builds Its Own Training Worlds. I Rebuilt One.

Author: Prompt Engineer 48 · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: AI Now Builds Its Own Training Worlds. I Rebuilt One. · Channel: Prompt Engineer 48 · 06:13 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, AI developer tools, AI models

Summary

The video explains CompoWorld, a method for creating tool-using agent training environments by combining reusable services through dependency graphs. The presenter reviews the paper’s reported benchmark results, noting that gains vary and have not been independently reproduced. The public repository contains 20 example trajectories and a model link, but not the environment generator, so the paper cannot be fully reproduced from it. The presenter builds a small Python toy version that generates and verifies tasks, then shows how longer service chains make tasks harder.

Key points

CompoWorld uses reusable services, implemented by coding agents or simulated by a world model, and links them through dependency graphs.
The paper reports 448 services, 10,132 tools, 3,000 supervised fine-tuning trajectories, and 1,000 reinforcement-learning tasks.
The reported average gain across eight benchmarks is 9.17 absolute points, but results vary by benchmark.
Automation Bench shows the largest cited improvement, rising from 10.33 to 32.33.
The presenter did not reproduce the paper’s benchmark results; they are the authors’ reported numbers.
The repository has no license file and includes 20 example trajectories, but not the environment generator or the 448 services.
The presenter’s roughly 100-line Toy World uses a calendar, file system, and task tracker, with tools connected by a dependency graph.
Toy World checks tasks by evaluating final state and verifying that the reference plan passes while an empty plan fails.
All 60 generated tasks were verified, and a deliberately weak agent’s success rate fell as task chains grew longer.
The presenter concludes that the small-scale idea is reproducible, but full reproduction of the paper requires the missing generator code.

Chapters

Start	Chapter	Summary
00:00	CompoWorld and its approach	The presenter introduces CompoWorld’s automatically generated tasks and describes how reusable services are connected to create practice environments for tool-using agents.
01:34	Benchmark results and repository limits	The video reviews uneven benchmark gains and explains that the repository lacks the generator needed to reproduce the paper’s environments.
03:08	Inspecting examples and building Toy World	The presenter examines a legal-operations trajectory, then builds a small Python environment with three services and a dependency-graph task generator.
04:41	Verifying tasks and testing difficulty	Toy World verifies tasks against final state, and a deliberately weak agent performs worse as the number of chained steps increases.
05:42	Takeaways and conclusion	The presenter sums up the toy reproduction, reiterates the limits of the repository and benchmark claims, and notes the resource demands of training a 35-billion-parameter model.

Quotes

A script generated it, checked it, and made 60 of them in under a second on this laptop. — 00:00

So, full reproduction from this repo is not possible, and I want to say that clearly. — 02:37

The checker, it does not compare the actions. It looks at the final state. — 04:41

So, the more services you chain, the harder the task gets. — 05:12

If you want to reproduce the paper, wait for the code. — 05:42

Concepts and entities

CompoWorld, Compositional Environment Scaling for General Agents, Qwen3.6 35B A3B, Claude Opus 4.6, Toy World, Python, AutomationBench, RunPod

Full transcript (original language)

00:00 Here is a five-step task. Read a note, put the meeting on the calendar, add a task, close an old task, and write a report. Nobody wrote it. A script generated it, checked it, and made 60 of them in under a second on this laptop. This is the core idea of a new paper called CompoWorld. By the end, I will show you how I rebuilt it and where the real project stops being open. So, this is the paper, CompoWorld, Compositional Environment Scaling for General Agents, 12 authors, first submitted on 27 September, and revised

00:31 on 30 September. The problem is simple. To train an agent that uses tools, you need practice worlds with tasks and a way to check the answer. Building those by hand does not scale. So, they keep a library of reusable services. Coding agents turn tool descriptions into working services with type state. When a tool cannot be implemented, a world model simulates it. Then, a random walk over the dependency graph links services together. So, information has to flow from one service into the next.

01:02 And the numbers, if you look here, 448 services, 10,132 tools, 3,000 trajectories for supervised fine-tuning, and 1,000 tasks for reinforcement learning. The model they trained is Qwen 3.6 35B A3B. Hold on to that because I want to check what the repo actually gives us. Here is the results table from the repo. Eight benchmarks, and the average gain is 9.17 points. Mind you, these are absolute points, not not percent. Tau cubed banking goes from 10.65 to 16.49.

01:34 Deep planning, 26.04 to 35.21. Vita bench, 38.94 to 49.44. The big one is automation bench, 10.33 to 32.33. That is plus 22 points. And skills bench goes from 32.52 to 47.71. The paper says this beats Claude Opus 4.6 on automation bench. Now, let me be a skeptic for a minute. We the bench 2.0 only moves from 34.47 to 36.09, about 1.6 points, and ALE goes from 7.77 to

02:07 13.59, a better jump, but still a low score. So, the gain is real in this table, but it is uneven. And these are the authors’ own numbers. I did not reproduce them, and I cannot and I will show you why. So, let’s go ahead and open the repo. It has 22 stars, zero forks, and there is no license file. Inside, the compo world folder has a readme and one data file with 20 example trajectories, about 3.85 MB. There is no environment generator here. There are no 448 services. The

02:37 model is linked on hugging face, but the thing that builds the worlds is not in this repo. So, full reproduction from this repo is not possible, and I want to say that clearly. What we can do is read those 20 examples. I loaded them with Python. Each one uses five to seven services. For example, one legal ops task uses seat base, service M, eight, neon, ABM dev, and brevo. Each one gives the agent only four tools, and the agent has to discover the rest through a gateway. The tasks run between 15 and 48 steps,

03:08 and between 37 and 95 tool calls. The capability tags say things like cross environment constraint, sequential dependency, and conditional branching. Let me open the first one. It is a legal ops task dated 15th January, and the user asks for a full settlement disbursement run for one matter with house rules to follow and one hard constraint, do not touch anything that belongs to the closed matter. The agent does not start with a big toolbox. It starts with four generic tools, list services, list tools, describe tool, and

03:39 call tool. The first call returns 15 services, and only five of them are in the task’s service list. This run took 74 messages and 51 tool calls, and it scored 1.0. So, what can I do on my own? I can rebuild the idea small. Okay, so this is my version. It is about 100 lines of Python. It is called Toy World. And mind you, this is my toy. It is not the paper’s code. I ran it on this 8 GB RTX 4060 laptop, and it does not even touch the GPU. There are three services, a

04:11 calendar, a file system, and a task tracker. Each one has its own state and its own tools. Then we have the part from the paper, the dependency graph. Each tool says what it consumes and what it produces. A file read produces a meeting note. A meeting note feeds the calendar. Create an event feeds a task. And a task feeds a report write. A random walk through this graph is a task. Let’s run it. The generator picks a node, builds a plan with three, four, or five hops, and writes the instruction in plain English. Here is one. Read the

04:41 retro note. Put the meeting on the calendar. Add a task prep retro due on the meeting date. Mark prep demo prep complete. And write the summary to a report file. Now, the checker, it does not compare the actions. It looks at the final state. Is the event there with the right date and time? Is the task there? Is this Is the file written with the exact text? A task only counts as verified if the reference plan passes and an empty plan fails. So, a task that is already done cannot slip through. I generated 60 tasks, and all 60 were

05:12 verified. To be fair, that is by construction since I wrote the reference solver. The interesting part is difficulty. I made a deliberately dumb agent randomly skips steps on three hop tasks. It passes 60% on four hops, 40% on five hops, 15%. So, the more services you chain, the harder the task gets. That is the dial the paper is turning at a much bigger scale. If I had another week, I would swap my dumb baseline for a real model with tool calling and see where it breaks. I did not run that, so

05:42 I’m not going to claim a result. So, here’s where I land. The idea holds up. I could rebuild the small version in an afternoon and the difficulty does scale with the chain. The numbers are the authors own and the biggest jump plus 22 on automation bench is one benchmark. I would not read it as plus 22 everywhere. And the repo gives you a read me 20 trajectories and a model link but not a generator. If you want to learn how environment scaling works, build a toy like mine. If you want to reproduce the paper, wait for the code.

06:13 Training at 35 billion parameter model is not something you do on a laptop. If you’re looking for a cheap and reliable GPUs, I use RunPod and the link is in the description. So in this video we have seen what Compoworld claims, what the repo really contains and how I built a three-tool version that generates and verifies tasks. Go ahead and try it on your own system and let me know in the comments if the next one should be skill two EMV from the same repo. I’ll see you in another one.