Why AI Coding Agents Fail (And How to Fix Them)

Author: Prompt Engineering · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Why AI Coding Agents Fail (And How to Fix Them) · Channel: Prompt Engineering · 17:00 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, Cybersecurity and software security, Software engineering with AI

Summary

The video explains how to build a software factory that uses parallel coding agents to handle GitHub issues, while preventing common failures such as shared-file conflicts, duplicate work, and agents grading their own output. Its architecture uses isolated cloud sandboxes, issue claims, structured handoffs, live documentation lookup, independent verification, code review, and a human-controlled merge. In a run against 10 issues, the factory opened nine pull requests and skipped one vague issue; all nine passed hidden tests, though review caught an additional bug and seven pull requests initially had merge conflicts. An integrator stage resolved those conflicts and reran tests before a human merged the code.

Key points

A software factory is an operating loop: an agent takes an issue and writes a fix, an independent system checks the work, and a human decides whether to merge it.
Parallel agents need isolated sandboxes, and an issue claim system prevents multiple agents from working on the same issue.
Workers should not grade their own work; a fresh verifier should check out their branch and run the tests independently.
A separate reviewer can catch logic bugs that tests miss, as demonstrated by an ordering bug involving links created in the same millisecond.
Live documentation lookup helps agents check current APIs instead of relying only on training data.
Structured JSON handoffs keep stages communicating through defined data rather than messy chat histories.
A human retains control of merging; agents may open pull requests but do not make the final merge decision.
Parallel pull requests can conflict when agents edit shared files, so the factory needs an integrator to merge changes and rerun the full test suite.
In the demonstration, the factory processed 10 issues in 4 and a half minutes, opened nine pull requests, and skipped the deliberately vague issue.

Chapters

Start	Chapter	Summary
00:00	What a software factory is—and why parallel agents break	The video defines the software-factory loop and introduces the problems that arise when agents share files, claim the same issue, assess their own work, use outdated knowledge, or open too many pull requests.
03:49	Cloud sandboxes and their core primitives	The video explains how cloud boxes provide isolated environments, reusable snapshots, pause and resume, network egress controls, and remote execution for agents.
07:10	The four-stage factory pipeline	Using the Tiny Links repository, the speaker describes triage, worker, verifier, and reviewer stages, along with structured handoffs and live documentation lookup.
10:26	Running the factory and checking its results	The factory runs five parallel pipelines, uses live documentation during a worker task, and completes 10 issues by opening nine pull requests and skipping the vague one.
13:10	Integrating parallel pull requests	After seven of nine pull requests encounter merge conflicts, the speaker adds an integrator that merges changes one at a time, resolves conflicts, and reruns tests.
15:56	Lessons and final architecture	The video recaps the need for isolation, verification, review, integration, and a human merge decision, then points viewers to the factory code and documentation.

Quotes

At its core, a software factory is just an operating loop. — 00:35

If you let an agent grade its own work, its report doesn’t tell you much. — 02:12

So passing tests does not mean the code is right. — 12:36

A sandbox keeps agents from stepping on each other’s files, but it doesn’t make the code right. — 15:56

Concepts and entities

software factory, sandbox, claim system, verifier, reviewer, integrator, Upstash, Claude Code, Codex, Context7, Tiny Links, Sonnet 5

Full transcript (original language)

00:00 So everyone talks about software factories but almost nobody’s defines what that actually looks like in code. Today I am going to break down the exact architecture of a software factory and then we’re going to build one and run it on real issues. Here’s a quick example. I pointed my factory at a repository with 10 real GitHub issues. Every agent spun up inside its own cloud sandbox, wrote code using cloud code, and then created the results against a hidden test suite. At its core, a software factory is just an operating loop.

00:35 Issues come in from your backlog and a coding agent picks up one and writes the fix. Then an independent system checks the word that work and a human decides whether to merge it. The simplest version of this is a single prompt. You hand one agent one issue and tell it to fix the bug, write tests, and open a pull request. That’s already a tiny software factory. And for one ticket, it works. The problem is speed. One agent works through your backlog one ticket at

01:08 a time. So 10 issues mean waiting for 10 runs back to back. The obvious move is to run 10 agents at once in parallel. And that’s the exact moment everything kind of starts to break. Now, in this video, we’re going to look at a software factory that is going to build a tiny URL shortener. Now, using this example, I want to show you what breaks if you try to run multiple agents in parallel. So, right now, our URL shortener accepts links missing HTTPS, which causes broken

01:40 redirects. The first one is shared folder problem. If all your agents work inside the same local checkout, now all their changes piled up in the same place. In a test I ran uh three agents sharing one folder produced, one commit touching six files across three separate issues. So the fix is one isolated sandbox per agent. So every worker gets its own copy of the code and its own machine. Now second is double booking issues. So two agents will grab the

02:12 exact same issue at the same time. The fix is a claim system. Before an agent starts working, it puts a claimed label on the GitHub issue. So every other agent skips it. The next problem with parallel agents is self-grading trap. If you let an agent grade its own work, its report doesn’t tell you much. Uh, for example, in my test runs, the worker agents reported that the tests pass every single time, including once when the verifier saw the test were actually

02:44 failing. So, the fix is to use a separate verifier. Uh, you spin up a fresh sandbox that checks out the workers branch and runs test itself followed by a separate reviewer agent that reads the actual diff. Now, another problem is with the training cutoffs. A model only knows the APIs that it has seen in its training data. And the problem is that frameworks change constantly. So the fix is give the agents a live documentation tool so it

03:15 can look up current documents before writing code. The last problem is PR flood. These agents love to open pull requests. Now the fix is that the factory is only allowed to open pull requests but the merge button stays strictly with the human decision. Now if you have five of these components together you essentially have a software factory and in the rest of the video I actually want to show you an example of how all of this comes together. Now I’m

03:49 going to show you through an example. For software factory to work, you need to have longunning agents which are running in the cloud. Now, for the sandboxes, I’m using Upstash for making this video possible through their partnership. So, what actually is a box? Think of this as on demand lightweight computer in the cloud built specifically to host agents. When you spin up one via the API call, you’re not just getting a bare container where you have to wire up tools yourself. Each box

04:24 comes preconfigured with its own isolated file system, a bash shell, get node, and pre-installed agent harnesses like clot code or codeex. Or you can just drop in your own custom loop if you want. Now, here’s what it looks like in code. And there are four specific primitives here that makes a factory possible. The first one is extremely fast boot and disk snapshots. So when I call create in about 2 and 1/2 seconds I have a live Linux machine. Now in a

04:57 software factory you don’t want 10 parallel agents each running git clone and npm installed from scratch. So instead I install my dependencies once on a base box and then take snapshots. A snapshot freezes the exact state of the box disk into reusable image. From that single snapshot, uh I can boot five more boxes in parallel and every single one wakes up with those dependencies already sitting on disk. So you configure it once and then you can replicate it

05:31 multiple times. Now the second primitive is file system and pause isolation. And this is critical. For example, if I write a file inside box one and then check for it inside box two, it is not there. Each agent has its own private disk. Now the beauty is that if an agent is waiting on human input, you can actually pause the box. So it basically freezes the disk state and it stops billing compute and when you resume it, the file is still right where the agent

06:04 left it, which is great. So, it gives you the ability to isolate and pause billing when you’re waiting for human input. The next one is network egress firewalls. So, if you’re letting an agent run untrusted code or install packages, you don’t want it talking to random servers on the internet. Upstach lets you lock down network egress per box. So for example on this box I allowed listed only the npm registry. So any connection attempt to anywhere else

06:39 is going to get blocked. And this is critical from a security standpoint. For longunning agents you want true remote execution especially if they’re going to be running at a specific internal while your laptop is off. So all of this runs on upstach’s cloud servers not on my laptop. So for example, when I ask the client code inside the box where it’s running, it reports a remote Linux host with a completely different IP from my MacBook. Okay. Now with all those

07:10 primitives out, let’s look at the example that we’re going to be running. So this is Tiny Links, a small URL shortener service uh with 10 open GitHub issues. Four are bugs, five are new features, and one issue is deliberately vague. uh just to see if the factory is smart enough to walk away when the ticket doesn’t make any sense. Now to grade the software factories work, I wrote one hidden test per issue. These tests live on my machine outside the

07:40 repository. So the agents are not able to see them at all. That’s how we verify if the fixes actually work rather than uh taking agents work for it. So right now on main the app fails all six of my baseline checks and passes zero out of nine hidden tests. Okay. So here’s how the factory pipeline is wired together. First we prepare one base box. We clone tiny links install all dependencies then wire up node the GitHub token and cloud code running sonnet 5 on this and we

08:16 basically take a snapshot of this base box. So from there the factory claims each GitHub issue and the issue moves through our four distinct stages. Now we have three brand new boxes each booted from the snapshot that we just created. So stage one is triage. A fresh box wakes up reads the GitHub issue and decides whether the requirements are clear enough to build or if it should reject the ticket. Stage two is the worker. If triage approves, a fresh

08:50 worker box writes the code fix and then adds unit tests and the factory pushes its branch. Then stage three is the verifier. So we never trust the workers test report. A fresh verifier box boots from the clean snapshot, then checks out the workers branch cold and runs the test suite itself. And then we have stage four which is the reviewer. So inside the same verifier box reviewer

09:21 agent inspects the actual git diff line by line to catch logic bugs that unit tests miss. Only if both the tests and the reviewer sign off then the factory opens a pull request. Okay. In order to make it reliable we have two architectural details that I want to talk about. The first one is typed object handoff. So uh stages don’t pass messy chat histories to each other. They communicate strictly through structured JSON objects. The worker is forced to

09:53 return a fixed schema which includes its status, a chart summary, the exact list of files it’s changed and whether it added tests and a boolean for whether the test pass. Then we have live docs via proxy key injection to solve the stale training data problem. I integrated context 7 which searches live official documentation via a single web request. Now I put a tiny doc cli command inside every box and instructed the worker to call it whenever it is not

10:26 100% sure about an API. And notice how the security works. The context 7 API key is attached by upstach outside the box on the outbound request. So the agent inside the box can read the docs but never actually sees the API key. Okay. So now let’s run the factory across 10 issues. Here we’ll have five parallel pipelines at a time. So on the left side of the screen you see the live events logs from each stage. On the right you can watch the sandbox spinning

11:00 up and tearing down inside the upstaches console. on GitHub. The issue picks up the claim label one by one. Now, let’s jump inside one of the live workers boxes while it’s working. So, here the agent isn’t sure about a node API. So, you can see it runs our docs command to check how URL search parameters behave in the current node before editing the file. And here’s the worker handling our HTTPS bug. Um so it reads the existing URL validation code, rewrites the check

11:34 to passse the URL object and strictly enforces HTTP or HTTPS protocols, then writes new unit tests, runs them and the branch gets committed and pushed. Okay, so the entire 10 issue run finished in 4 and 1/2 minutes. The factory opened ninepull requests and deliberately skipped the one wake issue which is the exact behavior we wanted. And when I ran my external test suite against those

12:04 nine branches, all nine pass the hidden test. Now remember earlier uh when I said you need both a verifier and a reviewer. So let me show you one of the traces where you actually see why you need them. So on that run the worker said it its tests passed. The independent verifier ran the test suite and it passed. Even my hidden evaluation test passed. But the reviewer agent still rejected the pull request. So when the reviewer read the diff, it realized

12:36 that if two links were created inside the exact same millisecond, the link list came back in the wrong order and the reviewer reproduced the bug. The workers unit test never caught that specific bug because the test data used pre-sorted slugs. So passing tests does not mean the code is right. having a dedicated code review agent separated from the worker caught an ordering bug that every other test missed. And that’s why you want to have this independent

13:10 verifier and reviewer. Now, here’s the most interesting part. So, all the nine verified pull requests work perfectly on its own. But when I tried to merge them into the main, seven out of the nine immediately through merge conflicts. Why? because all nine agents started from the exact same base snapshot and most of them added their own unit tests to the exact test file. Now the problem is that none of the agents knew the other eight existed. So I had to add a

13:42 fifth stage to the factory, the integrator. So the integrator is one more upstatch box that pulls the approved PRs one at a time into a unified integration branch. Whenever get merge hits a conflict, a clot code agent inside the integrator box opens the conflict marker combines both sides so no code or test gets lost and then reruns the full test suite after every single merge to make sure the combined code still works. Now in our case all uh

14:17 nine PRs went in through the integrator to merge cleanly and the agent resolve the other seven ones and the final step is a human decision maker at the merge step. So here’s the pull request for our HTTPS bug. I can see the verifier passed and the uh reviewer also signed off. So I can simply click merge. So right after that single merge main properly rejects links missing HTTPS.

14:47 Okay. So let’s talk about what this actually cost us. So factory run where we had 10 issues across four stages. The total cost was about 3.67 in cloud code token usage. the integrator stage which resolved seven conflicts and was running the suite. It was $6.37 in token usage. Now, upstute box, it’s about $.01 for the entire day, including all my test runs. Now, because uh coding

15:24 agent spends almost all of their time waiting for the LLM API to stream tokens back, and Upstash only builds for active CPU time, the sandbox computer is basically a rounding error at this point. Now, one heads up. Upstach includes built-in model usage up to a monthly cap. So for heavy factory runs, you will want to pass in your own API key or increase that monthly cap. Now, if you want to build your own software factory, a few things to keep in mind. A

15:56 sandbox keeps agents from stepping on each other’s files, but it doesn’t make the code right. Code quality still comes from your tests, your verifier, and your reviewer. Second, parallel agents can create git conflicts when they touch shared files like test suites or routers. Plan for an integrated step. So to recap, a software factory is just a loop where issues comes in. An agent builds the fix, a separate box verifies it, and a human merges it. And every

16:27 piece of the architecture is there because parallel agents break without it. You want to keep one snapshot box per agent. uh an issue claim lock an independent verifier live documentation lookup and integrator to merge conflicts and then a human at the end. I have linked the full um factory code and the stash box documents down in the description if you can so you can try this on your own repos. Let me know in the comments how you are running codeing

17:00 agents in parallel and what actually breaks. Anyways, I hope you found this video useful. Thanks for watching and as always, see you in the next one.