I Tested 4 Codex Frameworks on the Same Real Feature
Read the Portuguese translation →
Author: The Gray Cat · Watch on YouTube
Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.
Original language: English
Original video: I Tested 4 Codex Frameworks on the Same Real Feature · Channel: The Gray Cat · 09:47 · Watch on YouTube
Topics: AI agents, Software engineering with AI, AI content generation, AI models
Summary
Section titled “Summary”The video compares five Codex setups on the same feature for The Gray Cut: plain Codex, pstack (called Poteto in the transcript), Compound Engineering, Matt Pocock’s implement spec skill, and Superpowers. The feature uses AI to match screencast footage to spoken commentary and create an editable Premiere timeline. The longer framework runs generally produced stronger starting points, though every implementation had issues; Superpowers’ version received the best review and was chosen as the base. After further testing and bug fixes, the feature was merged into the main app.
Key points
Section titled “Key points”- Five isolated Codex setups received the same starting code and feature task; Claude Opus answered framework questions and reviewed proposed next steps.
- The feature aims to transcribe commentary, give the model visual context from the screen, and select matching B-roll for an editable Premiere timeline.
- Poteto finished in about 27 minutes and plain Codex in about 35 minutes, but both implementations needed substantial repairs.
- Compound Engineering took about an hour and a half, used 26 subagents, and kept narration aligned with the correct script sections, but had an overlapping-clips problem.
- Matt Pocock’s workflow produced 12 tickets and took about two and a half hours; its implementation correctly replaced only the covered portion of an existing screen clip.
- Superpowers took nearly four and a half hours and used 34 subagents; its implementation was rated the best starting point and handled retries by replacing AI-picked footage while preserving manually placed clips.
- The longer runs produced stronger foundations, but still missed requirements, including leaving visual notes out of the AI context; all five also miscounted screen clips as retained talking-head footage.
- The creator used Superpowers’ version as the base, fixed bugs, completed shipping checks, and merged the feature into main.
Chapters
Section titled “Chapters”| Start | Chapter | Summary |
|---|---|---|
| 00:00 | The comparison and the feature | The creator introduces five Codex setups and explains the goal of using AI to match screen footage to spoken commentary. |
| 01:42 | Quick runs: Poteto and plain Codex | Poteto and plain Codex finish first, but code review finds important editing problems in both implementations. |
| 03:24 | Compound Engineering and Matt Pocock | The video covers Compound Engineering’s results, then details the spec-and-ticket workflow used for Pocock’s skill. |
| 06:02 | Superpowers and the results | Superpowers takes the longest and receives the best review, while the creator compares run times, token use, and remaining issues. |
| 08:44 | Testing and choosing a base | The creator tests and fixes Superpowers’ implementation, merges the feature, and names it the preferred framework for this task. |
Quotes
Section titled “Quotes”The full run took nearly four and a half hours. — 00:00
Both quick versions mixed sections up. — 03:24
The review recommended its implementation as the best starting point — 07:07
The longer framework runs gave me stronger foundations to work with. — 08:44
The extra effort paid off here. — 09:47
Concepts and entities
Section titled “Concepts and entities”Codex, pstack, Compound Engineering, implement spec, Superpowers, GPT-6.1 Sol, Claude Opus 5.5, The Gray Cut, Premiere, Every
Full transcript (original language)
Section titled “Full transcript (original language)”00:00 Superpowers 7 was still working when I went to sleep Poteto had finished in 27 minutes. The full run took nearly four and a half hours. If you remember my infamous BMAD tests, those took whole weekends and still left me disappointed. This time, the longer runs produced stronger implementations You asked for it, so this is my first framework comparison in Codex. I tried Matt Pocock’s skills and pstack. Compound Engineering and Superpowers joined them, alongside plain Codex.
00:34 I’ll show you where that extra time went, and which implementation I kept for my app. Okay, so The Gray Cut already cuts my talking head footage more or less reliably. But finding the right screen footage still takes hours in Premiere. I scrub through long demos, then adjust the speed to fit what I’m explaining. I’d rather spend that time trying more tools. The app already has an LLM integration.
01:07 I wanted it to transcribe my screencast commentary and let the model see what’s happening on with screen. With that context, it could pick the right B roll for what I’m saying. I’d get an editable timeline in Premiere, so I could still fix any bad choices. I brainstormed the feature with Claude, then had it prepare prompts for Claude Design. The designs looked pretty decent, so I exported the HTML for the coding agents. For the comparison, I used five isolated Codex setups with the same starting code.
01:42 Every coding agent used GPT 6.1 Sol at medium effort. Plain got the prompt alone, without plan mode. I’ll leave that prompt on screen. Claude Opus 5.5 ran the experiment at extra high effort. It answered the frameworks’ questions from my original design and approved their next steps. I was pretty happy with it. It pushed back on proposals that changed my original direction. The app lists the video files I’ve added to my project.
02:14 Compound wanted to remove a video from that list and keep its clips on the timeline. Opus corrected that. Removing a video from the project should remove its timeline clips too. The original file stays on disk. That kept the sessions moving while I stepped away. Poteto was the first to finish. This is pstack, Lauren Tan’s collection of engineering skills. I’m using Michael Denyer’s port for Codex. The skills cover design and code review, with rules for keeping changes small and checking the result.
02:48 There’s quite a lot in there, but the entry point is simple. I paste the task into Poteto mode, and it chooses the workflow. For this feature, it used an architecture skill to plan the changes. It ran five subagents and asked no questions About 27 minutes later, it reported the work finished. Plain Codex took about 35 minutes, without any framework. But the code review found problems in both implementations. Once screen clips were placed, Poteto’s version blocked changes to the length of the talking head cut.
03:24 That would get annoying very quickly. I wanted to keep editing the talking head and leave the screen clips where I’d placed them. Plain refused to place one screen clip over another. It should replace only the covered part. So these were the quick results, but both still needed substantial work before I’d use either implementation. Compound Engineering comes from Every. Its workflow starts with brainstorming and planning. Then the work skill builds the feature, with simplification and code review included.
03:56 This run took about an hour and a half and used 26 subagents. The compound skill saves useful lessons for future work. Future sessions can use those notes instead of working everything out again. Here, it reported Documentation skipped Its rules allow that when there’s no useful lesson to add. In the review, it kept narration with the right script section. Both quick versions mixed sections up. But it had the same overlap problem as Plain.
04:28 Compound refused to place one screen clip over another. I’ve been an Every subscriber for a while, and they recently introduced the Builder Pack. I can’t really justify buying it, because I wouldn’t use most of those tools. But the Cursor subscription would be useful for the videos you keep asking for. So if someone from Every is watching, I wouldn’t mind a discount. I’d like to make more videos about Cursor and Grok 4.7. And if these comparisons help, subscribe for the next one.
04:59 Join Discord if you want to suggest a tool or discuss the results. Matt Pocock’s new implement spec skill handles the whole set of tickets. In my previous test, I called the implement skill for each ticket, or used my own automation. Opus played a similar role in this experiment. For this run, I wanted grilling to lead into a written spec. The spec would become tickets for the new skill to work through. The first attempt started coding straight after grilling.
05:29 My setup left that handoff unclear, so I stopped it and restarted. The replacement went through 28 questions, with Opus answering from my original design. It produced 12 tickets, and I redirected it into the new skill. Each ticket got its own worktree. That’s a separate copy of the code for that task, later merged into one branch. My initial rules banned extra worktrees, so I allowed an exception for this run. It took about two and a half hours, with 16 subagents.
06:02 The review put its implementation second. Dropping a new screen clip over an existing one replaces only the part it covers. The rest stays in place. The review also found that rearranging sections could put screen footage under the wrong topic. I wouldn’t normally do that with a prepared script. Superpowers was the last to finish. I tested version seven, which was only three days old. The biggest change is the rebuilt brainstorming skill. It adjusts the amount of planning to the size of the job.
06:35 For a larger feature, it writes a design. Another agent checks that design for questions still unanswered before implementation. In this run, it asked seven questions and produced an eleven task plan. Opus answered the questions and sent the first design back for corrections before approving it. The plan offered different execution routes. Opus followed my test instructions and chose subagent driven development. Agents implemented the tasks, with separate agents reviewing their work.
07:07 It used 34 subagents in total. I left it working and went to sleep. The full run took nearly four and a half hours. And the review recommended its implementation as the best starting point Like Pocock, it handled replacing the middle of a screen clip correctly. The retry behavior was useful too. If I ask the AI to try a section again, it replaces its earlier B roll choices. Clips I placed myself stay where I put them.
07:39 Compared with my BMAD weekends, this gave me much more to work with. There were still bugs to fix before I could use it for a real video. The time chart goes from 27 minutes to nearly four and a half hours. These spans include waiting, with all five runs sharing one machine and ChatGPT plan. Plain used the fewest tokens. Poteto finished sooner, but used about two and a half times as many. Pocock used sixteen subagents, compared with twenty-six for Compound.
08:11 Yet Pocock consumed more than twice the tokens. So agent count alone doesn’t explain the usage. Most of these totals are cached input, counted again each time it’s reused. Using the saved API rates, that’s about two dollars for Plain and twenty-five for Superpowers. I used a ChatGPT subscription, so those dollar figures are for comparison. The review put Superpowers first, with Pocock behind it. Compound followed, while the two quick implementations needed substantial repair.
08:44 That’s a judgment of the code produced in this test. The longer framework runs gave me stronger foundations to work with. The stronger versions still missed requirements. They left my script’s visual notes out of the AI’s context. All five also got the talking head check wrong. Screen clips were counted as talking head footage the app had kept. I took Superpowers’ version as the base and tested it further. During that work, I tried it on material from my Architecture Tools video.
09:15 The first attempt was pretty underwhelming, but the second was much closer to what I wanted. It picked relevant screen footage and even adjusted the playback speed to fit my explanation. That’s the part I usually spend hours doing manually in Premiere. Seeing it come together like this was pretty impressive. I fixed bugs that came up in testing and ran my usual shipping checks. Then I merged the feature into main. I’m choosing Superpowers again. They don’t sponsor me, and I don’t use it in my daily work.
09:47 For a feature this size, it’s my pick in Codex. The extra effort paid off here. But again, the models and framework versions have changed since my Claude Code tests. This was a different task too. I’ll edit this video with the new feature. I can report back in the comments or Discord on how well it worked. This time, I slept while the subagents worked. Apparently, my gray cat has trained me well. For the Claude Code comparison, watch my earlier Superpowers versus Matt Pocock test.