You Need This New Security Skill from Cloudflare

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: You Need This New Security Skill from Cloudflare · Channel: Rob Shocks · 10:54 · Watch on YouTube

Topics: AI agents, Cybersecurity and software security, Software engineering with AI

Summary

Rob tests Cloudflare’s open-source security skill against free scanners, a built-in security review, and a plain AI prompt using a deliberately vulnerable app. The tests show that scanners and AI reviews catch different kinds of problems: scanners found issues in Git history and dependencies, while AI tools found logic flaws. Cloudflare’s skill found all six planted bugs in this small test, but its fuller investigations can take time and cost money. The video recommends three levels of security checks, from basic scans to sandboxed audits and professional testing before launching critical features.

Key points

The test app contained six planted vulnerabilities: SQL injection, a leaked Git-history key, an outdated Next.js version, cross-company data access, unrestricted deletion, and an export that exposed all companies’ data.
A decoy that looked dangerous but was safe was included to check for false positives.
GitLeaks found the deleted key in Git history, while Trivy detected the vulnerable Next.js version; neither scanner read the application logic.
The built-in security review found four of the six planted bugs, but it reviews changes and may not scan an entire application unless asked.
Cloudflare’s skill maps the app and uses separate agents to hunt for and verify vulnerabilities; in this small test, it named all six bugs but did not run verifiers.
A plain prompt found five of the six bugs in the small app, missing the key because it did not inspect Git history.
All three AI tools also flagged an unintended hardcoded session password, illustrating why layered security checks matter.
The video recommends using GitLeaks and dependency scanning, reviewing changes before shipping, and reserving deeper sandboxed audits or professional testing for launches and critical changes.

Chapters

Start	Chapter	Summary
00:00	Why AI-written code needs security checks	Rob introduces Cloudflare’s new security skill and explains why AI coding agents can overlook data-ownership and access-control issues.
01:33	The vulnerable test app and free scanners	He describes the six planted bugs and decoy, then tests GitLeaks and Trivy against the app.
03:37	Comparing AI security reviews	The built-in review catches four bugs, while Cloudflare’s skill identifies all six in the small test and explains its verification process.
06:45	Plain-prompt results and test findings	A short prompt finds five bugs, and Rob compares the tools while emphasizing that scanners and AI reviews catch different issues.
08:48	Three levels of security practice	Rob recommends basic repository and dependency checks, per-PR reviews, and deeper sandboxed audits or professional testing before critical launches.

Quotes

Security is your problem, your responsibility, it’s not the tool, it’s not the agent. — 01:02

Agents are not good at grading their own work. — 04:40

Review the code base for security vulnerabilities, name the file, and what an attacker gets. — 06:45

scanners and then AI review tools catch different things and you need to have both. — 08:17

Concepts and entities

Cloudflare, GitLeaks, Trivy, Cloud Code, Leak Cast, Next.js, Lovable, Opus 5.5, Dependabot, Mozilla Observatory

Full transcript (original language)

00:00 So, Cloudflare just launched a new open-source security skill and they use it on their own software factory. And to be honest, it’s just in time. Hacking has become so much easier now with AI models and reverse engineering. In the last couple of days alone, we’ve had some serious breaches and it feels like it’s just speeding up. So, the timing couldn’t be any better for Cloudflare’s security skills. So, I built an app with six security holes in it on purpose. So, I did this basically to test out Cloudflare’s new open-source skill and then I put it up against some free scanners, a security review prompt

00:31 that’s already inside Cloud Code, and then a plain prompt with no skill at all. I’m Rob from Suite Dimension and by the end of this, you’ll know what each of these is good for and I’ll give you three different levels of security you can set up to protect yourself this week. So, why does all this matter now? We’re all using AI agents to code and they are pretty good at the textbook stuff. Ask Cloud Code for a search endpoint and it’s going to troll some parameters on a query. Where they tend to fall down and get a little bit lazy is the question of whose data is this actually. The agent will check that you’re logged in, but will forget to check that the actual record belongs to

01:02 you, simple row-level security. So, last year a researcher found more than 170 apps built with Lovable that were all leaking data because nobody had set up the database rules that say who can see which rows. To be fair to Lovable in that scenario, they say it’s down to each customer who owns their apps to manage the security and that is absolutely true. Security is your problem, your responsibility, it’s not the tool, it’s not the agent. So, going into this test, my prediction would be that the traditional scanners would catch the classic bugs and then you’d need something that was able to read

01:33 your code like an agent to pick out the bugs that the AI would write. And I was kind of half right. So, I built this little app called Leak Cast. It’s small, so it’s not an extensive test, but it’s a deliberately broken clone of another app. It’s using Next.js with a database and two companies in there, Acme and Globex. Each one of these has their own users and posts. So, three of the bugs are classics, SQL injection in the search, a secret key I committed and then deleted, and then an old Next.js version with a nasty known bug. And then three are the kind an AI agent would

02:05 write. So, Acme can read Globex’s posts, anyone can delete any post, and the export dumps every company’s data. And then I also put in one decoy code that looks dangerous, but it’s actually safe. And any tool that flags it loses a point. A lot of the problems with these code checkers is they raise a lot of false positives that can be actually quite time-consuming. I made sure to write an exploit for every bug and that everyone worked. And then each tool got its own sandbox, so it couldn’t see what anyone else was doing. Okay, so round one is the free stuff. So,

02:35 GitLeaks looks through your whole Git history for keys you’ve committed, even ones that you might delete later. So, it successfully found a key I’d removed three commits ago in under a second. Remember that deleting a key doesn’t take it out of your history, and that’s how a lot of keys are leaked. So, Trivy will check all your dependencies, your node modules, etc. against known vulnerabilities. So, it instantly flags our old Next.js version, which has a middleware bypass. It actually flagged 42 things here, but I’m only scoring it on the one I planted. Now, bear in mind, if your only authentication check is

03:07 middleware, one extra header will get you past it. Watch. You notice here Trivy told me the package is bad, but it didn’t tell me my admin page depends on middleware alone, which is half the bug. This is why you would need an agent as a scanner. I expected the scanners to catch the SQL injection, and neither of them did. GitLeaks is only looking for secrets, and Trivy only looks at a dependency list, but neither of them is going to actually read your code, which is needed. So, both of these only grabbed two of the six issues. Okay, so round two, let’s look at Cloud Code’s

03:37 built-in review. So, whether using CodeX, Cloud Code, Cursor, they all have one of these security review skills. You type {slash} security review, and it checks your changes. Now, the catch here is it reviews what has just changed. So, to be fair, I gave it the whole app as one change, which isn’t how you’d normally use it. If you step into a full-blown app and run security review, it’s only going to check the latest diff. It won’t actually do a full scan of the whole application, unless you ask it to. So, security review got four of the six, the SQL injection, the

04:09 cross-company read, the delete, and the export. Took about 84 seconds and 16 cents. It missed the middleware bypass. It also missed the key, but that’s probably my fault. It didn’t actually see the Git history. It also skips rate limiting by design, and the docs say that it’s not hardened against prompt injection. So, if you’re running this on open-source PRs or other people’s PRs, just be careful. Okay, so then Cloudflare’s security skill. So, this one is getting lots of traction on GitHub. It’s pretty simple. It’s just one command to install, and then you ask your agent for a full security audit.

04:40 So, this Cloudflare setup is different in many ways. What’s really different, I guess, is its process. So, first thing it does is it goes and maps the app, and then it sends what it calls a hunter against each kind of attack. And then, if an agent does find a bug, it’s never the one that actually checks it. A separate agent will do that. And this is a pretty good step. We’ve seen this before. Agents are not good at grading their own work. Cloudflare has put it perfectly here. If a hunter is allowed to grade its own homework, it will confidently validate everything it outputs. This is why I’ll often go as far to use a completely different model

05:11 when checking another agent’s work. Okay, so it loaded the skill, it went and it mapped the app, and then it wrote out every phase, and then it did all the hunting itself. In our small test, it started zero hunters against zero verifiers. To its credit, it marked its own run as incomplete. Because our app was so small here, it didn’t require a full investigation. I didn’t take the time to do this on a larger app, but the idea here is that it will fan out multiple agents. Haven’t tested it personally, so don’t hold me on that one. So, even though we just had a small run here, it still named all the six

05:43 bugs. It’s the only tool we tested here that got all of them, including the key in the Git history. Took about 5 minutes and 38 cents, and it even went and proved one of the bugs, the SQL injection on a dummy database. The other five came back then with needs validation. So, that’s not a miss on Cloudflare’s part. In this case, it just wouldn’t run because in order to prove those bugs, it needed dedicated lockdown sandboxes. And for this test, I haven’t set that up. Now, it’s worth bearing in mind, at scale, this scale is going to cost you money. So, 38 cents sounds cheap, but that’s because it didn’t run

06:14 to its full extent. Proving out each one of these bugs in separate sandboxes will add up. And then Cloudflare themselves say a full run only finds about half of the bugs. So, usually it needs to be run several times in order to be thorough. And that’s going to add up in terms of cost. So, you’ll need to weigh up how often you want to run this. It’s going to take time, and it’s going to cost you. It’s not something you might run on every single pull request. So, there’s one more side to this test. I wanted a control, basically just the same model and all the skills turned off, and basically one sentence. Because these

06:45 models are getting really good now. Do they need all this help? So, review the code base for security vulnerabilities, name the file, and what an attacker gets. If you’re going to take one thing from this video, just remember this one prompt. And while you’re pausing it to remember it, just hit the like and subscribe button there. It’s a big help to the channel. So, this run took about 34 seconds, it cost under a cent, and it found five out of the six bugs. In fairness, it only missed the Git key because it never looked at the Git history, something that might not be obvious. And it also called the decoy

07:15 safe. Now, before you screenshot that, this is a tiny app the model can read in one go, and three of the bugs are pretty much the same mistake. When it comes to larger applications, having something like a Cloudflare skills or a series of agents that fan out and work in parallel is going to be a must. What I didn’t expect was all three tools flagged a bug that I didn’t intend to plant. Their session password is hardcoded in in source, and the AI agent I used to help build this wrote it in that way. With that, anyone with the code can forge the

07:47 admin login and Git Leaks missed it. It’s just a reminder of how much we need multiple security checks and layers in our systems. It’s actually worth checking out my last video on Anthropic’s approach to review and security review. You find that on the main channel. So, this is kind of the silly stuff an agent will do. The agent got the textbook parts right and it still went and hardcoded the password that protects every session. And this was Opus 5.5. So, okay. So, here’s the full board. It shows found and proved for each tool with time and cost along

08:17 the bottom. So, as expected, the scanners caught what sits outside your code, your Git history, and your dependency list. Every AI tool caught the logic bugs. And the only tool that attempted to prove anything was Cloudflare because it’s built to do that. Even though it did skip the verifiers. And the decoy, nobody flagged it as an issue. But in fairness, it was an easy decoy. Didn’t have time to write something more complicated. So, remember in all cases, it was just one run on one small model. And the key thing you need to understand is scanners and then AI review tools catch different things and you need to have both. Now, none of these tools actually attacked the

08:48 running app. I did that myself with my own exploit scripts, which is how I know the answers. Tools that attack your app are a whole different category. If you want to see a video on that, let me know in the comments. So, what should you do? Here are three levels. So, level one is pretty simple. What you want to do is run Git Leaks on your whole history of your repository. Just to check if there’s any keys in there that shouldn’t be there. And then ask your agent to set up a hook so that Git Leaks is run on every pre-commit. So, that means any new code going up is going to get scanned. Then put something like Dependabot or

09:19 Trivy on all your dependencies. Dependencies and modules at any given point have some kind of a vulnerability appear and you want to be scanning to make sure that it’s not going to affect your app. Then before you ship, just ask your agent that one plain sentence to review or use these security review skill to check all the changes you’ve made before they go live. If you’re seeing that your agent repeatedly makes the same kind of security mistakes, then you bake that into your agent.md or Claude.md file. Maybe a rule that tells the agent to always scope every query to the organization. So, level two is for

09:51 every pull request. That’s where /security review belongs because reviewing changes is basically what it’s for. So, you could run Claude’s security review on every trusted PR, then GitLeaks and Trivy in your continuous integration, maybe with GitHub Actions, and then quick header checks with something like Mozilla Observatory. Level three then is before you launch something or whenever you touch something that’s critical like auth or billing. So, you might run Cloudflare skill in a sandbox where it can run your code. And then you want to check it’s run to make sure that the verifiers actually did run if it’s a big enough

10:22 app. And according to Cloudflare, you want to do this a couple of times to make sure that you’ve caught everything. Then you might point an AI pen tester at your staging server, not your production server. If you want to be thorough and get this absolutely right, pay a professional human to do this for you. A professional human will bring a big picture approach here that an agent can’t offer you. So, for most of us, honestly, level one this week, level two this month, and then level three before you’re starting to take payments in your app. If you want to build apps like this properly with AI, that’s what we do in the Switch Dimension Agent Accelerator.

10:54 The link is in the description down below. I’d love to know in the comments, what is the worst security bug you found in AI-written code? And I do read all the comments. And if this helped, please subscribe. If you want to see how Anthropic handle review when Claude code writes 90% of the code, watch this one next.