How Anthropic ACTUALLY Reviews AI Code (30,000 Changes a Day)

Author: Hyperautomation Labs · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: How Anthropic ACTUALLY Reviews AI Code (30,000 Changes a Day) · Channel: Hyperautomation Labs · 12:22 · Watch on YouTube

Topics: AI agents, Task and workflow automation, Cybersecurity and software security, Claude Code, Software engineering with AI

Summary

Anthropic’s Claude Code team scaled daily code changes from about 2,500 to more than 30,000 while more than doubling merged pull requests and reducing rollbacks by over 40%. Rather than treating human review as a requirement to read every line, the team asks what could go wrong and uses the appropriate automated or human check to catch it. Their process combines agent reviews, security checks, automated tests, risk-based approvals, merge queues, and incident response. The video also describes ways they limit unproductive AI review loops, keep reviewers accountable, and focus human attention on decisions that require broader context.

Key points

Anthropic treats the question “Did a human read the code?” as an imprecise proxy and instead asks what could go wrong and which check can catch it.
Claude is given runnable checks such as tests, builds, or screenshots, and must show the results; a separate agent then reviews the change in a fresh context.
Repeated review comments can be turned into project-specific lint rules, skills, or guidance in Claude.md and Review.md.
Cloud review uses multiple agents to identify bugs and a separate group to vote on findings; style-only comments are filtered out.
Security review, linting, type checks, tests, builds, and a pull request template provide additional safeguards.
Review requirements depend on risk: low-risk changes can need no human approval, medium-risk changes need one, and high-risk changes need two.
The merge queue reruns checks against the latest code, while flags and incident reviews help catch and respond to issues after merging.
To limit unproductive review loops, Anthropic caps review rounds, focuses later passes on new changes, and avoids repeating findings.
Past incidents and reverts become test cases for evaluating reviewer changes, while comment outcomes help measure review noise.
Humans focus on product choices, risky code, architecture, evidence, and accountability; the person who directs a change owns its outcome.

Chapters

Start	Chapter	Summary
00:01	Why code review needed to change	The video explains how code volume grew at Anthropic and why the team replaced line-by-line reading as the goal with checks aimed at identifying what could go wrong.
02:35	The review layers begin	A bug fix passes through agent-run checks, a fresh-context review, project-specific guidance, and a multi-agent cloud review that filters and prioritizes findings.
05:08	Security, tests, and risk-based approvals	The process adds blocking security review, routine automated checks, risk tiers for human approvals, and a merge queue that tests changes against the latest code.
07:43	Preventing endless AI review rounds	After describing post-merge safeguards, the video explains how Anthropic reduced repetitive AI review by limiting rounds and focusing follow-up reviews on new changes.
09:46	Human responsibilities and practical steps	The video outlines the decisions humans still own and gives teams a sequence of practical steps for adopting the review process.

Quotes

The better question is this, what could go wrong with this change? And which check, automated or human, is the right one to catch it? — 01:34

The goal is that most pull requests merge with at most one human in the loop, who can be the author — 02:04

Boris Cherny, who created Claude code, calls this probably the most important thing — 03:05

You land it, you own it. — 10:50

What could go wrong here, and what’s the right check to catch it? — 12:22

Concepts and entities

Anthropic, Claude Code, Boris Cherny, Revisão adversarial por agentes, Níveis de risco, Fila de merge, Conjunto de testes do reviewer, Flags

Full transcript (original language)

00:01 Anthropic’s Claude code team stopped reading every line of code >> [music] >> and their rollbacks fell by 40%. Here’s the problem they solved. It’s Monday morning. 14 pull requests are waiting for you. Half of them were written by an AI agent while you slept. You open the first one. You leave three little comments, rename this, add a space so it looks like you read it. And you click approve. Anthropic’s own engineers know this move. >> You would have someone go and review the

00:32 the code and they would like pick out like, you know, three little nitpicks to post. And it was kind of just like, I’m signaling to you that I read this code. >> Yeah. >> Here’s why that matters. Inside Anthropic, code changes landing every day went from about 2 and 1/2 thousand in May to more than 30,000 in September. 12 times more code. No human can read that. And yet, on the Claude code team, merged pull requests more than doubled while the share that

01:03 had to be rolled back fell by more than 40%. More code, fewer mistakes. This week, they published exactly how they do it. It’s seven layers plus one problem they created themselves that ran 50 rounds overnight. By the end of this video, you can copy every piece of it. First, they threw out the question everyone asks, did a human read the code? They call it an imprecise proxy. The better question is this, what could go wrong with this

01:34 change? And which check, automated or human, is the right one to catch it? That one question changes when review happens. It used to be a gate at the end of the road. Write the code, open the pull request, wait for a colleague. Now, review runs all the time, like spellcheck while you type. Most changes have been reviewed and fixed by Claude at least once before a human is even asked to look. And they’re clear about what they’re protecting. Human attention. Review costs time twice.

02:04 The time people spend reading, and the time they spend fixing what bad code breaks. If you cut the first, and the second goes up, you saved nothing. You just moved the time. So, the goal is that most pull requests merge with at most one human in the loop, who can be the author, >> [music] >> with no rise in reverts or incidents. That revert number is the guardrail. Keep your eye on it. So, let’s follow one change through the whole system. Say an engineer asks

02:35 Claude to fix a bug in the auto updater, the code that updates Claude code on your machine. Picture the system as slices of Swiss cheese. >> [music] >> Every layer has holes. Each one catches some problems and misses others, but very few bugs make it through all of them, and the ones that do get rolled back fast. Layer one happens before a pull request even exists. Claude gets something it can run, like tests, a build, or a screenshot to compare against the design. And it has

03:05 to show the output instead of just saying it’s done. Then a second agent reviews the change in a fresh context with none of the first agent’s assumptions. Boris Cherny, who created Claude code, calls this probably the most important thing, and estimates it makes the result two to three times better. >> When Claude creates a PR for me for Claude code, >> Yes. >> uh it tests it, it sends me screenshots for what it would look like, uh and then you kind of get confidence. >> And here’s the clever part. When the reviewer keeps flagging the

03:36 same mistake, they don’t keep fixing it by hand. They fix it at the source with a lint rule, a skill, or two small files. Claude.md tells Claude what the project is. Review.md, written by the people who own that code, tells the reviewer what matters there. The good ones say what failure looks like in that repo. For docs, failure is it lies. For config, it’s it misroutes. They cap the nitpicks, list paths to skip, and demand a file and line number as

04:07 evidence, not a guess. Layer two. Every pull request gets a cloud review before a person looks. It reads the change against the whole repo and asks three things. Does it solve the problem? Does it fit the system? Does it break something else? Here’s the part that surprised me. It isn’t one AI reviewer. Several agents hunt for bugs at the same time. Then a separate group votes on every finding. And a finding only gets posted if enough of them agree it’s real.

04:38 >> For each bug, you might do an adversarial review where you ask it look at the bug from three different, you know, opinions or perspectives and see if the bug is actually real. What that does is like it filters out all these bugs for you and gives you just the the most important ones that require or need your attention. >> Findings that are only about style get thrown out. What’s left shows up as comments sorted by severity. Each tagged important, knit, or pre-existing. So [music] a bug

05:08 the author didn’t create doesn’t look like their fault. When they launched it internally, the share of pull requests getting real review comments jumped from 16% to 54. And every finding has to be answered. Fix it or close it with a reason. Often an agent picks up the fix overnight. So you wake up to a pull request that’s already been through another round. Layer three is a separate security review and it can block. A pull request can’t merge while any

05:38 finding, bug, or security is still open. Layer four is the boring part that still matters. Lint, type checks, tests on every platform, and a build. A typical change runs dozens of checks before anyone looks at it. And their pull request template asks two questions every team should steal. How was this tested? And how do we undo it? Layer five decides how much of your time a change gets. >> [music] >> A typo fix in the docs and a change to the auto updater are not the same risk,

06:09 so they don’t get the same review. Low risk, like docs, copy, or things hidden behind a flag, needs zero human approvals once the automated reviews are green. Medium, meaning ordinary product changes plus sensitive areas like caching, the auto updater, and system prompts, needs one person. High risk needs two. Four rules keep this safe. One, the change sets the tier, not the author. A classifier reads the change and picks the tier.

06:40 And a second one asks whether this change needs a person anyway. Two, only trusted authors, mostly the team that owns the code, get the lighter paths. Three, an AI’s approval never counts as a human’s. And four, my favorite, whoever prompted the change is on the hook for it. If you directed the agent, you count as a reviewer, the same as if you type the code yourself. One more detail, an approval covers one exact commit. If new commits land, you approve again.

07:12 Layer six, the merge queue reruns every check against the latest code with the change applied. So, two changes that pass alone can’t combine into something broken. New behavior ships behind a flag to Anthropic employees first, then to the latest channel with stable trailing behind. Layer seven assumes something still gets through. Flags switch off without a deploy. There’s a weekly on-call rotation, and every incident review asks one question, which layer should have caught this? One engineer put it well.

07:43 Slowing down does lower your error rate, but mostly because you’re doing less of everything. They’d rather get faster at noticing and fixing. If this video is useful, hit subscribe because it’s free. And if you want to back the work, hit join here or the supporter button on Facebook. Now, here’s where the story turns. When the author is an AI and the reviewer is an AI, they can keep each other busy forever. One pull request went through more than 50 rounds of review and fixes overnight before its author stopped it.

08:14 50. So, they ran get blame on what the reviewer kept flagging. About 59% of it was code that had been there since the reviewer’s very first pass. It wasn’t finding new bugs. It was finding a few more old ones every round. And they admit they made it worse themselves by running several rounds by default on every change. Noise also costs more when the author is an agent. A person waves off a nitpick in 2 seconds. An agent fixes it, and that’s another commit and another round. So, they made four changes. After three rounds, the reviewer tells

08:45 the author to step back and rethink the approach. After the first few passes, it only re-reviews the new changes. It flags the whole class of bug, so you fix every copy of it, not just one line. And it never repeats a finding. The result, passing after one round went from 65% to 78%. Needing five or more rounds dropped from almost 9% to 1 and 1/2. Average reviews to pass went from 2.08 to 1.32.

09:16 They keep the reviewer honest with a loop you can copy. Every past incident and every revert gets confirmed by an engineer who knows that code, and it goes into a test set. The reviewer runs against that set every day. If they change its prompt, its tools, or its model, and the score goes up, they keep the change. If not, they drop it. They also track what people do with each comment. Fixed it, said it was wrong, said it was on purpose, or decided it didn’t matter.

09:46 That’s how they measure noise. So, what’s left for humans? This part matters for your career. Pulling a senior engineer out of deep work to catch a typo costs far more than an automated review. So, people moved to five decisions machines are bad at. Should we build this at all? Risky code like billing, the auto up data, and CI workflows. Architecture that sets a pattern others will copy. Reading the evidence, the test output, the screenshot, the findings, and owning the

10:19 outcome. A person is accountable for every change that ships, whoever typed it. >> What is important is understanding maybe the bigger picture. It’s like maybe Claude doesn’t have a full understanding of why is this API structured the way it is or you know the service boundaries are drawn in a particular spot for a for a reason that maybe Claude hasn’t fully internalized when it was writing the PR. >> And if machines do the reading, how does anyone learn the code? They ask Claude to write an explainer for exactly

10:50 the part they’re about to change. A one-line copy change teaches you [music] nothing. An explainer of the module you’re about to touch teaches you a lot. Their rule for the rest is simple. You land it, you own it. That’s not blame. It’s because you have the most context to fix it. Here’s how you start this week in this order. One, give your agent checks it can run listed in a short Claude.md. Two, run {slash} code review and {slash} security review on your own changes

11:22 before you open the pull request. Three, add a review.md and every time you write the same review comment twice, move it into that file. Four, add testing, risk, and how to revert to your pull request template. Five, write down your risk tiers. Start strict and loosen only with evidence. >> [music] >> Six, cap review rounds at three. And seven, turn every revert into a test case for your reviewer. I put all of it into a free kit, >> [music]

11:52 >> a starter review.md, the pull request template, a one-page risk tier sheet, and the seven-step checklist. Comment nitpick, and my bot sends it to you. Real talk, 90% of you watching haven’t subscribed. On Facebook, subscribe is the supporter button. That is what keeps the lights on here. On YouTube, it’s free, and it tells the algorithm to send you more. Subscribe to support us. Follow along on Instagram and Facebook for bite-size AI, and the deep dive

12:22 playbooks live here on YouTube. Subscribe, so you don’t miss one. So, next Monday, when you’re about to leave three little nitpicks to prove you read the code, ask the better question instead. What could go wrong here, and what’s the right check to catch it? I’ll see you in the next one.