Claude's OFFICIAL ENGINEER Mode: The Loop Anthropic Engineer Uses Themselves!

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Claude’s OFFICIAL ENGINEER Mode: The Loop Anthropic Engineer Uses Themselves! · Channel: AICodeKing · 11:21 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, Claude, Software engineering with AI

Summary

The video explains Anthropic’s approach to speeding up Claude’s web and desktop experience through an agent loop that combines measurable goals, repeated experimentation, and explicit verification. It describes techniques including instruction counting, CI performance limits, production monitoring, reducing React work, and moving code tokenization to a web worker. The speaker also emphasizes testing real user experiences, coordinating parallel agent work, using human approval and staged rollouts, and weighing performance gains against code complexity. The main takeaway is to give an AI agent a reproducible problem, a baseline, a budget, and a way to confirm that its changes help without breaking existing behavior.

Key points

Anthropic reported that 75th-percentile typing readiness on a fresh web load improved from 3.1 seconds to 0.55 seconds during a two-week sprint.
An effective agent loop defines success, repeats work and verification, and includes a stopping condition.
Hill climbing means making a change, measuring its result, and using that evidence to choose the next change.
Instruction counts can provide repeatable comparisons; CI limits can preserve improvements and flag regressions.
Production measurements complement controlled lab tests because users have varied devices, networks, and usage patterns.
The team used an early HTML input box, reduced sidebar re-renders, reused completed reply blocks, and moved tokenization to a worker.
Performance metrics such as CLS and frame rate should be considered alongside recordings, real interactions, and tests of reported problems.
Human owners reviewed changes and staged rollouts; the speaker cautions that more agents increase coordination needs.
The speaker favors small, measurable fixes and questions whether a tiny speed gain justifies substantial added complexity.

Chapters

Start	Chapter	Summary
00:00	The case for measurable agent work	The speaker introduces Anthropic’s reported speed improvements and explains why agents need measurable goals, verification, and a stopping condition.
02:02	Hill climbing, profiling, and performance guardrails	The video explains controlled experiments, instruction counting, CI regression checks, and how production data helps guide further optimization.
04:35	Interface and rendering optimizations	The speaker discusses preserving input during startup, reducing unnecessary React work, reusing completed reply blocks, and moving tokenization off the main thread.
07:10	User experience, evaluations, and rollout	The video considers layout shifts and representative evaluations, then covers parallel investigations, human approval, and staged rollouts.
09:46	Trade-offs and a practical workflow	The speaker weighs maintenance costs against small gains and recommends starting with a reproducible problem, a baseline, a limited budget, and post-fix verification.

Quotes

At the 75th percentile, typing readiness on a fresh web load improved from 3.1 seconds to 0.55. — 00:31

Define what success means, let the agent repeat the work and verification, and give it a stopping condition. — 01:02

It means making a change, evaluating the result, and using that result to decide which change to try next. — 02:02

They rejected a 900-line proposal saving 2 milliseconds per cent. — 09:15

The quality of the feedback deserves as much attention as the prompt. — 10:17

Concepts and entities

Anthropic, Claude, Claude Tag, Opus 5.5, Slack, hill climbing, Valgrind Callgrind, continuous integration, React, CLS

Full transcript (original language)

00:00 [music] >> Hi. Welcome to another video. So, Anthropic [music] says Claude’s web and desktop experience became about three times faster during a two-week sprint. And I wanted to spend some time on how they approach this because the engineering ideas are useful even if you don’t use Claude. When you ask an AI coding tool to improve an application, you still need some way to tell whether its changes helped. A confident explanation and a big code change don’t give you that answer. At the 75th percentile, typing

00:31 readiness on a fresh web load improved from 3.1 seconds to 0.55. That percentile means roughly three quarters of the measured loads were that fast or faster. It isn’t an average and it doesn’t describe the slowest users. Also, being ready to accept your typing is a different measurement from how quickly a model generates its response. My reading of what Anthropic is preaching here is that you should give the agent a job with a way to check its own progress. Their broader guidance on agent loops makes this explicit. Define what success means, let the agent repeat the work and

01:02 verification, and give it a stopping condition. I think that’s a much more useful discussion than counting how many lines an AI wrote. If I’m waiting for a document to open, I care about whether the document opens sooner and still works properly. So, let’s get into the technical side of this and then talk about what I take from it. Their team used Claude tag in Slack with an internal research model roughly comparable to Opus 5.5. Claude tag is their shared agent experience inside Slack. It can use the conversation in a thread, work with connected tools, and follow standing instructions. So, you can give an agent an ongoing

01:32 responsibility such as watching incoming issues instead of having to explain the whole job again whenever something happens. For a team, I can see the appeal. The person reporting a problem and the person reviewing the fix can discuss the same evidence in one place. But putting an agent in Slack doesn’t make the investigation good by itself. You still have to give it useful access and decide what it should be allowed to change. Now, the phrase hill climbing comes up in this discussion. It means making a change, evaluating the result,

02:02 and using that result to decide which change to try next. You need an objective that’s specific enough to measure, and the improvement needs to be larger than the noise in your evaluation. Imagine you’re trying to make a search box respond faster. You could change the way it filters results, run the same search again, and compare. If you’ve also changed the data set, the browser, and the network connection, then you don’t know how much credit the code change deserves. The experiment needs to be understandable. They measured instruction counts, tied those to elapsed time improvements, and enforced lower ceilings in continuous

02:32 integration. So, let’s unpack that. Wall clock time is the time that passes on a clock while an operation runs. That’s useful because waiting is what a user experiences, but timings can move around when the machine is doing other work. A profiler helps you look inside the operation and find where the work happens. Valgrind’s Callgrind tool can count executed machine instructions and attribute work to functions and the functions they call. That lets you investigate whether a particular operation is repeatedly doing unnecessary work. An instruction count is a different measurement from milliseconds, and fewer

03:02 instructions won’t automatically mean the entire application feels faster. For a fixed program and input, instruction counting can give you much more repeatable results than a stopwatch. That makes small changes easier to compare. You’re counting the machine’s executed work, not the number of lines in a source file. Think of a delivery service. You could make the paperwork twice as fast, but if the package spends a day waiting for a van, the customer might see no difference. In software, I’d want to know whether the function we’re optimizing is actually responsible for a meaningful part of the weight.

03:33 Continuous integration, or CI, is where automated checks run when code changes. A performance check can compare a new measurement against an accepted baseline and flag a regression. A regression just means that a change made something worse than it was before. The ratchet idea is that an improvement becomes the new standard. For a made-up example, suppose a repeatable task a thousand operations and you reduce that to 700, you update the limit to 700. So, a later change that pushes the count back up gets noticed. They checked production data after deployment and kept looking for the next bottleneck.

04:03 Production data is what you measure from people using the released application. Lab tests give you controlled conditions, while real users bring different devices, network speeds, and usage patterns. Both are useful. A change can look good on your test machine and still have little effect on the people experiencing the problem. I like this because the fix now comes with a way to remember why it mattered. Otherwise, someone can accidentally undo your optimization six weeks later while working on a completely different feature. You shouldn’t have to depend on everybody remembering every performance decision. They used an early HTML input box while React initialized and reduced

04:35 sidebar re-renders. React is a library for building interfaces. With an application that sends HTML first, you can show content before all the JavaScript is ready. For background, hydration is the usual React process of connecting component logic to existing HTML. The initial HTML and the React output need to agree or the handoff can cause problem. Claude’s early text box had its own handoff to the full React interface. I’d care about more than whether it appears early. If I type the first half of a prompt while the application is starting, those characters need to

05:06 survive the handoff. The box should keep focus and the text shouldn’t jump. A loading improvement that interrupts typing would be pretty annoying. Now, a React re-render means React calls components to work out what the interface should look like. It doesn’t necessarily mean the browser redraws everything. React then determines which changes need to be applied. But running that component logic can still cost time, especially when expensive work happens repeatedly. Memoization means reusing a previous result when the relevant inputs haven’t changed. For an interface, this can help avoid repeating

05:37 work for a component whose content is still the same. You have to check the inputs correctly, though, because reusing an outdated result gives you an interface that fails to update when it should. Imagine typing in a message box while a completely unrelated list recalculates itself on every key press. I’d want the agent to show me why the list is involved at all. That’s a specific investigation. And you can test whether the behavior changes after the fix. They also reused finished reply blocks and moved code tokenization to a worker, reaching 120 frames per second on a 120 hertz MacBook. Tokenization here means

06:09 breaking code into pieces so a syntax highlighter can recognize things like keywords and strings. This is work the interface does to display code. A web worker can run suitable JavaScript work on a separate thread, communicating results back through messages. It can’t directly manipulate the page’s document the way the main thread can. >> The browser’s main thread handles a lot of the work involved in running page JavaScript and updating the interface. When one task occupies it for too long, other work has to wait. That can show up as delayed input or an animation that

06:40 stutters. At 120 frames per second, there are only about 8.3 milliseconds between frames. That’s the time available for the whole frame. So an individual piece of JavaScript can’t casually use all of it. A high refresh rate makes unnecessary work easier to notice because the next frame is due so soon. For a streaming answer, imagine receiving the last sentence of a page and then formatting every earlier paragraph again. As the answer grows, that approach keeps giving you more work. Reusing the completed parts is sensible as long as you know they really

07:10 are complete. Otherwise, you can trade a performance problem for incorrect formatting. The sidebar jumps stayed within the usual good CLS threshold despite bothering users. A layout shift is when visible content unexpectedly moves. Cumulative layout shift or CLS scores these movements using factors such as how much of the screen is affected and how far things move. It’s useful for detecting unstable pages, but a score doesn’t describe every detail of someone’s experience. Suppose a small button moves just as you’re about to click it. The area involved might be tiny, but you still click the wrong

07:40 thing. That’s why I’d use recordings and real interactions alongside a dashboard. I’d also want a test that reproduces the particular movement people complained about. Anthropic’s separate guidance on evaluations makes a related point. The tasks you test should reflect the work you actually care about. An agent can improve a benchmark without improving real use if the benchmark is too narrow. Testing on examples outside the ones used for optimization helps expose that. That’s the part I’d keep coming back to if I were managing this work. You can make a dashboard look healthier by changing what gets measured or by avoiding a difficult case.

08:11 I’d want the team to be able to explain what a better score means for the person using the product. They ran over 150 threads with human owners, tests, and staged rollouts. I can see why parallel work helps when the investigations are separate. One agent might study startup while another looks at scrolling, but if they both change the same shared component, somebody needs to resolve the overlap. I’d expect review and coordination to become a bigger part of the job as the number of agents increases. A feature flag lets a team enable or disable a behavior without necessarily removing the code. A staged rollout

08:42 introduces a change to a limited group first, then expands it after the team checks the results. That gives you an opportunity to catch problems under real conditions before the change reaches everybody. Every change needed human approval. Anthropic reports over 3,000 merged changes with no customer-facing incidents or rollbacks. That’s an encouraging result for this sprint. I’d still be careful about treating it as a promise for another team with different tests and a different code base. The amount of support around the agent is part of what you’re evaluating here. Their engineers supplied ambition, taste, and direction,

09:15 pushing for further improvements, judging visible changes, and keeping investigations focused. I think that gives the human a demanding role. You need enough understanding to challenge a weak result and enough restraint to stop an unnecessary rewrite. I’d also want somebody watching the experience as a whole. Optimizing each individual component doesn’t guarantee that the entire application feels coherent. They rejected a 900-line proposal saving 2 milliseconds per cent. Personally, I like that detail a lot. A performance improvement still has a maintenance cost. Somebody has to understand the new

09:46 code when it breaks, update it when dependencies change, and decide whether it still belongs there. I’d want a stronger reason to accept that much complexity for such a small saving. So, the workflow Anthropic is encouraging gives the agent more room to iterate while making verification explicit. Their loop guidance also recommends testing on a small slice of work before scaling up, setting clear limits, and checking token usage. Those details matter if you’re paying for repeated attempts. For your own app, I’d begin with one problem you can reproduce. Ask the agent to capture a baseline and investigate the cause

10:17 before changing things. Then ask for a small fix, the same measurement afterward, and a check that the existing behavior still works. I’d also give that first experiment a budget. You can spend a lot of time improving something nobody notices if the agent has tried several approaches and the remaining improvement is tiny. I’d want it to explain the trade-off before continuing. More activity doesn’t automatically make the project more valuable. Overall, what I take from this is that the quality of the feedback deserves as much attention as the prompt. I would rather spend some effort making a useful test than keep asking an agent whether it’s sure the

10:47 app is faster. Once it can show the difference, I have something concrete to review, and I can decide whether the improvement is worth keeping. Overall, it’s pretty cool. Anyway, let me know your thoughts in the comments. If you like this video, consider donating through the Super [music] Thanks option, or becoming a member by clicking the join button. Also, give this video a thumbs up and subscribe [music] to my channel. I’ll see you in the next one. Until then, bye.

11:21 >> [music]