Why AI Didn't Actually Make You Ship Faster — Gabriel Spencer-Harper, Meticulous

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Why AI Didn’t Actually Make You Ship Faster — Gabriel Spencer-Harper, Meticulous · Channel: AI Engineer · 10:51 · Watch on YouTube

Topics: AI agents, Software engineering with AI

Summary

Gabriel Spencer-Harper argues that AI can generate code faster than teams can review and verify it, making verification a new bottleneck. He presents Meticulous, a tool that records user workflows and replays them against pull requests, comparing screenshots before and after code changes so developers can review what changed. The system uses mocked network traffic, a deterministic browser, and code-coverage-based workflow selection to reduce flakiness and focus testing. Together, these techniques aim to provide broad verification with little developer effort and help teams ship AI-generated changes with greater confidence.

Key points

AI-generated code can outpace human review, shifting the bottleneck from writing code to verifying it.
Assertion-based tests cannot feasibly describe every possible regression or application state in advance.
Meticulous records front-end workflows in nonproduction environments and replays a selected subset on pull requests.
It compares sequences of screenshots from before and after a change, leaving developers or agents to judge whether differences are expected.
Recorded network requests and responses are mocked during replay to make runs repeatable and tests isolated.
The browser is augmented to reduce sources of nondeterminism and test flakiness.
Meticulous maps recorded workflows to executed code and selects workflows to maximize code coverage.
The speaker argues that code coverage is more informative in this approach because screenshots are captured throughout each workflow.

Chapters

Start	Chapter	Summary
00:01	AI coding and the verification bottleneck	Spencer-Harper introduces Meticulous and explains why code review and verification can limit the speed of AI-assisted development.
03:04	Why exhaustive verification matters	He outlines the costs of regressions and test maintenance, then introduces the goal of making broad verification possible.
04:06	Workflow recording and visual diffs	The demo shows how Meticulous records browser workflows, replays them on pull requests, and presents before-and-after screenshots for review.
07:14	How the verification system works	He explains network mocking, browser determinism, and coverage-based workflow selection as the technologies behind the tool.

Quotes

if you have exhaustive verification then you can ship code at the speed that your agents write it. — 00:32

AI writes code faster than humans can review it. — 01:33

What it will show you is a set of diffs that show you what’s different about the application if you were to merge that code in. — 06:12

everything is mocked out. — 07:14

code covered is not the same as code tested — 09:50

Concepts and entities

Meticulous, Gabriel Spencer-Harper, Cypress, Playwright, verificação visual, determinismo do browser, cobertura de código, flakiness

Full transcript (original language)

00:01 [music] Hello everyone. Hi. Um, my name is Gabe. I’m one of the co-founders and CEO of Meticulus. Um, I’ll just wait one more minute for people to come in and then we can and then we can get started. Um so in today’s demo or talk I’ll talk about verification of code. Um I’ll spend one

00:32 minute introducing meticulous and then one minute giving some background context on the company and then sort of eight minutes giving a demo and technical technical overview. Um so if you have exhaustive verification then you can ship code at the speed that your agents write it. If you don’t, then someone somewhere at your organization is spending time verifying code and that is now your new bottleneck and you’re

01:03 forced into a position where you’re trading off velocity against bugs and user experience. So I’ll give um super quick context on Meticulus. So, Meticulus gives you exhaustive or near exhaustive verification of your front-end codebase with zero developer effort. Um, it’s deployed across the entire engine organization at all of these companies like Discord, Whiz, Dropbox, Notion, 11

01:33 Labs, Launch Darkly, and many, many other great companies. So, if an engineer touches front-end code at one of these companies, they use Meticulus every every day. However, before we get into the product and talk more about verification, um, let’s just cover the current sort of state of the world. So, AI writes code faster than humans can review it. And review and verification is now the new bottleneck.

02:04 The other state of the world is that assertionbased testing alone isn’t enough. So no matter how hard a human or an agent tries to define the correct behavior of software a priority upfront in assertions the space of possible regressions is too vast to exhaustively cover. And so a good clarifying question is if an engineer in your organization

02:34 uh AI generated this pull request, would you feel comfortable hitting the the merge merge and ship button? And in most organizations, the answer is no. You would want to go and do the hard work of checking out all the different feature flags, all the different roles, permissions, settings, configurations, and the various different edge cases in order to understand the exhaustive impact of this of this change and see if you can spot any issues.

03:04 So there are three sort of like downstream consequences of that current state. The first one is bugs or regressions uh which have a a business impact. The second one is engine organizations will spend doubledigit percentages of their time maintaining end to-end test suites. So across manual validation, review, uh debugging flakes or updating or maintaining the test suite, um all

03:34 spend a lot of time maintaining maintaining maintaining tests. And then the third like problem or downstream downstream consequence of of the state of the world is that um if you if you can get exhaustive verification then you can program in a new and different way. So all of a sudden you can bump all your dependencies, you can do sweeping refactors, you can make any AI generated change with complete complete confidence. So then the question becomes

04:06 um that sounds great. How do you how do you get exhaustive verification? What does it look like and how do you how do you get there? Um so I’ll now give you a demo of how how meticulous meticulous works. So, we give you uh give me one second to set this up. Great. Um, so we give you one line of

04:38 JavaScript to inject onto nonproduction environments. So that’s localhost, QA, dev, staging. That JavaScript instruments the web browser and records thousands or tens of thousands of flows or workflows. So someone clicking the login button, clicking the settings panel, clicking the analytics panel. And then when you open a pull request on your CI runner, you spin up your web application on localhost 3000 and Meticulus takes a subset of those recorded workflows and it replays them.

05:10 So it dispatches each event one by one. And as it dispatches those events, so click the login button, click the senses panel, click the analytics panel, it takes a screenshot at each atomic moment throughout time. And so it generates a sequence of screenshots for every workflow, one before your code change and one after your code change. It then diffs those two sequences together to show you what would be different about your application if you were to merge that code in before before you do. So in

05:41 this example, there’s two changes here. One is someone’s changed the name and the second is they’ve introduced an error to a to a drop down. And so when you open a pull request, Meticulus posts a comment typically within a few minutes. And when the developer clicks into this comment, the Wi-Fi is slow, so I’m just going to switch tabs. But when a developer clicks into this comment, Meticulus will not tell you whether or not there’s a bug. What it will show you is a set of diffs

06:12 that show you what’s different about the application if you were to merge that code in. And so each diff has a before state and an after state. And so by reviewing the diff, you can determine whether that change is expected or unexpected. So here’s the name change on light mode. Here’s the name change on dark mode. And then this is a manifestation of that logical logical error. So typically when you write an assertionbased test, you um use your

06:43 business context and judgment to encapsulate the correct behavior of software up front and and encapsulate that into your assertion based test. In meticulous, we just show you what’s different and you exercise your judgment at review time or or or an agent agent does. Um give me one second to switch this back. Cool. Um, so there are sort of three

07:14 technologies that are really important for helping build a mental model of the tool and how it works, how it works and why it works. So the first one is that everything is mocked out. So um at record time we record all the network requests and responses and then at replay time we stub those in and we perform that mocking for two reasons. One is to make the test item potent. So you can run them a thousand times in a row and get the same results each and every time. And the second reason is it

07:45 means that every test is isolated from every other test which eliminates the risk of race conditions and allows you to paralyze the test horizontally and get get the results in it in a few minutes. The second key technology is that you get radically less or orders of magnitude less flakes with meticulous than any other tool including cypus or playright. And the reason why is we augment the browser from the scheduling engine layer up to be fully deterministic. So when browsers were first designed determinism was not a

08:17 design goal. And so there’s many different sources of randomness. For instance, an animation spinner depends upon the clock speed of the CPU of the machine rendering the browser or the interle between set timeout and set interval. And so we handle all of that randomness and uh make the browser deterministic or near deterministic so that you have radically radically less flakes. And then the third technology which is the most um important one for grocking like why this works is if you want this

08:48 to be exhaustive it has to cover every feature plan combination every permission every role every config every possible branching through the application. So how do you how do you do that? So one technique that we use is that for every session or flow that we record, we replay it once against main branch or master branch and we monitor what lines of code get executed and then we build a map or an index that maps each individual workflow to lines lines of code executed and choose a subset of

09:19 workflows that maximizes code coverage across your application. And typically uh give me one second. Typically I would say that code coverage in our opinionated view is a terrible metric for every tool in the world apart from meticulous and the reason why is you could write a cypress test or playright test that covers 100% of your codebase but only makes a single assertion and so code covered is not the same as code tested but with meticulous it actually

09:50 is approximately the same because we do this really unusual and radical thing which is we’re taking a screenshot at every possible moment throughout a flow. So if there’s a 412 set flow and there’s a single pixel diff, meticulous meticulous will will will flag it. And so you can start to see how all these technologies layer together. Um you only get exhausted verification because of this code coverage algorithm. That algorithm only really works because you take a screenshot at every possible

10:20 moment. If you take a screenshot every possible moment, you’re taking on the order of tens, hundreds of millions of screenshots. And so, traditionally, you would just drown in noise or drown in flakiness. And so, you have to solve flakiness at the root level. Um, which is which is what we what we what we do by by augmenting augmenting the browser. Um, these are some nice things that our customers customers have said about us. Um, and that concludes today’s talk. If you’re interested in finding out more

10:51 then come to booth LG6 and thank you so much for listening. [applause] >> [music]