What Technical Due Diligence Finds When the Deck Says “70% Automated” — Soukaïna Abelhad

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: What Technical Due Diligence Finds When the Deck Says “70% Automated” — Soukaïna Abelhad · Channel: AI Engineer · 06:11 · Watch on YouTube

Topics: Task and workflow automation, AI evaluation and benchmarks, AI governance and regulation, Markets and economy

Summary

The talk offers a framework for assessing AI automation claims in technical due diligence, arguing that a real measurement can still be used to support a conclusion it does not justify. It recommends checking whether workload growth is accompanied by reduced fully loaded human labor, defining exactly what “automated” means, and validating automated quality measures against expert judgment. Claims that pass those checks must still make economic sense, including the costs of AI, review, escalation, and rework. The central question is not whether a number is impressive, but what must be true for it to matter.

Key points

A reported 55.8% speed improvement came from a controlled experiment involving 95 professional developers, but it tested one JavaScript HTTP server built from a template against 12 visible tests.
A genuine measurement may be carried into a conclusion it cannot support; diligence should examine how the measurement was produced and what it actually establishes.
Faros observational data found that high-AI-adoption teams merged 98% more pull requests, while review time rose by 91% and average pull-request size by 154%.
Compare workload volume with fully loaded human labor, including operators, reviewers, trainers, escalation, and rework; if labor tracks workload, the evidence indicates assistance rather than labor removal.
Distinguish AI-attributed work from autonomous work, and define whether the automation boundary ends at drafting, approval, or production.
Automated graders should be validated against human ground truth: the talk cites a 24-percentage-point gap between an automated grader and merge decisions in a Meta study.
Ask who labeled the ground truth, whether experts agree, how that agreement is measured, and what bad output escaped evaluation.
Translate claims that survive these checks into unit economics, including AI costs, human review, escalation, and rework, and measure cost per accepted autonomous task.

Chapters

Start	Chapter	Summary
00:00	What a headline automation number really shows	The speaker introduces a 55.8% speed claim and explains why the experiment’s limited task and test scope matter.
01:03	Check the arithmetic: workload and labor	Observational data on pull requests motivates comparing workload growth with all human labor associated with AI.
02:04	Define what counts as autonomous	The talk separates AI-attributed work from end-to-end automation and asks where the automation boundary lies.
03:06	Validate the measurement against experts	A gap between automated grading and human merge decisions shows why evaluation methods and ground truth need scrutiny.
04:08	Turn surviving claims into economics	The speaker connects diligence failures to decision-making and recommends calculating the cost of accepted autonomous tasks.

Quotes

The dangerous number is not always fabricated. It’s often a genuine measurement carried into a conclusion it cannot support. — 00:30

AI attributed and autonomous are two different claims. — 02:35

This is why evaluation maturity is first credibility signal I would look for. — 03:38

The metric I would want is cost for accepted autonomous task. — 05:09

It’s to ask what must be true for the number to matter. — 06:11

Concepts and entities

Soukaïna Abelhad, Faros, Uber, Meta, Builder.ai, straight-through processing, due diligence técnica, avaliação automatizada

Full transcript (original language)

00:00 My name is Sukaina, and I’m an AI tech lead, and today I’m here to talk about what a technical due diligence finds when the deck says 70% automated. Someone has probably shown you a number like this. 55.8% faster. And you’ve been wondering whether to believe it or not. Well, this one comes from a controlled experiment with 95 professional developers. But the task was one JavaScript HTTP server built from a template against 12 visible tests. Today, I’ll give you three checks for

00:30 what technical due diligence finds in the AI era. I’ve actually spent more than 80 years on both sides of the question. On the front side, I built AI system that investment professional use. On a build side, I now lead AI evaluation in production and help create the numbers that reach executives. The dangerous number is not always fabricated. It’s often a genuine measurement carried into a conclusion it cannot support. Chun Chan described the buyer side in a nice way. Built for the memo, not for the demo.

01:03 I want to extend that standard to operational metrics. And the first check requires no knowledge of models. It’s pure arithmetic. In Faros observational data, high AI adoption teams merged 98% more pull pull requests. Pull request review time increased by 91% and the average pull request became 154% larger. And the sample covered more than 10,000 developers across 1,255 teams.

01:34 This does not prove that AI increased head count. It’s an observational telemetry, not a controlled labor study. But it still give us a warning. More work entered the system and the downstream review burden grew with it as well. So in diligence, ask for the company’s own 18-month theory. Put workload volume next to the fully loaded human labor. Count everyone whose work exists because of AI, for example, operators, reviewer reviewers, trainers, escalation and rework.

02:04 If human labor tracks workload volume, the deck has described assistance, not labor removal. But when the arithmetic survives, the definition call still can fail. Uber, for example, says that 70% of its pull requests are AI attributed. This is impressive. It still does not tell us the full autonomous rate. The period and total count were not disclosed, either. So, every automation claim sits somewhere in this ladder.

02:35 It’s either AI drafted it, or edited by humans, or approved by humans, or no human intervention. Finance already has a useful term for the last one. It’s called straight-through processing. Define it as a complete end-to-end without human intervention inside a boundary you name. So, does the boundary here end when the code was drafted, or when the pull request is approved, or when the change reaches production? So, remember, AI attributed and autonomous are two different claims.

03:06 Okay, now suppose the definition is precise. Can we trust the measurement? Meta found 24% point gap between automated grader and merge decision after adjustment to the human baseline. Four maintainers from three repositories reviewed 296 AI generated pull requests across different unique issues. They did not know whether the patch came from AI or human. The automated grader was substantially more optimistic than the people responsible for accepting the work. Not

03:38 a surprise. I have seen the same thing from the other side. One of the projects I worked on, 10 out of 10 output would pass the automated validation. Once you bring the the experts on the table, barely few survive. This is why evaluation maturity is first credibility signal I would look for. So, always ask who labeled ground truth, and whether the expert agrees, and whether anyone measures that agreement. Ask what the automated judge was validated against, and ask what bad output escaped.

04:08 So, when arithmetic’s definition and evaluation all fell, a word it still has to become a decision, right? This is not a claim that weak AI positioning caused builder.ai collapse. Its insolvency involved revenue financing issues beyond AI claims. The diligent lesson started way back in 2019, when the Wall Street Journal publicly questioned the claim of the AI capability. In May 2025, the The Financial Times

04:39 also reported that its 2024 revenue estimated had been revised from $220 million to roughly $55 million. On June 2nd, the company filed Chapter 7. So, the warning existed, but the process did not turn it into a gate. So, even when a claim survives those check, it still has to survive the economics. So, start with eligible workflow value. Apply actual adoption,

05:09 straight-through process success, and accepted quality. Then, subtract the cost of AI, human review, escalation, and rework. The metric I would want is cost for accepted autonomous task. If the task fails, escalates, or need rework, this belongs to the cost. So, always put those checks together, and this is the card I would probably take with me to the next meeting. To sum up, three main checks. The arithmetic’s, compare workload volume

05:40 with fully loaded human labor. Two, definition, identify the wrong and and the straight-through rate inside a name workflow. Three, the tell. Ask who defines quality. Whether expert agrees in what that output escapes. Then translate only the surviving claims into a unit economics. And of course, believe it tested or walk away. The printable card in all the source notes are in the website and this QR code will lead you to it. So, take a card,

06:11 scan it, print it, and take it to your next vendor meeting or investment discussion. And always remember, the job of diligence is not to ask whether the number is impressive. It’s to ask what must be true for the number to matter. Thank you.