Same Local AI, Two Agents — Only One Caught Its Own Mistakes

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Same Local AI, Two Agents — Only One Caught Its Own Mistakes · Channel: Zero to MVP · 08:37 · Watch on YouTube

Topics: AI agents, Task and workflow automation, AI evaluation and benchmarks, Local and open-source models, Privacy and data protection

Summary

The video compares the same local Qwen model performing office tasks through Pilot Deck and the lightweight Pi agent. Across nine runs, Pilot Deck made no mistakes, while Pi made two errors and confidently reported the results as correct. The key difference was not Pilot Deck’s Excel and PowerPoint skills, but its instructions to run scripts, inspect the output, and independently compare calculations. Pilot Deck was more reliable in this test, but took longer and used more energy; the creator cautions that nine runs are not enough to guarantee the same result in general.

Key points

The experiment used the same model, computer, and tasks with two different agents: Pilot Deck and Pi.
The model was Qwen 3.8 with 27 billion parameters, run locally with llama.cpp and medium reasoning.
The tasks used synthetic Stripe data covering eight months, with nearly 800 payments and nearly 300 subscriptions.
Both agents completed the simple monthly revenue table correctly in all three runs, but Pilot Deck took longer.
For the metrics task, Pilot Deck compared its workbook calculations with an independent Python calculation and caught a currency error.
Pi made two mistakes across nine tasks: applying the euro exchange rate to dollar payments, and failing to exclude test accounts.
Pilot Deck’s independent checking behavior came from its agent instructions; it passed the difficult tasks even when its spreadsheet and presentation skills were removed.
Across all nine tasks, Pilot Deck took 2 hours and 8 minutes, compared with under an hour for Pi, and used more than twice as much graphics-card energy.
The creator concludes that a lightweight agent may be sufficient for simple, quick tasks, while Pilot Deck may be useful when report accuracy and local data handling matter.

Chapters

Start	Chapter	Summary
00:00	The experiment and the two agents	The creator introduces a comparison of Pilot Deck and Pi using the same local model, computer, and office tasks.
02:08	Data and task setup	The experiment uses synthetic Stripe exports with data traps and tests a monthly report across three levels of difficulty.
03:47	Metrics and investor package results	Pilot Deck catches a currency mistake by comparing its workbook with an independent calculation, while Pi makes two errors across the harder tasks.
05:57	What made Pilot Deck more reliable	The creator finds that Pilot Deck’s instructions encourage independent checks, and that its skills were not necessary for passing the tests.
07:34	The cost and conclusion	Pilot Deck takes longer and uses more energy, but the creator considers it a workable local option when accuracy matters.

Quotes

In one agent, the model didn’t make a single mistake in nine runs. — 00:32

The numbers didn’t match. — 03:47

The model found the mistake, fixed the formula, and then double-checked the customers a third way. — 04:19

Save the script to a file, run it, look at the output, and fix it. — 06:30

In my test, yes. — 08:06

Concepts and entities

Pilot Deck, Pi, Qwen3.8, llama.cpp, LibreOffice, Stripe, Verificação independente, skills de Excel e PowerPoint

Full transcript (original language)

00:00 In a recent video, I gave a local Quen 3.8 with 27 billion parameters a few real tasks. It handled some of them really well, but on others it made mistakes and didn’t even notice. Today, I wanted to test a different idea. What if it’s not just about the model, but also about the agent it runs in? So, I took the same model, the same computer, the same tasks, just two different agents and run one more experiment. And the results surprised me. In one agent,

00:32 the model didn’t make a single mistake in nine runs. In the other, it produced a wrong result twice and confidently reported that everything was correct. But the reason turned out to be completely different from what I expected. For today’s experiments, I’m using Pilot Deck. It’s an open-source AI agent for office work. It’s available on GitHub, so you can experiment with it, too. The main idea behind this agent is simple. You describe a task in plain language and the agent plans the steps, works with your files, and hands you

01:03 finished documents, Excel, PowerPoint, Word, not text in a chat, but files you can open and edit. It’s a general-purpose agent and you can connect any model to it, including a local one. Installing Pilot Deck takes just one command, but there’s one important thing. If you need to work with spreadsheets and presentations, like I do in today’s experiment, you also have to install LibreOffice separately. Without it, the agent can create files, but it can’t check them. For the model, I’m using Quen 3.8 with

01:37 27 billion parameters. I’m running it with llama.cpp on my desktop with a Radeon Pro with 32 GB of memory. The only difference from my last test is the reasoning level. Last time, it was set to the maximum. This time, I chose medium. At the maximum level, this model is reportedly unstable. I connect the model to the agent as a regular OpenAI compatible server, and that’s it. By the way, anonymous telemetry in Pilot Deck is off by default. So, after

02:08 installation, the agent doesn’t send anything anywhere without your permission. Okay, onto today’s task. I made up a small SaaS and prepared Stripe exports for it covering eight months, almost 800 payments, and almost 300 subscriptions. The job is a monthly report for an investor, an Excel workbook with metrics and a short presentation. The data is made up, but it’s exactly the kind of thing you don’t really want to upload to the cloud. Revenue, customers, their email addresses. I split the work into three

02:41 levels, a simple table, metrics by the rules, and a full investor package. And of course, the data has traps, test accounts, payments in euros, customers who click at cancel, but are still paying. All of this is described in the rules. The agent just has to follow them carefully. The checker is hidden. I run it after the agent is done. I run each task three times in Pilot Deck and three times in Pi, the simple agent from my last video. So, the first task. It’s a simple one. Build a table of revenue by month from

03:16 the payments export. The model got it right in both agents in all three runs. Not a single mistake. But in Pilot Deck, the task took 7 to 9 minutes. In Pi, 2 to 4. Why such a difference? Pilot Deck gives the model a lot more context, instructions, descriptions of tools and skills. The model takes more steps and checks more. As a result, 10 times more tokens went through it for one simple task. So, on a simple task, Pilot Deck didn’t add anything yet. The second task

03:47 is harder. Calculate MRR, churn, new customers, and revenue strictly by the rules. And this is where probably the most interesting moment of the whole test happened. Pilot Deck built the workbook, then recalculated it in LibreOffice, and compared it with its own separate calculation in Python. The numbers didn’t match. And [clears throat] here’s what the model wrote in its reasoning. Column H is monthly price local. Wait, I made a mistake. The MRR formula was taking the price in the subscriptions currency

04:19 instead of in dollars. The model found the mistake, fixed the formula, and then double-checked the customers a third way. In the end, all three runs without a single mistake. Now, the same model in Pi. First run, the euro exchange rate was applied to every payment, including the ones in dollars. $9 turned into 9.72. And in its answer, the model confidently says, “Euros converted at the month’s rate.” Exactly the same mistake Pilot Deck made. Only there, the

04:49 model caught it because it checked the result. Here, it didn’t. The third task is the hardest. It’s a workbook plus a presentation of up to six slides. August’s key metrics with the change from July, an MRR chart, the reasons customers canceled, and plans for next month. Here’s what Pilot Deck produced. Clean slides with proper charts. All the numbers match the workbook. And most importantly, the model correctly found the main reason for cancellations, price. And almost half of the price complaints weren’t in

05:22 a dedicated field, only in free text comments. Before handing over the files, the agent checked on its own that they contained no customer email addresses, rendered the presentation, and made sure no text run off the edges of the slides. Three runs out of three. All 30 checks passed. In Pi, the first run scored 23 out of 30. The model didn’t exclude the test accounts. Because of that, every month had two extra customers and 58 extra dollars. And the answer said, “Test accounts excluded everywhere.” The next two runs in Pi had no mistakes. By

05:57 the way, two comments in the data really are debatable. “Cheaper alternatives do enough for us.” Is that about price or about switching to another tool? So, I counted anywhere from 11 to 13 cancellations as price-related. Now, for the most interesting part, Pilot Deck has special skills for Excel and PowerPoint with instructions on how to create files and how to check them. I was sure that was the reason, but when I went through the logs, it turned out that on the first two levels, Gwen never opened the skills. Not in the first run

06:30 of the third level, either. And it still scored 30 out of 30. It came up with its checks and wrote them all by itself. So, I run one more test. I removed the skills from Pilot Deck completely and repeated the two hardest tasks. Same result. All checks passed. It just took longer. And as it turned out, what mattered most in my test wasn’t the skills, but the agent itself. Its instructions contain a simple rule, “Save the script to a file, run it, look

07:01 at the output, and fix it.” And the model really does that. It calculates the same thing two different ways and compares. If they don’t match, it goes looking for the mistake. In Pi, the model checks its work, too, but differently. It opens the finished file and makes sure the formulas calculate and there are no customer emails in it. What never occurs to it is to verify the numbers against the source data with an independent calculation. That’s why a mistake in the logic itself goes unnoticed. But, this reliability comes

07:34 at a price. Across all nine tasks, Pilot Deck took 2 hours and 8 minutes. Pi, under an hour. And the graphics card used more than twice as much energy. And a few things worth knowing. Web search in Pilot Deck only works through cloud services. Fully local means working with files only. And of course, nine rounds and final statistics. The model in Pi made a mistake two times out of nine. It’s a real difference, but it’s still too early to say it’ll always be like

08:06 that. So, can an agent make a small local model more reliable? In my test, yes. The same Qwen in Pilot Deck didn’t make a single mistake in nine rounds. And it was the agent with its system prompt that made the difference. So, here’s my conclusion. If the task is simple or you need a quick answer, a lightweight agent is enough. But if a mistake in a report would cost you, and you don’t want to send your data to the cloud, a local model in Pilot Deck is a perfectly workable option. You just have to wait a

08:37 little longer. Links to the Pilot Deck website and the GitHub repo are in the description. Try connecting your own model to it and see how it handles your tasks. And if you’re into real-world tests of AI models like this one, subscribe to the channel and to my weekly newsletter. See you soon. Take care.