This Tiny 2B AI vs Claude Code's Token Bill (Real Test)
Read the Portuguese translation →
Author: Prompt Engineer 48 · Watch on YouTube
Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.
Original language: English
Original video: This Tiny 2B AI vs Claude Code’s Token Bill (Real Test) · Channel: Prompt Engineer 48 · 04:43 · Watch on YouTube
Topics: AI agents, Claude Code, Model inference and optimization, Local and open-source models, RAG, search and information retrieval
Summary
Section titled “Summary”The video tests Call Grep Agent, a small local model intended to reduce Claude Code’s token use by searching a codebase and returning relevant file locations. The creator compares it with normal grep-based Claude Code use on an 8-GB laptop and a roughly 109,000-line repository. The agent reduced total token use by 8% and cost by about 13% in this small test, but was much slower and produced fewer complete answers. The creator concludes it may be worth trying on large codebases where token costs matter, while the reported savings did not match their scale on this machine.
Key points
Section titled “Key points”- The blog’s test covered 13 public repositories and 52 questions, each run three times; it reported 34% fewer tokens and costs dropping from 7.3 cents to 4.4 cents per question.
- In the reported results, correct answers went from 83% to 81%, while time per question increased from 13 seconds to 21.6 seconds.
- The creator installed the agent on Windows without Python, a server, or an API key; the default Q8 model file is about 2.7 GB, with a smaller Q4KM option.
- The creator’s test repository had about 109,000 lines of Rust and Python, and indexing took 513 seconds.
- Across six questions, the agent used 944,000 tokens versus 1.03 million without it, about 8% fewer, and cost about 75 cents versus 86 cents.
- The agent was cheaper on five of six questions and used fewer tokens on four of six, but total time rose from 98 seconds to 646 seconds.
- The creator judged all six answers correct without the agent, compared with four complete and two partial answers with it.
- The creator notes that fixed Claude Code overhead was included in both runs and says the token-saving percentage might look larger on a cleaner setup.
- The creator’s conclusion is that the direction of the reported savings held up, but not their size in this test; it may be more worthwhile on larger codebases.
Chapters
Section titled “Chapters”| Start | Chapter | Summary |
|---|---|---|
| 00:00 | The reported savings and how the agent works | The video introduces Call Grep Agent and explains the reported token, cost, accuracy, and speed results. |
| 01:35 | Installing and testing on an 8-GB laptop | The creator installs the model, describes the laptop and test repository, and outlines the six-question comparison. |
| 02:38 | Question-by-question results | Examples show both token-saving wins and a case where the agent was slower, more expensive, and less helpful. |
| 03:40 | Overall results and conclusion | The creator totals the test results and weighs the agent’s potential value for small versus large codebases. |
Quotes
Section titled “Quotes”the cost per question drops from 7.3 cents to 4.4 cents. — 00:00
Cheaper, yes. Faster, no. — 01:03
That is the win. — 03:09
The size doesn’t on this machine in this test. — 03:40
On a big code base where you pay per token it’s worth trying. — 04:43
Concepts and entities
Section titled “Concepts and entities”CallGrep, CallGrep Agent 2B, Claude Code, Sonnet, Opus, llama.cpp, next-plaid, Apache License 2.0, Busca semântica em código, GPU integrada
Full transcript (original language)
Section titled “Full transcript (original language)”00:00 A 2-billion parameter model sits in front of Claude Code, and the cost per question drops from 7.3 cents to 4.4 cents. That’s what Llama reports. But the same table shows it got slower, and the accuracy moved, too. So, I ran it on my own 8-GB laptop to see which of those numbers hold up. This is the blog post from Llama, published October 12th. It’s called Call Grep Agent. And the code lives in the next plaid repo on GitHub, about 585 stars right now. Apache 2.0
00:30 license. Three names get mixed up. So, once Call Grep is the semantic code search engine, Call Grep Agent 2B is the model, and Call Grep {dash} {dash} Agent is the command that runs the model. When Claude Code needs something in your repo, it grabs, reads files, and grabs again, and every turn costs API tokens. But this one question goes in, the small model searches locally, and file paths and line ranges come back. Here is where the 34% comes from. 13 public repos, 52 questions, each run three times, so 624
01:03 sessions. Claude Code ran on Sonnet, and Opus graded the answers. Plain Grep, 142,000 tokens per question. Call Grep Agent, 93,000, that is 34% fewer tokens, and cost goes from 7.3 cents to 4.4. But correct answers went from 83% to 81, and they say that gap is not significant, though the interval can’t rule out a real drop. And time per question went from 13 seconds to 21.6. Cheaper, yes. Faster, no. And this is code search, not full software engineering tasks. They
01:35 say that themselves. Now, does it run on this laptop? This is the GGML page. The default is the Q8 file, about 2.7 GB, and there’s Q4KM at 1.56. If memory is tight, it’s Apache 2.0. The title says 2B. The GGML metadata says 3B. Okay. Install on Windows is the PowerShell installer, then Call Grep {dash} {dash} install agent. No Python, no server, no API key. My first try failed with out of memory while downloading llama.cpp because I
02:05 had only about 2 GB of RAM free. I ran it again and it went through. Then it printed GPU AMD Radeon graphics. That’s the integrated GPU in this laptop, not the RTX 4060. So, keep that in mind for the speed numbers. My test repo is the next plaid repo itself, about 109,000 lines of Rust and Python. Indexing took 513 seconds, about 8 and 1/2 minutes one time. Six questions, same Claude code, Sonnet, fresh session each. Run A uses normal grep, glob, and read. Run B tells Claude
02:38 to call grep-dash-dash agent first and read only the lines it returns. One difference from the blog, they installed the Claude code hook. I didn’t want to touch my global config, so I put that instruction in the system prompt, same idea, not their exact setup. And it’s six questions, one run each. That’s a small test. Question one, without the agent, five turns, 19 seconds, 21 cents. With the agent, three turns, 20 cents. Barely different, but the time was 125 seconds because the local agent call ran on the integrated GPU.
03:09 Question three, how does call grep pick which llama.cpp build to download? Without the agent, 267,000 tokens, 17 cents. With the agent, 93,000 tokens, under 10 cents. That is the win. Question five, where is the search route in the server? With the agent, 292,000 tokens, 15 cents, 207 seconds. Without, 145,000 tokens, 12 cents, 15 seconds. That is the loss. And the answers on question one, the agent version said, “I did not locate it
03:40 or this CLI arg wiring.” The normal version listed the enum, the parser, the stored field, and the CLI handler. Totals over six questions. Tokens, 1.03 million without, 944,000 with 8% fewer cost 86 cents without 75 with about 13% cheaper. Light on reported 34 and 40 on my small test. I did not get close. It was cheaper on five of six questions and used fewer tokens on four of six time. 98 seconds total without 646 with about
04:11 six and a half times slower. Correct answers checked by me against the code six of six without the agent four complete and two partial with it. And these token counts include about 135,000 tokens of fixed overhead per question from my own Claude code setup the same in both runs. So the real percentage would look bigger on a clean setup. My read, the direction matches their claim. The size doesn’t on this machine in this test. They report the savings grow with repo size 19% under 50,000 lines, 40%
04:43 for 150 to 600,000. So on a small project it’s barely worth the setup and the extra seconds. On a big code base where you pay per token it’s worth trying. Claude code has full hook support, open code, Codex and Hermes are in their own words basic for now. If you’re looking for cheap and reliable GPUs, I use RunPod and the link is in the description. That’s call grab agent, repo and model links are below. Tell me in the comments what repo size you would test it on and subscribe if you want the next small coding agent model test.