A 35B Model on an iPhone Using 3GB? I Checked the Claim

Author: Prompt Engineer 48 · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: A 35B Model on an iPhone Using 3GB? I Checked the Claim · Channel: Prompt Engineer 48 · 08:22 · Watch on YouTube

Topics: AI evaluation and benchmarks, Model inference and optimization, AI infrastructure and hardware, Local and open-source models, Multimodal models

Summary

The video checks claims about MiniCPM-V 4.7 against its GitHub and Hugging Face pages, benchmark tables, and a local test of the 1B model. It explains how look-ahead expert scheduling prefetches experts while preserving the model’s normal router decisions, but treats byte-for-byte identical outputs as an unverified claim. The presenter finds a mismatch in the announcement’s OCR benchmark comparison: the 5.2-point lead applies to 4× compression, not 16×. On an 8-GB laptop GPU, the 1B model uses about 3 GB of VRAM; it reads a receipt better at 4× than 16× compression and gives a broadly accurate description of a football clip with one invented name.

Key points

The repository lists an Apache 2.0 license; the presenter notes that no 4.7 GitHub release tag was available when the pages were checked.
The 35B A3B model has about 3 billion active parameters per token out of 35.2 billion total; the smaller model is about 1.55 billion parameters.
Look-ahead expert scheduling predicts which experts the next layer will need and preloads them while the current layer is computing; actual computation still follows the router.
The presenter could not find an exact statement that outputs are byte-for-byte identical, though the prefetch-only design makes the claim plausible.
The reported 13+ tokens per second on iPhone combines LES, layer-wise quantization, and DeepSpark speculative sampling; the reported 5× speedup was measured on an iPad Pro M4 with 16 GB.
The OCR benchmark comparison is 90.4 versus 86.0 at 16× compression, a 4.4-point gap; the 5.2-point gap comes from the 4× score of 91.2.
Across 34 benchmarks, the model leads in 15 of 26 vision rows and is best in two of eight language rows, while trailing on AIM, Life Code Bench, and IF Bench.
On an RTX 4060 laptop GPU with 8 GB VRAM, the 1B model uses about 3.11 GB in BFloat16 and runs at roughly 8.6 tokens per second in the receipt test.
In the receipt test, 4× compression correctly extracted the items and totals, while 16× compression introduced errors and ran to the 300-token limit.
The football-clip test used 16× compression and peaked at 4.23 GB VRAM; the model identified the score and teams but misnamed BHA as the Blue Helmets.

Chapters

Start	Chapter	Summary
00:00	Checking the model and release claims	The presenter reviews the repository and model collection, then explains the 35B and 1B parameter counts and how look-ahead expert scheduling works.
02:35	Speed claims and benchmark discrepancy	The video examines demo speeds, image-token compression, and the OCR chart, finding that the announced 5.2-point lead belongs to 4× rather than 16× compression.
04:44	Local tests of the 1B model	On an 8-GB laptop GPU, the presenter compares receipt extraction at two compression settings and tests the model on a football clip.
06:50	Other features and final assessment	The presenter summarizes untested fine-tuning and hardware claims, separates confirmed claims from unverified ones, and gives recommendations based on the local tests.

Quotes

The computation still follows the real router — 02:05

The headline mixes the two columns. — 04:13

At four times, the same model reads all four items with the correct prices — 05:48

That is one image and one clip. So, take it as a first look, not a benchmark. — 06:50

the 5.2 gap at 16 times, which is really the four times number. — 07:51

Concepts and entities

MiniCPM-V 4.7, OpenBMB, Hugging Face, look-ahead expert scheduling (LES), Mixture of Experts, RTX 4060, Transformers, llama.cpp, SGLang, vLLM, Ollama

Full transcript (original language)

00:00 This is a 35-billion parameter model, and the repo says it generates 13 plus tokens per second on an iPhone using just 3 GB of memory. That is a screen recording from the phone itself. All right, in this video I’m going to check that claim against the actual GitHub page, the Hugging Face page, and the benchmark tables, and then run the 1B version on this 8-GB laptop GPU and show you the real numbers. And one number in the announcement does not match the chart. I’ll show you which one. So, this is the Open BMV repo. Mini CPMV, you can

00:31 see about 26 and 1/2 thousand stars, 2.1 thousand forks, 37 contributors, and the license says Apache 2.0. The readme was updated 2 hours before I captured this page, and the latest GitHub release is still from last year. So, there’s no 4.7 release tag yet. The weights are on Hugging Face. If we open the Hugging Face collection, you can see the 35B A3B model, the 1B model, GGUF versions of both, a small one called Deep Spark, and quantized versions in FP8, GPTQ, AWQ,

01:03 and INT4. Mind you, all of this was updated about 2 hours before I took this screenshot. So, the download numbers here are tiny. It is day one. The readme says the 35B A3B is the first mixture of experts model in the Mini CPMV series. A3B means about 3 billion parameters are active for each token out of 35.2 billion in total, and the 1B is not exactly 1 billion. The collection page rounds it to 2B, and the model data says about 1.55 billion. Here is the problem.

01:33 The readme describes a mixture of experts model has to keep all those expert weights somewhere, and a phone cannot hold 35 billion parameters in memory. You can leave the weights in storage and load an expert whenever the router asks for it. But then, every layer waits for a read, and decoding becomes too slow to use. You could also freeze a small set of experts for the whole prompt. That is fast. But the readme points out it can exclude experts the model would have chosen and that costs capability. The fix is called look-ahead expert scheduling, LES. While the phone is computing layer N, it

02:05 predicts which experts layer N + 1 will need and starts loading them right then. So, loading and computing overlap. Here is the part I wanted to check. If the prediction is wrong, does the answer change? The readme says no. Predictions only decide what to prefetch. The computation still follows the real router and any expert the prediction missed is loaded and used as normal. The announcement says the outputs are byte for byte identical. I could not find that exact phrase on the pages, but the prefetch only design is why it is believable. So, I’ll call it a claim.

02:35 Here’s the recording from the repo before and after optimization side by side. Look at the decode speed label under the answer. Before optimization, it stays at 2.0 tokens per second. After, it moves between about 6 and 12. And here is the 35B 83B on the iPad unedited. The prompt is a chart from 1812 asking for the coldest temperature in Celsius with a short explanation of the conversion. It finds -30° Réaumur on December 6th, multiplies by 1.25, and answers -37.5 Celsius. That is correct.

03:07 The 5x speedup in the announcement was measured on an iPad Pro M4 with 16 GB, and the announcement says 17+ tokens per second there. The repo page itself only gives the iPhone number. And remember, 13 tokens per second is LES, layer-wise quantization, and DeepSpark speculative sampling stacked together. So, you cannot copy LES onto another model and expect 5x. Second idea is image tokens. The model compresses visual tokens early inside the vision encoder, and you pick 4x for detail or 16x for cheaper

03:40 inference. In the Transformers example, it is one line, downsample mode = 16x. Now, the benchmark table. This is the vision-language chart from the repo, 35B 83B against QN 3.6 35B 83B QN 3. 27B, QN 3.8 27B, and Gemma 431B. Look at the OCR bench row at four times, it scores 91.2 and at 16 times, 90.4. QN 3.6 35B, A3B scores 86.0. So, here’s a number that

04:13 does not match. The announcement says that even at 16 times, it is 5.2 points ahead. 90.4 - 86 is 4.4. The 5.2 is the four times score, 91.2 - 86, still ahead. But, the headline mixes the two columns. Across the two charts, there are 26 vision rows and eight language rows. So, the 34 benchmarks in the announcement checks out. The vision chart highlights the new model as best in 15 of the 26 rows. On the language chart, it is best in two of eight, CMMLU and Towel Squared bench. And it is

04:44 behind on AIM, Life Code bench, and IF bench. So, it is a strong vision model that is competitive on text, not a clean sweep. And these are the labs’ own numbers from public reports and local evaluation, which is what the footnote says. Now, let’s go ahead and test this. Mind you, everything here is running on my local system, an RTX 4060 laptop GPU with 8 GB of VRAM. The 35B will not fit here, so I’m testing the 1B. And every number from now on belongs to the 1B. It loads in about 15 seconds once the files

05:15 are on disk and sits at 3.11 GB of VRAM in BFloat16. First test, I generated a receipt image myself, small monospace text, four items, a subtotal, GST, a total, and a UPI reference. Same prompt, two settings. At 16 times compression, you can see it writes dollar signs in front of every price, and there are no dollar signs on this receipt. It also puts a subtotal under cold brew and again under veg sandwich, and then it runs all the way to my 300 token limit. It did get the total 26.51 and the UPI

05:48 reference, right? At four times, the same model reads all four items with the correct prices, the subtotal, GST, the total, and the reference. No dollar signs, and it finishes in 111 tokens, 12.7 seconds. The 16 times run took 35 seconds because it ran to the limit. Speed is about 8.6 tokens per second in both cases on this card. I did not install the fast kernels that Transformers.ai asks for, so it is running a slower called back path, and I would not read this as the best this card can do. Second test video, the

06:20 football clip from the Hugging Face examples, 37 seconds, 16 times compression, up to 128 frames. That is 2607 prompt tokens, and peak VRAM goes to 4.23 GB. Look at the scoreboard, BHA against MCI 00 with the live badge. In the frames I sampled, the clock runs from 22:32 to 23:02, and the models range 22:29 to 23:05. is close. It got the score, the live badge, and the team codes. It also invented a name. It

06:50 called BHA the Blue Helmets, and the rest of the description is generic. The ball moves forward, players converge. 8.81 tokens per second. So, on this laptop, the 1B is usable, and in my receipt test, four times was the better setting for small text. That is one image and one clip. So, take it as a first look, not a benchmark. Two more things from the page, self-replay fine-tuning. The readme says it lets you add a capability cheaply while forgetting less of what the model already knows. I did not train anything,

07:20 so I cannot tell you how well it works. And hardware, the readme lists iOS, Android, and Harmony OS, and day zero support for Intel, AMD, ARM, Rockchip, and Homo. The announcement also names Flag OS, and I could not find that on the pages I checked. For software, it lists SGLang, vLLM, llama.cpp, and Ollama with GGUF, AWQ, GPTQ, and FP8 versions. So, here’s where I landed. Confirmed on the pages, Apache 2.0, 35B, 3B plus 1B LES as prefetch only, 13 plus

07:51 tokens per second with 3 GB on iPhone, and 34 benchmarks across the two charts. Not on the pages, byte-for-byte identical outputs, 17 plus on the iPad, and the 5.2 gap at 16 times, which is really the four times number. From my test, the 1B runs in about 3 GB of VRAM on a laptop, reads a receipt correctly at four times, and made mistakes at 16 times. If you want a small vision model on a laptop, try the 1B and use four times for text-heavy images. If you want the 35B, you need a big GPU or a recent

08:22 iPhone or iPad, and it is day one, so expect rough edges. If you are looking for cheap and reliable GPUs, I use RunPod, and the link is in the description. So, in this video, we checked the MiniCPM-V 4.7 claims against the repo, the Hugging Face page, and the charts, and tested the 1B on this laptop. Try it on your own system, and let me know how it goes. Next, I’ll put the 35B on a rented GPU and run the same prompts. I’ll see you in another one.