Can a 1B AI Model Actually See? I Tested 29 Task

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Can a 1B AI Model Actually See? I Tested 29 Task · Channel: Zero to MVP · 05:59 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, Local and open-source models, Multimodal models

Summary

The video tests MiniCPM V4.7 in 29 tasks, comparing its 1-billion-parameter version with a 35-billion-parameter version. The smaller model performs well at reading text, recognizing objects, and tracking events in video, but struggles to compare images, interpret charts, and complete some reasoning and browser tasks. It consistently passes 14 tasks, compared with 25 for the larger model. The presenter concludes that a 1-billion-parameter model can be useful for on-device visual tasks, but higher-accuracy work may require a larger model.

Key points

The tests compare MiniCPM V4.7 models with 1 billion and 35 billion parameters across 29 tasks and six difficulty levels.
The small model reads several kinds of text and tables well, but struggles to locate a specific table or read tiny text.
It can identify and count objects, but comparing images can lead to mistakes, such as failing to notice removed glasses.
The small model struggles to read values from charts and interpret a heat map and a mermaid diagram.
Both models failed a task asking how many hours had zero visitors.
Both models correctly tracked a video sequence and noticed when a lighter disappeared, though they misidentified an AirPods case.
The small model completed nine of 12 browser tasks, while the larger model completed all 12.
The model consistently passed 14 of 29 tasks; the 35-billion-parameter model passed 25.
The presenter says the smaller model can be useful for local document reading, object recognition, simple screenshots, and video tracking, while demanding tasks may need a larger model.

Chapters

Start	Chapter	Summary
00:00	The test and reading tasks	The presenter introduces the 29-task comparison and finds that the small model reads text and tables well but struggles to find specific information.
01:34	Object recognition and charts	The small model identifies some objects but struggles with image comparisons, chart values, and diagrams.
02:42	Reasoning and video understanding	The model gives inconsistent answers on some reasoning tasks, yet both versions track a sequence of events in a video.
03:51	Taking action and version comparison	The small model completes nine of 12 browser tasks, and the presenter compares V4.7 with the previous version and describes visual compression.
04:55	Results and takeaways	The presenter reports the overall scores and explains where a 1-billion-parameter model is useful and where a larger model is preferable.

Quotes

The small model can read text, but finding the right text is a different story. — 01:03

The model can recognize individual objects, but comparing images is much harder for it. — 02:07

Even the small model correctly identified the sequence and noticed when the lighter disappeared. — 03:15

To my surprise, more than I expected. — 04:55

1 billion parameters can already be quite useful — 05:26

Concepts and entities

MiniCPM-V 4.7, MiniCPM V4.6, OpenBMB, Raspberry Pi, CSV, Compressão de informações visuais

Full transcript (original language)

00:00 This AI model is so small that it can potentially run on a phone or even a Raspberry Pi. But can it actually understand what it sees? In my tests, it understood what was happening in a video and even controlled a browser. But when a pair of glasses disappeared from a table, it failed to notice. So, I put MiniCPM V4.7 to the test with 29 tasks to find out what a 1 billion parameter model can really do. And some of the results surprised me. For testing, I

00:30 chose the new MiniCPM V4.7 from OpenBMB. I will compare two versions, a small one with 1 billion parameters and a larger one with 35 billion. All tasks are divided into six difficulty levels and I run each task three times. We will start with simple tasks and finish with the ones that even large models might struggle with. So, level one. Here, I am testing how well the model can read text, numbers in a GitHub screenshot, my handwriting, tables from a research

01:03 paper. And the small model reads all of them confidently. It even converted a table to CSV without making any mistakes in the numbers. [clears throat] But when I asked it to find a specific table among several similar ones, the model simply transcribed half the page. Neither model could correctly read the tiny text on an SSD. So, what are the conclusions after the first group of reading related tests? My conclusion is as follows. The small model can read text, but finding the right text is a different story.

01:34 Okay, moving on to level two. Here, I am testing how accurately the model can identify objects. It correctly counted eight objects on the table. It also found the star button in a screenshot. But when I removed the glasses from the table, the model decided that the comb had disappeared. However, the large model handled the task with no problem. And it called a bright yellow space bar on the keyboard gray. And again, no issues with this for the larger model. And this is where things get interesting. The model can

02:07 recognize individual objects, but comparing images is much harder for it. Now, level three, understanding charts and diagrams. If a value is labeled directly above a bar, the model read it correctly. But when it has to determine a number from its position on a chart, problems start. Out of 32 values on a daily chart, it identified only 13 correctly. On a heat map, it answered 26 instead of 89 all three times. And it got the mermaid diagram wrong. The larger 35 billion parameter model

02:42 handled all of these tasks. Okay, moving on. Level four, reasoning. The small model can calculate the difference between two numbers in a table, but when I asked it to identify the days with the fewest views, I got three different answers across three runs. And the task of determining how many hours had zero visitors turned out to be too difficult for both models. Now, we are moving on to the most interesting and challenging part, video. I recorded a video where I placed objects on a table and then removed a

03:15 lighter. The model needs to not only recognize the objects, but also reconstruct the sequence of events. And this is where I expected it to fail, but both models succeeded. Even the small model correctly identified the sequence and noticed when the lighter disappeared. However, neither model could correctly identify the AirPods case, although both managed to track the object itself. By the way, the video test pleasantly surprised me and were unexpected. For a model with 1 billion parameters, that is an unexpectedly impressive result. And finally, level six. Here, the models

03:51 need to perform a specific action. Now, we will test whether the model can do more than just look at things and actually take action. The small model generated an almost correct function call based on a screenshot, but it left out the repository owner, so the call would not work. However, in the browser, it managed to open the correct sections, notes, and projects in nine out of 12 tasks. All the mistakes were related to scrolling down the page and finding a link. The larger model successfully completed all 12 tasks.

04:25 By the way, MiniCPM-V supports compressing visual information by up to 16 times, which saves resources with relatively little loss in accuracy. Not a bad trade-off for low-powered hardware. I also compared the new version 4.7 with the previous 4.6. Across 10 challenging tasks involving charts, diagrams, and an AI agent, the older version did not pass a single one, but the new version passed five. That is a major improvement. All right, I think

04:55 it’s time to wrap things up. So, how much can a 1 billion parameter model actually see? To my surprise, more than I expected. Out of 29 tasks, the small model consistently passed 14, and the larger 35 billion parameter model passed it almost all of them, 25 out of 29. And my main takeaway from this testing is this. If you need to read documents locally, recognize objects, analyze simple screenshots, or track what is happening in a video, 1 billion

05:26 parameters can already be quite useful, and you can do all of this directly on your device without sending your images to the cloud. But if you need high accuracy, especially when analyzing charts, documents, or working with an AI agent, 1 billion parameters is not enough yet. That is where it makes sense to to a larger model. I’ll leave the official links to both MiniCPM v2.7 models in the description. Try them yourself, explore the repository, and see how they perform on your own tasks. And if you enjoy real-world tests of AI

05:59 models like this, subscribe to the channel and my weekly newsletter. See you soon. Take care.