8 Local AI Models on a Raspberry Pi 5: Only 3 Are Usable

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: 8 Local AI Models on a Raspberry Pi 5: Only 3 Are Usable · Channel: Zero to MVP · 09:52 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, Model inference and optimization, AI infrastructure and hardware, Local and open-source models

Summary

The video tests eight local AI models on a Raspberry Pi 5 with 8 GB of RAM, measuring response speed and performance on four practical tasks. The presenter defines chat usability by speed and startup time, and background usability by task success and running without overheating or throttling. Five models meet the chat speed criteria, but only three pass all four tasks: Granite, Gemma 4 E2B, and Gemma 4 E4B. The presenter recommends Gemma 4 E2B or Granite for chat, Gemma 4 E4B for background work, and Gemma 4 E2B as the single model to keep.

Key points

The Raspberry Pi 5 setup has 8 GB of RAM, an active cooler, and two SSDs; all models run through llama.cpp.
For chat, the presenter requires around five tokens per second and a response starting within 10 seconds.
Background models must pass all four tasks and run for half an hour without causing overheating or throttling.
Qwen 3.5 with 0.8 billion parameters is the fastest model, at almost 15 tokens per second.
LFM 2.5 generates at more than 10 tokens per second, but its reasoning means the first words take 17 seconds to appear.
Using two CPU threads made most models generate text 18 to 38 percent faster; Bonsai was the exception.
The four tasks were summarization, sorting files, converting an order email to JSON, and calling a tool.
Granite, Gemma 4 E2B, and Gemma 4 E4B passed all four tasks.
The presenter recommends Gemma 4 E2B and Granite for chat, and Gemma 4 E4B for background tasks.

Chapters

Start	Chapter	Summary
00:00	Usability criteria and test setup	The presenter defines what counts as usable for chat and background work, then describes the Raspberry Pi 5 setup and model testing conditions.
01:33	The eight models and their speeds	The video introduces the models and compares their response generation speeds, startup times, and performance with different thread counts.
06:15	Four practical task tests	The models are tested on summarization, file sorting, JSON output, and tool use, with only three passing every task.
08:52	Final recommendations	The presenter recommends models for chat and background tasks and names Gemma 4 E2B as the one model to keep.

Quotes

just because a model runs doesn’t mean it’s actually usable. — 00:00

I’ll consider a model usable for chat if it can generate at least around five tokens per second. — 00:32

the practical takeaway for me is simple, use two threads. — 05:14

At these speeds, even a simple function takes too long to generate. — 08:20

If that isn’t critical for you, you can try them with tool use, but I wouldn’t use other models on Raspberry Pi 5 — 09:22

Concepts and entities

Raspberry Pi 5, llama.cpp, llama.bench, Q4_K_M, Qwen3.5, Bonsai, MiniCPM-5, Granite, Gemma 3n E2B, Gemma 3n E4B, LFM 2.5

Full transcript (original language)

00:00 There are already plenty of small local models that can physically run on Raspberry Pi 5, but just because a model runs doesn’t mean it’s actually usable. So, today I’m going to try to answer two questions. First, is there any real value in running an LLM on a Raspberry Pi? And second, which model should you use? So, I tested eight local models on my Raspberry Pi 5 in order to answer these questions. I measured the speed of each one in tokens per second and gave them the kind of work that a small model

00:32 should realistically be able to handle. So, by the end of this video, you’ll know which ones are actually usable, which are only suitable for background tasks, and which ones you can probably skip. First, let me explain what I mean by usable so that my conclusions at the end aren’t based on how I happen to feel, but on objective criteria. So, I’ll consider a model usable for chat if it can generate at least around five tokens per second. That’s roughly the minimum speed at which a person can read

01:03 a response reasonably comfortably. Another important criterion will be how quickly it starts answering a short question. That should take no more than 10 seconds. Okay, now let’s talk about using models in the background. Here, speed isn’t that important. However, the model has to pass all of the test tasks and run for half an hour without causing the Raspberry Pi to overheat or throttle. If a model doesn’t meet these criteria and can’t work either in chat mode or in the background, then I

01:33 consider it unusable for use on a Raspberry Pi. Okay, a few words about today’s setup. I’ll be using a Raspberry Pi 5 with 8 GB of RAM with an active cooler and two SSDs connected to it. The operating system is installed on the first SSD and the Pi boots from it, while the models are stored on the second one. All of the models run through llama.cpp, the same agent used under the hood by LM Studio and Ollama. It includes a benchmarking tool called

02:04 llama.bench, which give us two number that we’ll care about today. How quickly the model processes your prompt and how quickly it generate the response. All of the models except one use the same 4-bit quantization, Q4_KM, and they all use the same context size. The exception is Bonsai. It was released as a 1-bit model from the start, but I’ll talk more about that one later. I disabled reasoning mode everywhere it could be disabled. For short tasks, it isn’t necessary, it

02:36 only wastes time. But one model, LFM 2.5, always reasons. So, I include that reasoning time in its results. Okay, here’s the list of participants ordered from the smallest model file to the largest. They are Qwen 3.5 with 0.8 billion parameters, Bonsai with 8 billion parameters, MiniCPM-5 with 2 billion parameters, Granite 4.2 with 3 billion parameters, Qwen 3.5 with 4 billion parameters, Gemma 4e2b,

03:07 Gemma 4e4b, and LFM 2.5 with 8 billion parameters. As you can see, there’s a pretty wide range in parameters counts. From a tiny model with less than 1 billion parameters all the way up to 8 billion. Almost all of them were released this year, and these are some of the most popular small models that can fit with 8 GB of memory. Two of the model here are unusual. The first is Bonsai with 8 billion parameters, but only one bit per weight. So, the entire model file is

03:39 just a little over 1 GB. The other is LFM 2.5. It has the largest file in the list, but thanks to its special mixture of expert architecture, it doesn’t use all of its parameters when generating each token, only around 1 and 1/2 billion. That allows it to stay fairly fast and run reasonably well on a Raspberry Pi. Okay, now let’s move on to the actual testing. We’ll start with response generation speed. I measured each number several times because on a Raspberry Pi,

04:11 one run can differ quite noticeably from another. The fastest model is the smallest one, Qwen 3.5 with 0.8 billion parameters, almost 15 tokens per second. In second place in the largest model, LFM 2.5. It runs at more than 10 tokens per second, but keep in mind that only 1 and 1/2 billion parameters are active in this model. Next come Mini CPM 5 and Gemma 4 82B, around seven. Bonsai and

04:42 Granite are slightly above the threshold. And Gemma 4 84B and Qwen 3.5 with 4 billion parameters are at around 3 and 1/2. That’s slower than the limit I defined at the beginning. Remember, it needs to be at least five. In practice, it looks like this. At 15 tokens per second, the text appears faster than you can read it. At seven, it’s a comfortable pace. And at 3 and 1/2, you are waiting for every word. And one thing I didn’t expect, almost all of these numbers were measured using two

05:14 CPU cores, not four. With two threads, the models generate text 18 to 38 percent faster. The exception is Bonsai, it needs all four. I think that in this case, generation on the Raspberry is limited by memory bandwidth rather than CPU performance. So, the practical takeaway for me is simple, use two threads. The second criterion I defined was how quickly the response starts. All of the models except one begin responding within 2 and 1/2 seconds or

05:44 less. The exception is LFM 2.5. It always reasons first, and the first words only appear after 17 seconds. So, overall, five out of eight models passed the threshold for chat. Qwen 3.5 with 0.8 billion parameters, MiniCPM-5, Gemma 4 e2b, Bonsai, and Granite. Gemma 4 e4b and Qwen 3.5 with 4 billion parameters are too slow for chat, while LFM 2.5 spends too long reasoning. But, speed

06:15 alone doesn’t make a model usable. A fast model that gives the wrong answer is worse than a slow model that gives the right one, especially for background tasks. So, now let’s move on to evaluating what kind of tasks these models are actually capable of handling. So, each model got the same four tasks. They weren’t chosen randomly. These are useful tasks that a Raspberry Pi can realistically handle. There are no partial points. The answer either works or it doesn’t. Task one, summarization.

06:48 An 800-word article summarized into three bullet points. All the facts in the article are real, not made up. And here, four out of eight models failed. Qwen 3.5 with 4 billion parameters wrote something that wasn’t in the article. LFM 2.5 changed one of the facts. Bonsai added something that wasn’t there. And MiniCPM-5 missed the main point of the article. Task two, sort 25 files from the download folder into six folders.

07:19 This is a typical job for a home server. Here, almost all of them handled it. So, overall, everything was fine. But, two models failed. Qwen 3.5 with 0.8 billion parameters lost eight files, while LFM 2.5 spent 5 minutes thinking and then simply forgot about two files. Task three, turn an order email into JSON. The criterion here is simple. The output has to be valid JSON because otherwise you can’t reliably do anything with that

07:50 response afterward. And here’s the surprise. Every model produced valid JSON. However, some models made mistakes in the content. Those were the smallest model in today’s test, Qwen 3.5 with 0.8 billion parameters, as well as MiniCPM 5. Task four, call a tool. It’s a single function, move a file into a folder with three files processed one after another. If a model can’t call tools, you can’t build an agent around it. MiniCPM 5

08:20 failed here. It replied that it was moving the file now and did nothing. And LFM 2.5 tried to put a photo into archive folder that doesn’t exist. And one important moment, I deliberately didn’t include a coding test. At these speeds, even a simple function takes too long to generate. So, using models like these for coding doesn’t really make sense. And in the [clears throat] end, three models passed all four tasks. Granite, Gemma 4 E2B, and Gemma 4 E4B.

08:52 Bonsai and Qwen 3.5 with 4 billion parameters failed only the summarization task. You can try them to organize files, but not to write summaries. So, I think it’s time to wrap things up and decide which model are actually usable on a Raspberry Pi 5. For chat, I would choose Gemma 4 E2B and Granite with 3 billion parameters. They are fast enough and passed all four tasks. For background tasks, definitely Gemma 4 E4B. It handled everything, but

09:22 unfortunately it’s too slow for chat. Bonsai and Qwen 3.5 with 4 billion parameter almost made it to the final list. They handle everything except text summarization. If that isn’t critical for you, you can try them with tool use, but I wouldn’t use other models on Raspberry Pi 5 because they didn’t pass my tests. And if I could keep only one model on this Raspberry Pi, I would keep Gemma 4 82B. It passed everything and it’s the fastest of the finalists. And

09:52 that’s it for today. Don’t forget to subscribe to the channel and to my weekly newsletter about IT and AI. You’ll find the link in the description. See you soon. Take care.