What Local AI Models Can a Regular Computer Run?

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: What Local AI Models Can a Regular Computer Run? · Channel: Zero to MVP · 11:53 · Watch on YouTube

Topics: AI evaluation and benchmarks, Model inference and optimization, AI infrastructure and hardware, Local and open-source models

Summary

The video tests which local AI models can run on a regular modern computer, using a GEEKOM A9 Max Mini PC with Ollama and 24 GB of video memory allocated in the BIOS. The presenter runs models from small to large with the same prompt and default settings, measuring token-generation speed and checking system resource use. Several models run at acceptable speeds, including Gemma 4 with 26 billion parameters and Qwen 3.6 with 36 billion parameters; Qwen 3.8 with 27 billion parameters also runs at nearly 14 tokens per second. The presenter plans to keep Qwen 3.8 and Gemma 4 with 26 billion parameters, noting that all tests used Q4KM quantization and were not performance-tuned.

Key points

The test computer is a GEEKOM A9 Max Mini PC, used headlessly over SSH and tmux.
The presenter chose Ollama, noting that it uses the llama.cpp library internally and that their tests found no meaningful performance difference.
The BIOS setting allows video memory to be adjusted; the presenter allocated 24 GB.
Gemma 4 E2B has 5 billion parameters, while the “2B” refers to its effective parameters per token.
The smallest Gemma 4 model generated at more than 46 tokens per second.
Gemma 4 with 12 billion dense parameters ran at just over 10 tokens per second.
The 20-billion-parameter model labeled GPT-4 ran at more than 21 tokens per second.
The mixture-of-experts Gemma 4 (26B) and Qwen 3.6 (36B) reached almost 31 and 38 tokens per second, respectively.
Qwen 3.8 with 27 billion parameters ran at almost 14 tokens per second and fit within the allocated video memory.
All models used Q4KM quantization and default parameters; the presenter selected Qwen 3.8 and Gemma 4 (26B) to keep on the computer.

Chapters

Start	Chapter	Summary
00:00	Choosing the computer and testing setup	The presenter introduces the question of which local models a regular computer can run and explains the mini PC, memory allocation, and choice of Ollama.
02:04	Small and medium-sized models	The presenter tests small Gemma 4 and then a 12-billion-parameter version, checking generation speed, GPU use, memory, and fan noise.
05:40	Larger models and mixture of experts	The tests move to the 20-billion-parameter GPT-4 model and then to Gemma 4 (26B) and Qwen 3.6 (36B), with their generation speeds compared.
09:15	Testing Qwen 3.8 and drawing conclusions	The presenter confirms Qwen 3.8 fits in video memory, reviews the test settings, and chooses it and Gemma 4 (26B) to keep on the computer.

Quotes

in the BIOS, you can configure how the memory is divided between system RAM and video memory. — 01:02

a speed of more than 46 token per second shows that the hardware performs very well with this model. — 03:05

Oh, this is even better than Gemma, almost 38 tokens per second compared to 31 tokens with Gemma. — 08:44

The generation speed turned out to be almost 14 tokens per second. — 10:19

So, based on today’s test, I’ll keep two models on this computer. — 11:21

Concepts and entities

GEEKOM A9 Max, Ollama, llama.cpp, vLLM, tmux, btop, Gemma 4 E2B, Gemma 4, GPT-4, Qwen3.6, Qwen3.8, Mixture of Experts, Q4_K_M, Memória de vídeo

Full transcript (original language)

00:00 Recently, I talked about how I run and use small LLMs on my Zima Cube. Also, from time to time, when testing various larger local models, I use my desktop with a fairly powerful graphics card and more than 100 GB of RAM. However, recently, I became curious in one interesting question. What models can you run on a regular computer? In other words, what models can you use on the kind of computer that most people have at home? To put it another way, a computer that can be an everyday

00:30 workhorse. I won’t bore you with the long process of choosing one. I’ll just tell you what I ended up with and why. So, my choice was the A9 Max Mini PC from GEEKOM. In a few words, here’s why I chose it. First, I didn’t need another full-size PC. I already have one. So, the small form factor is important for me. Second, as far as I understand, it was designed with local models in mind. So, the both the processor and the graphics are powerful and quite well-balanced. It also has a very solid

01:02 active cooling system specifically designed for this hardware. And third, and this was probably the deciding factor for me, in the BIOS, you can configure how the memory is divided between system RAM and video memory. This way, I’ll have better control over the size of the model I can run. By the way, I’ll be using this mini PC in headless mode today and connect it to it over the local network. However, I keep it on my desk next to me for now to see how much it heats up under heavy loads.

01:33 And as always, I’ll use SSH and tmux to connect to it. Okay, I’ll probably start by installing everything I need. First of all, that will be Ollama for running local models. And I want to immediately answer the questions of why not llama.cpp or something similar that works at a lower level. And the reason is simple. Ollama uses the same llama.cpp library internally and lately they have optimized everything so well that there is almost no overhead when using a

02:04 llama. I run some tests which I’ll make a separate video about soon and the difference was within the margin of error. In other words, there is was practically no difference. So for me, there isn’t much benefit in using llama.cpp or vllm. Okay, I’ll start with small models and gradually move on to larger one. This way, I want to find the point where the quality of the answers is as high as possible while the response speed is still acceptable. The first model I use is Gemma 4, its

02:34 smallest version. By the way, pay attention to the name. It’s E2B and many people think this means it has 2 billion parameters, but that’s actually not the case. It has 5 billion parameters while the 2 billions in the name refers to the number of effective parameters. These are the parameters that participate in the calculations at each token generation step. Because of this architecture, the model runs faster. But let’s get back to our testing. I’m downloading this model and running it in

03:05 verbose mode so that the metrics are displayed at the end. To make it easy to evaluate the model response speed, I’ll use a prompt that immediately starts generating a long continuous stream of text. It will generate a simple JSON file based on my description and I don’t think this task should be difficult for Gemma 4. Okay, as you can see, the model responds very quickly, but let’s take a look at the number at the end. So a speed of more than 46 token per second shows that the hardware performs very

03:36 well with this model. Great and this means that small models are not a problem for my computer. So it’s time to move on to larger ones. For the medium-size model, I also use one from the Gemma 4 family with 12 billion parameters. This is already a fully dense model. So, all 12 billion parameters will be used to generate each token. I’ll keep the prompt the same as for the smaller model because first, it’s important for me to understand the overall picture. And then, if necessary, I’ll dig deeper into the details. Okay,

04:08 while the model is busy generating the response, let’s take a look at the load this process creates. I’m switching to the second tab in tmux and launching btop. It’s a free open-source utility that, in addition to the usual CPU and memory usage metrics, also shows GPU load. As you can see, the GPU is currently fully loaded while the CPU isn’t. That’s good because it means all of the work is happening on the graphics card as it should. I’m also happy to see that there is a lot of free video memory

04:39 left. As you may remember, I set it to 24 GB in the BIOS. So, almost 2/3 of it is still free. And that means I can continue at testing larger models. The cooling system has also started working more actively, and I can hear the fan a little now. This is important to me because, at least for now, I plan to keep this computer on my desk. Personally, I would describe the noise level as quiet. When I’m sitting about an arm’s length away, I can hear it. But I think if it were sitting on the other

05:09 side of the room, I probably wouldn’t notice it. Here, listen to how it sounds from closer range. Okay, generation is finished. So, let’s look at the results. This time, the number of tokens per second is significantly lower than with the previous model, just over 10 tokens per second. But that’s not surprising because of this model is not only

05:40 larger, but also has a different architecture. And all 12 billion parameters are used to generate each token rather than only a portion of them, like with the previous model. But, nevertheless, the speed is perfectly fine and acceptable for me. So, I won’t stop at this model and will move on to the next one. Okay, let’s move away from the Gemma family now, and I’ll try a 20 billion parameter model from OpenAI. This is GPT-4 and I think you are probably quite familiar with it. I’m

06:11 downloading it, launching it, and trying the same it generates. Again, while the model is busy generating the response, let’s take a look at the resource usage. Yes, everything looks good with this model as well. Uh the GPU is working, not the CPU. It’s good. And there’s still plenty of memory available, too. Great. Okay, I won’t show the entire generation process because it’s pretty boring, and I’ll jump straight to the results.

06:41 Oh, and the results are quite good. The generation speed is more than 21 tokens per second, which is twice as fast as the previous model. That’s a very impressive result for a 20 billion parameter model. Great. I’ll definitely keep this model here and use it. By the way, if you enjoy experiments like these with local models and want to support the growth of the channel, you can subscribe to my Patreon. I also post more personal videos and materials there that usually don’t appear on the main channel. I’ll leave the link in the

07:12 description. Okay, I think it’s time to move on to even larger models. From this category, I’ll try two more models that have performed well for me before. These are Gemma 4 with 26 billion parameters and Qwen 3.6 with 36 billion parameters. Thanks to their special architecture, mixture of experts, both of these models use only around three or four billion of their parameters when generating each token, rather than all of them. This

07:42 allows them to be quite fast while still producing high-quality answers. First, I’ll try Gemma 4 with 26 billion parameters. Everything as usual. I’m downloading the model and asking it to generate the JSON. You can already tell from the response speed that it works significantly faster than the previous model, but that’s not surprising since only part of the parameters are used on each step. Okay, what about the results? And the results are very good, almost 31

08:13 tokens per second. Great. It’s about third faster than GPT-4 with its 20 billion parameters. Okay, now it’s time for the Qwen 3.6 model with 36 billion parameters. I’m downloading it and giving it the same prompt. This model is quite large and heavy, so it will be interesting to look at the resource usage. Well, everything here is almost perfect. The model fits within the video memory allocation to it, just like all the previous models. Most of the load on

08:44 the GPU. Excellent. Now, let’s look at the results. Oh, this is even better than Gemma, almost 38 tokens per second compared to 31 tokens with Gemma. I didn’t even expect it to be higher. I thought it would be about the same. A nice surprise. By the way, here you may notice that Qwen’s total response generation time is almost twice as long as Gemma’s. This may seem strange considering its higher generation speed, but the reason is that after Qwen generated the required JSON,

09:15 it also decided to run additional checks to make sure the file was correct, as well as check whether any information was duplicated across the JSON elements. That’s why the whole process ended up taking so much time. Okay, we figured out this category of models. The results are good, and from the resource usage, I can see that this is already close to the limit, but for the sake of completeness, I’ll still try running one more heavyweight model. And that will be the recently released Qwen 3.8 with 27

09:47 billion parameters. It turned out to be a very successful model, and I recently made a video about my first impression of it. Be sure to check it out if you haven’t seen it yet. The process with this model is the same as with the previous one. I am downloading the model, launching it, and sending the same prompt to generate JSON. The first thing I am happy about is that the model actually launched, and it’s not a small one. But let’s immediately check whether there was enough video memory for it. I am switching to the second tab, where I

10:19 already have Btop running, and let’s take a look. Yes, everything is fine. There was enough video memory. Great. Now it’s time for the final results. So, the results, and they are unexpectedly good. The generation speed turned out to be almost 14 tokens per second. Of course, that’s slower than with the previous models, but it’s still acceptable for me. And on top of that, this is already the next generation of Qwen models, and they are much more powerful and advanced. So, yeah, I am

10:50 really happy with this result. It means you don’t need a powerful desktop to run Qwen 3.8, and this model is perfectly suitable for regular modern computers. Okay, I think I have now figured out what I want to know, and it’s time to sum things up. But before moving on to the conclusions, I also want to mention that all the models used Q4KM quantization. I also intentionally run them with the default parameters. At this stage, my goal wasn’t to squeeze

11:21 the maximum performance out of any particular model, but simply to understand which set of models makes sense to use on a regular modern computer. Fine-tuning will be the next step, and that’s exactly why for now I was paying attention to only one matrix in the results, namely token generation speed. So, based on today’s test, I’ll keep two models on this computer. The first is quant 3.8, which was the clear winner today. It’s a new generation of models, and the speed is also acceptable. I think that after some

11:53 fine-tuning, the speed could be even higher. The second model is Gemma 4 with 26 billion parameters. Since it comes from a different model family, and its speed is also significantly higher, it will be an excellent complement to quant. Later, I’ll definitely make another video about using this model together. And that’s all for today. Don’t forget to subscribe to the channel and to my weekly newsletter about IT and AI. The link in the description. See you soon. Take care.