A 125B Local Model on a 16GB GPU: Fast, But Is It Smart?

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: A 125B Local Model on a 16GB GPU: Fast, But Is It Smart? · Channel: Zero to MVP · 06:47 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, Model inference and optimization, AI infrastructure and hardware, Local and open-source models

Summary

The video tests Strata, a project that runs the 125-billion-parameter Frontier model When 3.8 Flash Next on a consumer GPU with 16 GB of VRAM. On an RTX 4060 Ti, the setup averaged about 65 tokens per second, but it also used substantial system resources and required at least 64 GB of RAM. The model successfully built a clock, stopwatch, timer, and pixel editor, but struggled with a more challenging text-based 3D pyramid task. Overall, the creator’s first impressions were positive, though further testing is planned.

Key points

The test uses an RTX 4060 Ti with 16 GB of VRAM; Strata also requires 64 GB of system RAM.
When 3.8 Flash Next is a mixture-of-experts model with more than 24,000 experts, of which 10 are used for each token.
The model load takes about 30 seconds, and the system is largely dedicated to running it while active.
A benchmark across text and code prompts averaged around 65 tokens per second.
Using the Pi agent without a prior plan, the model completed a clock, stopwatch, and timer webpage successfully.
The model built a working pixel editor in 5 minutes and 20 seconds, using substantial GPU, RAM, and CPU resources.
The model understood the text-character pyramid task but did not complete it successfully.
The creator’s initial impression was better than expected, with more testing planned over the following weeks.

Chapters

Start	Chapter	Summary
00:00	Strata and the test setup	The creator introduces Strata, chooses a 16 GB RTX 4060 Ti for testing, and explains the system RAM requirement and mixture-of-experts design.
01:31	Building Strata and benchmarking performance	The creator builds the agent, notes its heavy use of system resources, and measures an average speed of about 65 tokens per second.
03:06	Clock app and pixel editor tests	The model completes a simple clock-and-timer webpage and then builds a functional pixel editor in 5 minutes and 20 seconds.
05:12	The pyramid task and first conclusions	The model makes a promising but unsuccessful attempt at a text-based 3D pyramid, leaving the creator with positive initial impressions.

Quotes

I’m not as interested in simply getting the model running as I am in whether it can actually handle practical tasks. — 00:00

A good GPU alone is not enough. You also need a lot of RAM. — 01:01

Okay, I got an average of around 65 tokens per second. — 02:35

I did not expect the model to complete this task successfully on the first try. — 04:39

But my first impressions are very positive. — 06:47

Concepts and entities

Strata, Frontier, Mixture of Experts, RTX 4060 Ti, Pi, GitHub, Opus 5.5, GPT-6, Anthropic, OpenAI

Full transcript (original language)

00:00 A few days ago, a project called Strata was released that let you run the 125 billion parameter Frontier model when 3.8 flash next on a regular GPU with 16 or even 12 gigabytes of video RAM. Of course, I could not just ignore something like this. So, I decided to try it myself. However, I’m not as interested in simply getting the model running as I am in whether it can actually handle practical tasks. Right now, I have two GPUs that would work for

00:30 today’s test. One is an RTX 4060 Ti with 16 gigabytes of RAM, and the other is Radeon with 32 gigabytes. I will choose the first one because it is much closer to what you would find in a typical PC. And that is exactly the kind of hardware today’s project is targeting. There is also one important point. Strata itself requires 64 gigabytes of system RAM to run when 3.8 flash next. I have 128

01:01 gigabytes installed, so that is more than enough. However, if you want to try running it as well, keep this in mind. A good GPU alone is not enough. You also need a lot of RAM. Now, just a few words about how it is even possible to run 125 billion parameter model on a regular GPU. The reason is that this is a mixture of expert model, meaning it consists of a large number of experts. It has more than 24,000 experts, but only 10 of them are used to generate

01:31 each token. So, at every step, only around 6 billion parameters are actually active rather than all 125 billion. Okay, enough theory, and let us move on to practice. I connect to the desktop over SSH and start the server. Strata does not have ready-made Linux built yet, and the releases currently only include Windows. So, I build the agent myself. Loading the model itself takes around 30 seconds. By the way, Strata basically takes over the entire computer. Right now, the agent is using

02:03 her own 36 GB of system RAM. Video memory is almost completely full. So, this is not a background service that runs alongside with other workloads. While Strata is running, the computer is basically dedicated to it. To measure the model’s performance on my hardware, I wrote a Python script that benchmarks it. It uses two prompts. One requires a text response, while the other asks the model to write code. And then, it sends several requests to the model so that I can get meaningful average numbers.

02:35 Okay, I got an average of around 65 tokens per second. Considering that my GPU is weaker than the one used by the project’s authors, the result is quite close to what they reported. However, as I said at the beginning, token generation speed is great, but I am much more interested in how good the model’s answers are in this configuration. So, let us move on to some practical tests. For the tests, I will use my repository with a collection of prompts. They are grouped from simple to complex, and I

03:06 will go through them in that order. By the way, this repository is available on GitHub, so you can download it and experiment with it yourself. I will use Pi as the agent. I really like it because of how minimal and extensible it is. So, for the first test, I will use a task where the model needs to create an HTML page with a clock, stopwatch, and timer. This is a fairly simple test for such a large model, so it is more of a check that the model actually works on my machine at all. And since this is the

03:38 easiest task, I will not create any plan and will simply ask it to implement the project based on the description. Okay, the model finished the work pretty quickly. So, let us see what we got. Yes, everything looks the way it should. There is a clock, the stopwatch works, and [snorts] there is also a timer. Yeah, everything looks great. The easiest test is passed. Let us move on to something more difficult. Next, I move on to a medium-difficulty task. The goal is to build a pixel editor. Again,

04:09 I switch to the Pi agent and ask it to implement the project still without creating an implementation plan beforehand. Let us see if the model can handle it. By the way, while the model is working, I will show you the system load. As you can see, the GPU is fully utilized. Ignore the second graph, that is the integrated GPU. Around 40 GB of system RAM are being used and roughly half of the CPU cores are fully loaded. Okay, the model has finished the work.

04:39 It took 5 minutes and 20 seconds. Very good for a local model, but let us take a look at the result. Okay, it looks the way it should. Let me check if it works. Yes, it looks like the pixel editor is working. The tools and colors can be changed and there do not seem to be any errors. Great. I did not expect the model to complete this task successfully on the first try. Well, in that case, I will move on to even more difficult tasks, the kind that I usually give to models from Anthropic and OpenAI. And

05:12 the third task will be an unusual one. It will test how well models can orient themselves in our world and understand it. In this task, the models need to create an image of a pyramid using text characters that, when printed on a regular sheet of paper, would create a 3D illusion. In other words, the pyramid should appear three-dimensional when viewed from an angle. Again, I send this prompt to Pi and wait for the result. And while the model is working, I will show you how Opus 5.5 and GPT-6 solved

05:45 handled this task. You can see the detail in my recent video where I compared and testing these models. At the time, I liked the Opus version more. Okay, let us get back to today’s model. It finished the task and generated several files. The first one is a Python script. It looks like the model used it to generate the images. Then there are the images themselves. Let us look at the PNG file. Yeah, you can see that the model understood the task correctly and

06:16 even started moving in the right direction. But it still could not complete the task successfully. Let us also take a look at the SVG file. Yes, it is roughly the same here, but there also seems to be an error somewhere in the file. And the text file basically shows the same thing. Okay, unfortunately, the model could not handle this task, but to be fair, it is a very difficult one. Still, overall, it was a pretty good attempt. Well, I think we can already draw some conclusions. Despite my initial skepticism, both of

06:47 this way of running the model and the model itself performed very well. I expected it to be worse. Of course, these are not my final conclusions yet, and I will continue testing all of this over the next few weeks. But my first impressions are very positive. And that is all for today. Do not forget to subscribe to the channel and to my weekly newsletter about IT and AI. The link in the description. See you soon. Take care.