This 125B AI Model Runs 5x Faster on My PC Than in the Cloud (Using Strata)

Author: Better Stack · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: This 125B AI Model Runs 5x Faster on My PC Than in the Cloud (Using Strata) · Channel: Better Stack · 08:38 · Watch on YouTube

Topics: Model inference and optimization, AI infrastructure and hardware, Local and open-source models

Summary

The video demonstrates Strata running Qwen 3.8 Flash Next, a 125-billion-parameter model, locally on a gaming PC and compares its speed with a cloud version. Strata divides the workload across the GPU, CPU, RAM, and SSD, using expert routing and speculative decoding to run a model larger than the GPU’s VRAM. In the presenter’s tests, the local setup was often several times faster, though the comparison is affected by differences in hardware, quantization, and cloud-provider load. The video also covers setup requirements, example tasks, and the electricity cost of a local run.

Key points

Strata is a free, open-source project for running large models across a PC’s GPU, CPU, RAM, and SSD.
Qwen 3.8 Flash Next is a mixture-of-experts model with 125 billion parameters, of which about 6 billion are active for a given token.
Strata keeps shared model components and frequently selected experts in VRAM, stores the experts in RAM, and retrieves small portions of a large lookup table from the SSD.
A GPU-to-CPU coordination mechanism called a “doorbell” lets the two processors work on the same layer at the same time.
Speculative decoding uses a small helper model to predict tokens; the presenter says it made generation about 1.6 to 1.8 times faster.
The presenter ran the two-bit quantized model on an RTX 5090 with a Core Ultra 9 285K and 62 GB of RAM; setup required about 70 GB of downloads and around 80 GB of free disk space.
The local model was faster on several tests, including coding, creative writing, and a long home-lab guide, while a short logic puzzle was nearly a tie.
The presenter cautions that this was primarily a speed comparison: the local model was quantized, and the cloud provider may use different hardware or a larger model.
The electricity for one 33-second local answer cost less than a tenth of a cent, excluding the price of the GPU.

Chapters

Start	Chapter	Summary
00:00	A large model running locally	The presenter compares local and cloud generation speeds and introduces Strata as a way to run a model too large to fit in GPU memory.
01:36	How Strata distributes the workload	The video explains the model’s expert structure and how Strata uses GPU memory, RAM, SSD storage, CPU coordination, and speculative decoding.
04:20	Setup and side-by-side tests	The presenter describes the hardware and setup requirements, then compares Strata with the cloud model on coding, logic, and creative-writing prompts.
06:30	Long-form test and caveats	A long home-lab guide test favors Strata, and the presenter discusses comparison limitations, local electricity cost, and the potential of local AI.

Quotes

my PC was done in 6.6 seconds while the cloud took 33 seconds. — 00:00

both of them end up working on the same layer at the same time. — 02:43

it just gets there about 1.6 to 1.8 times faster. — 03:17

the electricity for that 33-second answer was less than a tenth of a cent. — 07:01

running AI locally keeps getting more and more realistic. — 08:07

Concepts and entities

Strata, Qwen3.8 Flash Next, Qwen3.8 Flash, Mixture of Experts, speculative decoding, RTX 5090, OpenRouter, Better Stack

Full transcript (original language)

00:00 What you’re looking at here right now is a 125 billion parameter AI model running on my own computer right here at my desk and it’s pushing over 250 tokens per second. On the right, I’ve got Qwen 3.8 flash running in the cloud through open router and it’s doing around 60. I gave them both the exact same prompt and my PC was done in 6.6 seconds while the cloud took 33 seconds. And this wasn’t just a one-off either because in pretty much every test I did, my machine did

00:34 the task three to five times faster. Now, how is that possible? Well, the thing that makes this possible is a free and open source project called Strata. Now, normally a model this big lives on a server somewhere because even the smallest decent version of it is a 66 GB file and it all has to sit in memory while it runs. Even my RTX 5090 graphics card only has 32 GB of VRAM. So, it can’t fit on the graphics card, but Strata gets around that in a pretty

01:06 clever way by getting your whole PC to pitch in. Your graphics card, your CPU, your RAM, and even your SSD all end up doing a piece of the work. So, in this video, I’ll show you what Strata is, >> [music] >> how it actually pulls this off, and then we’re going to go through some test runs side by side. It’s going to be a lot of fun, so let’s dive into it. So, first, a quick word about this model itself. Qwen 3.8 flash next is a mixture

01:36 of experts model and we’ve talked a lot about mixture of experts models on this channel before, so I won’t go deep into it here, but the short version is that each layer has 512 of these experts and it only picks 10 of them at a time. So, even though the whole thing is 125 billion parameters, only about 6 billion of them are actually doing anything for any given token. And that’s what Strata builds on. The stuff that every token needs is only about 3.5 GB, so that lives in your graphics card, and the rest of the card gets filled with

02:09 the experts that get picked up the most. Then, all 24,576 experts sit in your regular RAM, and the SSD comes in because the model also has a huge lookup table that’s almost 29 GB on its own. So, Strata leaves that on the SSD and only grabs a few tiny rows of it for each token, and it grabs them a step early, so you never feel it. Now, the part that I found really cool is that for every layer, the GPU figures out which 10 experts it needs and leaves

02:43 their IDs in a little spot in shared memory, which the paper actually calls a doorbell. The CPU is just waiting for it, and as soon as it rings, it splits up the job. Experts that are already on the GPU get handled there, and the CPU crunches the rest right into RAM. So, both of them end up working on the same layer at the same time. And on top of that, there’s speculative decoding. The model comes with a tiny helper that guesses the next few tokens, and the big model checks them all in one go and keeps the ones it agrees with. The

03:17 answer comes out exactly the same, it just gets there about 1.6 to 1.8 times faster. And in my own test, the big model agreed with the helper about 79% of the time. Now, the readme states it gets up to about 94 tokens per second on a 12 GB RTX 5070, and in my tests, I was getting over 250. And that’s because I ran it on my RTX 5090 with 32 GB, and according to the paper, VRAM matters way

03:49 more than raw GPU speed. Every extra gigabyte fits around 700 more experts, so on a 5090 most of them fit and the CPU barely has anything left to do. And my prompts and answers were also shorter than the ones in the readme and it does slow down a bit as chats get longer, so it’s not a perfect apples-to-apples comparison. And we have to take into account that the cloud might also be slower because the open router provider is sharing its hardware between tons of

04:20 people while my GPU is only serving me. Okay, but with that said, let’s actually try it. Setting it up is really easy because you just run this one script and it asks you which model and which size you want and then it downloads about 70 GB and opens a little chat app in your browser. I went with the two-bit quantized version, which is the smallest and fastest one and I’m running it on my 5090 with a Core Ultra 9 285K processor and 62 GB of RAM. And just a

04:52 heads up, the first time it loads your PC might freeze for a couple of minutes and you’ll need at least 12 GB of VRAM, 32 GB of RAM and around 80 GB of free disk space to actually get it off the ground. And also once it’s loaded, it pretty much takes over your whole GPU. On my machine it used almost all of the 32 GB of VRAM plus another 39 GB of normal RAM. And to make the comparison fair, I put together a little app where I send the same prompt to Strata on the

05:23 left and to Qwen 3.8 Flash on the right and it keeps track of the speed, the time and the cost. So I first gave them a simple Python coding task to find the longest palindromic substring in a string and Strata was done in about 4 seconds while the cloud took almost 19 seconds. As far as the output, they both seem to be pretty similar. Even the test cases they provided were both very similar. And then I tried a quick logic puzzle, and this one was pretty much a tie at 1.4 versus 1.5 seconds, mostly because the cloud barely thought about

05:57 it at all. Funnily enough, they both fell for the classic bat and ball trick and said that the ball costs 10 cents, when in reality the right answer is actually five, but I’m just going to blame this on the model itself, not on the tools I’m using. And then I gave them a fun prompt to write description for a toaster that’s secretly alive, and that took about 3 seconds for Strata versus 10 for the cloud version. Then I gave them both a big task. I asked for a detailed guide on building a home lab server, and I increased the limit to

06:30 9,000 tokens, and Strata wrote over 8,000 tokens in 33 seconds. And as for the cloud model, well, it just kept thinking and thinking and thinking. It burned through the whole 9,000 token limit on reasoning, and after almost 3 minutes, it still hadn’t written a single word of the actual answer. Now, to be fair, I don’t think this part is really about speed, because how long a model thinks depends on how the provider is set up. But even if it had written

07:01 the same answer at around 54 tokens per second, it would have needed about 2 and 1/2 minutes, which is still 2 minutes more than Strata’s result. So, just to keep things fair, I ran it again with thinking turned off, and this time they both actually gave me a full answer, and Strata was done in 6.6 seconds, while the cloud took 37 seconds. And if you’re wondering about cost, well, the electricity for that 33-second answer was less than a tenth of a cent. Of course, that’s not counting the price of the RTX 5090 itself, which is a whole

07:35 ’nother story. >> [laughter] >> And also keep in mind that I’m running a 2-bit version locally, and the cloud is probably running a bigger version of the model. So, I would only really look at this as a speed test. But, to be fair, in my tests, the two-bit version gave answers that were pretty much on par with the cloud model. So, there you have it, folks. That is Strata in a nutshell. Honestly, I didn’t expect the model this big to run on a single gaming PC at all, let alone beat the cloud model by that

08:07 much. And it’s kind of crazy how fast this space is moving because projects like these keep finding smarter ways to squeeze huge models onto the hardware we already have at home and running AI locally keeps getting more and more realistic. So, if you’ve got a beefy GPU lying around, I’d definitely give Strata a try. And if you do give it a shot, let me know in the comments what kind of speeds you’re getting. And folks, if you like these types of technical breakdowns, please let me know by smashing that like button underneath the

08:38 video. And also, don’t forget to subscribe to our channel. This has been Andris from Better Stack, and I will see you in the next videos. >> [music] [music]