35B AI on a Laptop With 15GB RAM — Straight From the SSD

Author: Prompt Engineer 48 · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: 35B AI on a Laptop With 15GB RAM — Straight From the SSD · Channel: Prompt Engineer 48 · 13:57 · Watch on YouTube

Topics: Model inference and optimization, AI infrastructure and hardware, AI models, Local and open-source models

Summary

The video explains Edge0, a system designed to run a 35-billion-parameter model by keeping most of its weights on an SSD and predicting which experts will be needed a token ahead. The paper reports 20.4 tokens per second with 2.9 GB of active memory on a 24 GB Mac, while prefetching improves speed on a tighter 16 GB Mac. The presenter tests the Windows release on a laptop with 15 GB of RAM and finds that the 35B model runs at roughly 7–10 tokens per second, but does not consistently outperform plain llama.cpp. The video emphasizes that the paper’s prefetching results are from Apple Silicon and that the Windows build does not implement that same advantage.

Key points

The model’s weights are larger than the laptop’s RAM, so the Windows test must load experts from the SSD.
Edge0 uses a small prediction head to anticipate the next token’s expert routing, allowing SSD reads to overlap with computation.
The predicted routing determines which experts are used; a separate, unmixed adapter helps recover quality lost through quantization and routing.
The paper reports 20.4 tokens per second using 2.9 GB of active memory, compared with 3.9 tokens per second and 18.2 GB for the fully loaded setup on a 24 GB Mac Mini.
On a 16 GB Mac, the paper reports that prefetching improves speed by about 80% compared with on-demand streaming.
The 35B model scores about four points lower on average than the full-precision model across five benchmarks, with a larger gap on the AIME math benchmark.
The presenter’s Windows setup uses a patched llama.cpp build, and the project’s Windows documentation says the patches do not change speed compared with plain llama.cpp.
On the presenter’s laptop, the 35B model runs at about 7–9 tokens per second after a 29-second wait for the first word.
In the presenter’s comparisons, Edge0 and plain llama.cpp trade wins depending on settings; the overall result is described as a tie.
The presenter concludes that running a model too large for RAM is useful, but the paper’s specific prefetching advantage has not been demonstrated on Windows in this test.

Chapters

Start	Chapter	Summary

Quotes

Basically, it’s not the math that is slow. It’s fetching the bytes. — 01:31

So a full token of lid is what makes the reads overlap with the compute. — 05:10

So, this is not the papers pre-outer running on Windows. — 08:47

Honestly on this laptop it’s a tie. — 11:53

The machine on your desk is already big enough. The weights just needed somewhere to live. — 13:27

Concepts and entities

Edge0, AutoArc, The Other Half of the Memory Wall: Serving 35 Billion from SSD with Trained Routing Prediction, Mixture of Experts, pre-router, recovery LoRA, llama.cpp, MLX, RTX 4060

Full transcript (original language)

00:00 All right. So, look at this laptop. It has an RTX 4060 with 8 GB of video memory and 15 GB of RAM. Now, here’s a 35 billion parameter model. The file alone is about 20 GB. That is bigger than my entire RAM. So, this should not run, right? Well, this paper says it does. It’s called edge zero. And in their tests, a 35 billion parameter model decodes at 20 tokens a second while using under 3 GB of active memory. Now compare that to the normal way of running it where all the weights sit in

00:30 memory. Same machine, same model that gets 3.9 tokens a second and holds 18 GB. So 3.9 up to 20.4. And the trick is that most of the model is not in memory at all. It stays on the SSD. In this video, I’ll explain how that works. I’ll show you what the paper actually proves and then we will go ahead and run it on this laptop and I’ll show you what really happens. Let’s get started. So this is from a team called AutoArc. The paper is titled the other half of the memory wall serving 35 billion from SSD

01:00 with trained routing prediction. It’s a mouthful I know. And the thing I really like here is that it is all open. The framework is on GitHub. The checkpoints are on hugging phase. The adapters too and the license is Apache 2.0. So you can go ahead and download everything yourself which is exactly what I did. Now, mind you, this is a preview release and I’ll be honest about that as we go cuz there are a few things in the paper and in the repo that you should know before you get excited. Okay, so before we go into the trick, we need to

01:31 understand the problem. When an AI model writes an answer, it writes one word at a time. And for every single word, the computer has to read the model’s weights out of memory. Basically, it’s not the math that is slow. It’s fetching the bytes. The paper puts it nicely. The model does only a few operations for every bite it reads. So the limit is memory, not compute. Now memory holds two things. One is the KV cache. That’s the model’s short-term nodes and it grows with every word. The industry has done a lot of work on shrinking that part. But the second part is the

02:02 weights. The weights are fixed. They’re just there tens of gigabytes. And the usual answer was just put that in a data center. So let’s make this simple. Think of a scholar writing an essay. He sits at a small reading desk. That desk is your RAM and all the books he might need are in a giant warehouse across the road. That warehouse is your SSD. Now, this model is a mixture of experts. In AI terms, each layer has a lot of small specialist networks and only a few of them are used for each word. This one has 40 layers and 256 experts in every

02:34 layer and only four get picked for each word. So, in our story, the essay has 40 chapters. Each chapter has 256 reference books. And for every word, the scholar opens only four of them. Now, you might be thinking, great, only four books. So, the desk only needs to fit four, right? Well, that’s the catch. The scholar doesn’t know ahead of time which four. So, normally you keep the whole warehouse inside the building. That’s 19.5 GB at 4bit. And on a 24 GB machine, that leaves almost nothing for everything else. That’s why the normal

03:05 setup crawls at 3.9 tokens a second. So, how do people deal with this today? There are really three ways and they all have a problem. Option one, shrink the books. That’s quantization. Basically, you photocopy them at a lower resolution down to 4bit. That works fine. But go below 4 bits and the error grows until the model is not usable. So, that’s a floor. Option two is offloading. Tools like Llama CPP with its CPU mo option keep the experts in system memory and others catch the popular experts. But

03:37 the paper makes a good point here. All of these move the footprint. They don’t shrink it. The weights still take tens of gigabytes somewhere. And the ones that do go to disk don’t know which experts the next step needs. Which brings us to option three. Just stream the experts from the SSD on demand. Sounds good, right? Well, here’s where it breaks. Look at how the model works. Chapter 12 needs to pick its four books based on what happened in chapter 11. So, the scholar finishes chapter 11. And only then does he know which books to send the runner for. and the runner has to walk to the warehouse and back. And

04:08 that happens at every chapter for every single word. So the scholar spends most of his time just waiting. So to sum things up, the weights are too big for the desk. Shrinking them has a floor. Moving them just moves the problem. And fetching them on demand makes you wait at every layer. We need a way to fetch the books before the scholar knows he needs them. And that is exactly what edge zero does with something they call the preotterer. Here’s the idea. Instead of waiting for chapter 11 to finish, a small prediction head looks at what the

04:39 model did on the previous word and guesses which books chapter 12 will need on the next word. So in our story, there’s a runner who watches the scholar and thinks last time he was in chapter 11, he used his books. So next time he’ll probably want these for chapter 12. And the runner starts walking to the warehouse right away. By the time the scholar gets there, the books are already on the trolley. Now if we go a bit deeper, here’s how it works in technical terms. Each layer has a small head and it predicts the routing of the next layer, but one whole token ahead.

05:10 Not one layer ahead, one token. And that matters the paper tests the other way predicting inside the same token and found it cost between 30 and 100 milliseconds per step to synchronize. That is more than the load time you’d be hiding. So a full token of lid is what makes the reads overlap with the compute. And here’s the part I think is really smart. The prediction is the routing. So the scholar is only allowed to open the books the runner brought. Other methods guess and when they guess wrong they fall back to loading again or they drop the expert here. The guess

05:41 becomes the actual rule. So there’s nothing to fall back to and nothing drop. But you might be thinking wait if the model is forced to use guest experts won’t the answers get worse? Well actually yes. And that is the second piece. Quantization to 4bit costs some quality and the guest routing costs some more. So they train a small adapter called a recovery Laura to pay that back. It’s trained while the guest routing is already in place with the original full precision model as a teacher. Now there’s one detail I really like. The adapter stays unmmerged. So in

06:11 our story, the original page is untouched and the corrections are sticky notes on top. Why not just merge it? Because the paper tested that the adapter’s changes are smaller than the gap between two 4-bit values. So when you merge and re-quantise, only about 18% of the effect survives at the output. That’s like rewriting a page with a thick marker. The fine print is gone and the sticky notes cost 42 megabytes and no measurable speed. Quite a good design. The training itself is three steps. Train the prediction heads

06:41 first, then fine-tune the adapter with the guest routing active on about 2 million rows. Then a last pass on the model’s own outputs. They say the order was not negotiable. It’s a bit too technical for this video, but the paper goes through it in detail. Now, let’s look at the numbers because this is where it gets interesting. On a Mac Mini with 24 GB, Edge0 gets 20.4 tokens a second using 2.9 GB versus 3.9 and 18.2 for the fully loaded version. That’s about five times faster using a six of

07:12 the memory. But here’s something I want to point out. In that same table, the paper says if you switch the pre router off, you get 19.9. So on that machine, the pre-outer doesn’t add much. The big win there came from not loading everything into memory at all. The pre-outer shows its value on a tighter machine on a MacBook with 16 GB where the model doesn’t fit. They compare the plane on demand streaming against pre-fetching. And you can see 3.5 tokens a second goes up to 6.4 and at the other settings 4.8 to 8.6 and 1.8 to 3.3.

07:45 That’s about 80% more speed. And the quality, the 35 billion model averages 79.2 across five benchmarks versus 83.2 for the full precision one. So about four points lower, but look at aime, the hard math one. 86.6 versus 92.7. That’s six points. The paper itself says the loss shows up in long chains of reasoning. And two more honest limits. Adjacent words only agree on about a quarter of the experts. So a lot of what

08:16 the runner fetches never gets used again and it handles one request at a time. Also everything in the paper runs on Apple silicon. The cuta part in their words is architecture not code. So that is the paper. Now what about my laptop? Okay. So now let’s go ahead and test this out on my own laptop. And mind you everything here is running locally. This is an RTX 460 laptop GPU with 8 GB of video memory, 15 GB of RAM, and a 1 TB NVME SSD. Now, before I show you the

08:47 numbers, one thing I need to be upfront about. The paper’s results are on Apple Silicon Max using a framework called MLX. I don’t have a Mac. What the project gives us for Windows is a different build. It’s a patched version of Llama CPP. the big model weights. The experts stay on the CPU side and get memory mapped from disk and the GPU through Vulcan handles the rest. And the Windows readme itself says that compared to plain llama CPP, these patches don’t change the speed. So, this is not the

09:17 papers pre-outer running on Windows. Keep that in mind because I think it’s important to be honest here. Anyway, setup was straightforward. I downloaded the models from hugging face, built the engine from source, and the app converts the weights to a GB GUF file on your machine. That took about 11 minutes for the 35 billion model. Let’s start with the small one, the 8 billion model. It’s about 5 GB, so it fits fine. And you can see I get about 30 tokens a second. To be exact, the median of three runs is 30.8 tokens a second. The Windows readme

09:48 shows 44.8 8 on their reference machine which has 48 GB of RAM and a newer GPU. So a bit lower on my laptop but really good for a laptop. Now the prompt processing is slower about 26 tokens a second here. So the first answer takes a few seconds to start. Now the main thing the 35 billion model look at the file it’s about 20 GB. My RAM is 15. So the whole thing cannot sit in memory and yet it loads and it runs. Now let’s look at what these models actually do. I give both of them three simple prompts. a

10:19 short explanation, a bit of code, and a JSON answer. And I ran the code to check it. The Python function passed all five of my test strings on both models. And the JSON was valid on both. But look at what the machine is doing. On the 8 billion model, the SSD is basically idle while it answers about 5 mgabytes a second because everything fits in RAM. On the 35 billion model, the SSD is reading about 220 megabytes a second on average up to about 630 and the RAM is completely full, 15.3 out of 15.3 GB.

10:50 That is the experts being paged in from disk and you can feel it. The very first answer took 29 seconds before the first word appeared. After that, it settles at around 7 to 9 tokens a second. And the speed with the repo’s own benchmark, I get about 6.9 tokens a second. Slower than the 8 billion model, of course, but it’s a 35 billion model running on a laptop where it should not fit. And the paper’s 20 tokens a second is on a Mac with 24 GB and fast unified memory. So that isn’t a fair comparison with my

11:21 machine. Now, the question I really wanted to answer, the paper says the trick is better than the normal approach. So, I took plain llama CPP built from the exact same source version without any edge zero patches and ran the same 35 billion file with the same adapter and the same settings. And I alternated which one ran first because on this kind of machine the numbers jump around a lot. And here’s what happened with the apps settings. Edge0 got a median of 9.7 tokens a second and plain llama CPP got eight. Edge0 won two

11:53 rounds out of three. But then I ran it again with the model’s full context length and plane llama CPP was faster in all three rounds 11.6 against 7.4. So which one is faster? Honestly on this laptop it’s a tie. The result flips with the settings and the paper itself says these runs can vary by about 40%. So here’s my takeaway. On Windows today Edge0 lets a 35 billion model run on a machine that cannot hold it. That part is really cool. But that is something plain llama CPP can also do. The thing

12:24 that is special about the paper, the train parer is a MAC result for now. I’d love to see someone port it to CUDA because in the paper’s own words, that part is architecture, not code. So let’s put the pieces together. We have a model that’s too big for fast memory. The weights live on the SSD. A small train head predicts which experts the next word needs one whole token ahead. So the reads overlap with the compute. The prediction is the routing. So nothing is dropped and a small adapter kept separate pays back the quality lost from

12:56 4bit and from the guessing. Four points lower on average, six on the hardest math. And the result in the paper is a 35 billion model at 20.4 tokens a second inside 2.9 GB of active memory against 3.9 tokens a second and 18 GB a normal way. On a tighter 16 GB Mac, prefetching adds about 80%. And what I found on my own laptop is that the model does run around 7 to 10 tokens a second, but the win over plain llama CPP isn’t there on

13:27 Windows yet. But I really like the idea behind it. The paper ends with this line. The machine on your desk is already big enough. The weights just needed somewhere to live. So in this video, we have seen what Edge Zero does, how the pre-outer works, and how it runs on a normal laptop with 15 GB of RAM. If you’re interested, you can go ahead and check this out on your own system and let me know how it goes, especially if you try it on a Mac or a different Windows machine. The paper, the code, and the models are all linked in the description below. I’ll see you in

13:57 another one. Check out the next video.