What is Strata ? 6x Faster than Llama.cpp for local AI

Author: Data Science in your pocket · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: What is Strata ? 6x Faster than Llama.cpp for local AI · Channel: Data Science in your pocket · 09:52 · Watch on YouTube

Topics: AI evaluation and benchmarks, Model inference and optimization, AI infrastructure and hardware, Local and open-source models

Summary

The video presents Strata, a free, MIT-licensed C++ engine designed to run a 125-billion-parameter sparse model on consumer hardware by distributing work across GPU, system RAM, CPU, and SSD. It describes how the engine uses expert sparsity, speculative decoding, and overlapping prompt processing to achieve the author-reported speeds of up to 94 tokens per second on the reference PC. The video also covers memory requirements, model variants, setup, local API features, and limitations such as single-user serving and support for only one model family. It cautions that the performance figures are self-reported and that insufficient available RAM can make the system very slow.

Key points

Strata is a free, MIT-licensed C++ engine built specifically for one model family rather than a general-purpose inference engine.
The featured model is sparse: while it has 125 billion parameters, only about 6 billion are used to compute each token.
Strata distributes model data and computation across GPU memory, system RAM, CPU, and SSD; the video says extra GPU RAM can hold more hot experts.
The engine offers several memory and quality options, including a coder build that fits its core in 32 GB of RAM.
Speculative decoding drafts up to three tokens ahead, which the full model checks in a single pass; the video says this is exact rather than approximate.
Strata overlaps prompt processing and expert transfers, with the video reporting prompt-processing peaks of 2,650 tokens per second.
The author reports 94 tokens per second in short chat on the reference PC, while stock llama.cpp reaches 15 on the same machine.
Setup uses a Windows batch file or Linux shell script, then downloads about 70 GB of model data and opens a local browser app.
Strata provides local API compatibility and optional image input; the video says data stays on the user’s PC.
Limitations include one request at a time, support for just one model family, self-reported benchmarks, and performance degradation when RAM is occupied by other apps.

Chapters

Start	Chapter	Summary
00:00	Running a huge model on a gaming PC	The video introduces Strata and explains why running a 125-billion-parameter model on consumer hardware challenges conventional expectations.
02:04	Sparse experts and memory requirements	It explains the model’s mixture of experts, Strata’s GPU-RAM-SSD arrangement, and the available model sizes.
04:08	Speculative decoding and long-context processing	The video covers draft-and-verify generation, prompt processing, cache management, and reported performance.
06:13	Benchmarks, setup, and local integrations	It presents author-reported speed figures, installation steps, local API support, and other features.
08:48	Limitations and why Strata stands out	The video lists Strata’s constraints and closes by emphasizing its approach of sharing work across the GPU, RAM, CPU, and SSD.

Quotes

only about 6 billion actually compute on any given token. — 02:04

The full model checks all three guesses in a single pass through its 48 layers. — 04:08

These are the author’s own measurements. — 06:13

This is a single user engine. No batching, no multi-tenant serving. — 08:48

Why is the GPU the only chip allowed to think? — 09:52

Concepts and entities

Strata, llama.cpp, Mixture of Experts, speculative decoding, cache KV, MCP, Claude Opus

Full transcript (original language)

00:00 What if I told you a 1-00-20-5- billion-parameter AI model bigger than GPT3 class is running right now on an ordinary gaming PC? Not a server rack, not the cloud. One consumer graphics card with 12 GB of video memory, writing text at 94 tokens per second. That’s faster than most people can read. For a setup like this, the conventional wisdom says you’d need two to four data center GPUs just to

00:30 load the official checkpoint. A new open- source engine called straighta says otherwise. Here’s how it pulls that off and what it costs you. To see why this is surprising, look at the math everyone assumed. 125 billion parameters, even at aggressive precision, is on the order of 250 GB of weights. The official server deployment recipe wants two to four data center class GPUs with a 51 GB side table parked in host memory on top. Your

01:01 gaming card 12 GB of RAM. That is a 20 times gap. And that’s exactly why local models this size were considered a fantasy. Every attempt so far boiled down to one move. Shrink the model until it fits the card. Straighta threw out that assumption entirely. Strata is a free MIT licensed C++ engine built by a solo developer in Slovenia and it was born on Reddit. The author first got this model running through llama.cpp at about 15 tokens per second on his

01:33 RTX570 and posted it on r/local law ma. Then he did something different. Instead of treating the model as generic, he studied this one architecture, hardwired the engine around it and jumped from 15 to 65, then up to 95 tokens per second. Four to six times faster on identical hardware. The repo went up on September 24th. 10 days later, the internet noticed. The first thing Strata exploits is that

02:04 this model is sparse. Quen 3.8 flash. Next is a mixture of experts. It has 24,576 experts, 512 in each of 48 layers. But for every single token, a small router network picks just 10 of them. So while the model owns 125 billion parameters, only about 6 billion actually compute on any given token. That’s the entire opening. 95% of the model is asleep at any moment. The question becomes, where do you put the sleeping experts and how

02:36 fast can you wake the right ones? Strata’s answer is a kitchen analogy. The counter, the pantry, and the seller. The GPU is the counter. It holds everything every token touches. Attention layers, routters, the cache, plus as many hot experts as the remaining RAM fits. Here’s the killer scaling law. Every extra gigabyte of RAM holds roughly 700 more experts. And every expert on the GPU is one the CPU never has to compute.

03:06 System RAM is the pantry. It holds all 24,000 experts. And when the GPU misses, the CPU computes that expert right there in RAM in parallel with the GPU. And the SSD, that’s the seller. A 20-8-8 GB lookup table that only hands over a few rows per token. How much brain you download depends on your RAM. Strat ships four sizes. Q20 is the fastest and needs 37.6 GB of combined memory.

03:38 IQ2XS, the recommended one, needs 39.2 and is a bit smarter. IQ3XX needs 47. IQ3S needs 54.8 and matches the full model on the published tests, but it’s the slowest. Then there’s the wild card, a coder build that prunes the experts in half, keeps 91.3% of the model’s S.WE bench score, and its core fits in just 32 GB of RAM, the only variant that does. Rule of thumb, 32 gigs means

04:08 coder, 48 means the small two, 64 runs anything. Trick number two, the model writes its own rough draft. Built into the architecture is a small prediction module, and Strata uses it to draft up to three tokens ahead. Then here’s the clever part. The full model checks all three guesses in a single pass through its 48 layers. On average, 2.4 to 3.2 two tokens get accepted per pass, which makes generation 1.6 to 1.8 times

04:40 faster. And this isn’t approximation. The check is exact. So the output is bit for bit what the model would have written anyway. It’s like an intern typing ahead and an editor approving whole lines at once. Trick three is about reading, not writing. Before the model can answer, it has to ingest your prompt. And that’s usually the slow part. Strata reads prompts and chunks of up to 8,192 tokens. And while one layer’s attention

05:11 runs, the next layer’s experts are already streaming over the PCI Express bus. Everything overlaps. The result, over a,000 tokens per second of prompt processing, peaking at 2,650. In human terms, a 30,000 token document, a small book, takes about 1 minute before the first answer token appears. Follow-up messages start in seconds because what it already read stays

05:41 cached. Long conversations have their own memory problem. The KV cache, the model’s running notes on everything it’s read, grows with context, and it competes with experts for VOM. straight his answer is streaming. Past 64,000 tokens of context, most of the cash moves to ROM and only the part attention is actively reading stays on the card, freeing room for thousands more GPU resident experts. There’s also an optional 4-bit mode that uses a mathematical rotation before

06:13 compression, having cache memory at a measurable cost to precision. It’s a trade you can switch on. The safe 8-bit default stays on the point. 262,000 tokens of context is reachable on a 12 gig card. Now, the receipts with a caveat we’ll repeat later. These are the author’s own measurements. On the reference PC, an RTX 570, a 6 core Ryzen, 64 gigs of RAM. The fastest size hits 94 tokens per

06:43 second in short chat, 76 at 128,000 tokens of context, and still 60 at the full 262K. The quality maxing size manages 53 on an AMD RX 970 XT 60. And the estimate for a 24 GB RTX 30090, a 6-year-old card is 100 to 140 tokens per second because more ROM means the CPU barely works at all. Meanwhile, stock lama.cpp

07:15 on the same 570 15. Getting it running is genuinely one click. Doubleclick start here.bat on Windows or run setup.sh on Linux. It asks four questions. Model size, context, length, images. Then it installs its own Python, its own libraries, and downloads about 70 GB of model resumable thankfully. Then your browser opens a local app with chat, a live monitor showing tokens per

07:46 second, GPU load, temperatures, the works, and an about tab. One honest warning, on first start, the engine pins 34 to 55 GB into RAM, and your PC may freeze for 1 to 3 minutes. That’s normal. Don’t close the window. Go make coffee. Strata isn’t just a chat window. It’s a local API server pretending to be the big clouds. It speaks OpenAI’s interface at /v1 chat completions. So most tools

08:17 just work against localhost port 8080. It also speaks anthropics interface/v1 messages which means clawed code can run against your own GPU by changing one environment variable star. There’s an MCP server so an AI assistant can install, start, stop, and benchmark strata itself. You can dial the model’s thinking level from off to high. And with the optional vision encoder under a gigabyte, it reads images, too. Nothing leaves your PC.

08:48 Time for the cold water. First, one request at a time. This is a single user engine. No batching, no multi-tenant serving. If you want that, you want VLM. Second, Strata runs exactly one model family. Llama.cpp CPP and OAMA run hundreds of architectures. Strata is fast precisely because it’s hardwired to this one. Third, those benchmarks are self-reported by the author on two PCs, so treat them as vendor numbers. Fourth,

09:20 the coder build is weaker in Chinese, Japanese, and Korean, a known bug. Fifth, if other apps eat your RAM, experts spill to disk and everything crawls fast, but not magic. So, why did this blow up? 8.6,000 stars in 10 days, 38 releases in the same span, trending across all of GitHub 3 days after launch and a delightfully meta detail. The commit log openly credits Claude Opus’ co-author, a local AI engine substantially written with II,

09:52 but the real story is the philosophy. Everyone else asked, “How do we shrink the model to fit the GPU?” Strata asked, “Why is the GPU the only chip allowed to think? Spread the work across GPU, RAM, CPU, and SSD. Overlap everything, and suddenly the data center is optional. Links below. Go read the docs. They’re half the fun.