Strata vs Llama.cpp vs vLLM

Author: Data Science in your pocket · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Strata vs Llama.cpp vs vLLM · Channel: Data Science in your pocket · 14:12 · Watch on YouTube

Topics: Model inference and optimization, Local and open-source models

Summary

The video compares Strata, llama.cpp, and vLLM by explaining how each handles LLM inference and the trade-offs behind its design. It presents llama.cpp as a flexible, quantization-friendly engine for local use, and vLLM as a high-throughput serving system designed for concurrent users. Strata is described as a specialized engine for Alibaba’s Qwen 3.8 8B Flash Next model, using GPU, system RAM, and SSD together to achieve impressive reported speeds on consumer hardware. The video emphasizes that Strata’s performance figures come from its author, with no independent benchmarks available at the time, and that its comparison with llama.cpp is complicated by the latter’s early support for the new model.

Key points

LLM inference involves parallel prompt prefill and sequential token-by-token decoding, which is typically limited by memory bandwidth.
The KV cache grows with context length and can consume more memory than the model weights.
llama.cpp prioritizes broad hardware support, quantization, and running models locally, but is not designed to scale efficiently under concurrent load.
vLLM uses paged KV-cache management and continuous batching to serve many concurrent requests efficiently.
Strata is specialized for Qwen 3.8 8B Flash Next, a mixture-of-experts model with a large total footprint but relatively few active parameters per token.
Strata’s design distributes work across GPU VRAM, system RAM, and SSD, with CPU and GPU computation happening in parallel.
The video reports Strata’s author-claimed decode speeds of roughly 53–94 tokens per second on the tested machine, depending on quantization and settings.
The performance figures for Strata were measured by its author on one machine; the video says independent third-party benchmarks were not available.
The reported four-times speed advantage over llama.cpp is qualified because llama.cpp’s support for the new model was still immature.

Chapters

Start	Chapter	Summary
00:00	Strata’s sudden rise	The video introduces Strata’s rapid popularity and sets up a comparison with llama.cpp and vLLM.
01:36	Why LLM inference is a systems problem	The video explains sequential decoding, parallel prompt processing, memory bandwidth, and the growing KV cache.
03:42	The two established engines	It outlines llama.cpp’s local-first flexibility and vLLM’s paged attention and continuous batching for concurrent serving.
08:23	How Strata is designed to work	The video describes Strata’s Qwen specialization, its use of GPU, RAM, and SSD, and its speculative decoding techniques.
11:03	Reported results and the comparison caveat	The video presents Strata’s author-reported performance numbers, notes the lack of independent benchmarks, and begins a comparison of the engines.

Quotes

generating a 500 token reply means 500 sequential passes. — 02:07

inference serving is not a model problem, it’s a systems problem. — 03:10

don’t treat the GPU as the computer. Treat the whole PC as the computer. — 08:57

Every figure you’re about to see was measured by Strata’s author on one machine. — 11:03

it’s not the same as llama.cpp is slow. — 13:08

Concepts and entities

Strata, llama.cpp, vLLM, Qwen3.8 Flash Next, Mixture of Experts, GGUF, KV cache, PagedAttention, continuous batching, speculative decoding

Full transcript (original language)

00:00 On September 30th, 2026, a post hit the local law ma subreddit with a title that sounded like a typo. Quen 3.8 flash. Next, 50 tokens per second generation and 1500 tokens per second prompt processing with 12 GB of VRAM and 64 GB of RAM on a laptop. 100 dash and dash 20-5- billion-p parameter model on a laptop at reading speed.

00:32 That post pulled around 150 upvotes and 140 comments. And it wasn’t even the first one. A few days earlier, another thread claimed 65 tokens per second. Quote, four times faster than llama.cpp. The engine behind both posts is called Strata. It appeared on GitHub on September 24th, built by a single anonymous developer. 3 days later, it was the number one trending C++ repository on GitHub. 3 days after that, it was number one

01:04 across all languages. 10 days after it existed, it had over 8,000 stars and people on 4chan were calling it the llama.cpp killer. Half meme, half serious. Now, if you’ve been running local models for a while, your skepticism alarm should be screaming right now because this community has seen miracle engines before. So, in this video, we’re going to do this properly. We’re going to put Strata up against the two reigning champions of LLM inference. Llama.cpp, the engine that runs literally anything,

01:36 literally anywhere, and VLM, the engine that serves thousands of users at once in data centers. Three engines, three totally different philosophies. By the end, you’ll know exactly which one belongs on your machine and whether the Reddit hype is real. But first, we need to understand why any of this is hard. Here’s the thing most people miss. A large language model doesn’t think in paragraphs. It generates exactly one token at a time. Each new token requires a full forward pass through the entire

02:07 network, every layer, every weight that’s active. And then that token gets appended to the input and we do it all over again. That’s called auto reggressive decoding. And it means generating a 500 token reply means 500 sequential passes. You can’t parallelize it away because token number 500 literally cannot be computed until token 499 exists. This creates two very different workloads. First, there’s prefill processing your prompt. Quote, “All those prompt tokens already exist, so

02:39 they can be chewed through in parallel. It’s a big dense matrix multiply that GPUs love. Then there’s decode generating the answer one token at a time where the GPU spends most of its time just reading weights for memory rather than computing. Decode is memory bandwidth bound. That’s why prompt processing is measured in thousands of tokens per second. while decode crawls along in tens. And then there’s the KV cache. Every token the model has ever seen in this conversation leaves behind key and value vectors at every layer.

03:10 And the model needs all of them, every step to attend. The rough size is two for keys and values times the number of layers times the number of attention heads times the head dimension time the sequence length time bytes per element. The killer term there is sequence length. Double your context, double your KV cache. At long context, the KV cache can eat more memory than the model weights. So, inference serving is not a model problem, it’s a systems problem. Where do the weights live? Where does

03:42 the KV cache live? How do you keep a thousand requests in flight? The three engines in this video are really three different answers to those questions. Champion number one, llama.cpp. In March of 2023, GRG Gurgengh ported Midas Llama model to run in plain C on a MacBook as basically a side project. Two and a half years later, that side project has around 130,000 GitHub stars and is without exaggeration the foundation of the entire local AI

04:12 movement. Its first big idea is GGUF, a single file that bundles the weights, the tokenizer, and the metadata. one file, drag it anywhere, it runs. Second big idea is quantization as a first class citizen. The Kqu Quants, you’ve seen names like Q4KM, Q5KM, squeeze weights down to four, even three or two bits per weight with handwritten kernels that dequantize on the fly. That’s what shrinks a model to a quarter of its size with surprisingly little

04:43 quality loss. And it’s the reason consumer GPUs can play at all. Third, it’s CPU first. CS SIM kernels for a VX2 and a VX512. Memory mapped loading, so a model bigger than your RAM still technically runs. An optional GPU offload where you push as many layers as fit into VRAM. And the backend list is absurd. CUDA, Metal, Vulcan, Rockom, SYCL, Plain CPU, Windows, Mac OS, Linux, Android,

05:14 iPhones, Raspberry Pi. If it has a processor, Llama CPP probably runs on it. And here’s the part people forget. When you use a Lama or LM Studio or Llama file, you’re using Llama CPP. Mozilla.i benchmarked all four in September 2026 and found them within a few% of each other because they share the same core. Build flags and speculative decoding settings moved the numbers by up to 63% more than the choice of server did. Its weakness, it’s

05:45 built for one user. Under concurrent load, throughput barely grows and time to first token balloons. It also serves new model architectures on community time, which matters. Remember that for later. MIT licensed, maintained by the GGML or community with a huge contributor base. The people’s engine champion number two, VLLM. This one came out of UC Berkeley’s Sky Computing Lab in early 2023. Wu Kuan, Juan Lee, Ayan Stoka and colleagues. And

06:17 the paper won best paper at SOSP, one of the top operating systems conferences. That detail matters because VLLM’s killer idea is literally borrowed from operating systems. Remember the KV cache from scene 2? Before VLM, serving engines reserved a big contiguous chunk of VRAM for every request, sized for the worst case. Most of it set unused. The paper measured 60 to 80% of KV memory wasted to fragmentation. VLM’s answer page detention is exactly

06:51 how your OS manages RAM. Chop the KV cache into fixed-siz pages, hand them out on demand, and keep a page table mapping each request’s logical tokens to physical blocks scattered anywhere in memory. No contiguity needed. Waste dropped from 60 to 80% to under 4%. which means far more requests fit in the same GPU. Then stack continuous batching on top. Instead of waiting for a whole batch to finish, theuler lets new requests join mid generation at every

07:21 iteration. The original paper claimed two to four times the throughput of the 2023 era serving stack and the project has kept sprinting since chunked prefix, prefix [music] catching, disagregated prefill, and decode. Multi-GPU parallelism FP8. Today it’s at roughly 93,000 stars, Apache 2.0 with contributors from Nvidia, Red Hat, [music] and Anthropic. And the receipts in September 2025, Red Hat ran VLM

07:52 against Llama.CP, same model, same hardware, and ramped up the load. At peak load, VLM delivered over 35 times the request throughput and over 44 times the output tokens per second. Its P99 time to first token stayed flat out to 64 concurrent users while llama.cppps grew exponentially. That is what a serving engine is for. The catch, it wants data center hardware. Nvidia, AMD, Instinct, TPUs, Tranium, Linux servers. It’s overkill

08:23 for your desktop and proud of it. And now the challenger. To understand Strata, you first need to understand the one model. It was built for Alibaba’s Quen 3.8 8 flash next open weights since late August 2026. It’s a 125 billion parameter mixture of experts model, but only about 6 billion parameters are active per token because it roots each token to just 10 experts out of 24,576. 512 experts per layer, 48 layers, plus a

08:57 separate 51 billion parameter ungram embedding table, pure lookup, no compute, and a 4 billion parameter MTP multi-token prediction head. Total footprint around 180 billion parameters of stuff to store somewhere. A general purpose engine sees that and panics. Straight as insight is, don’t treat the GPU as the computer. Treat the whole PC as the computer. Three tiers. [music] Tier one, the GPU USVM. Attention

09:28 layers, the Delta mixers, routters, shared experts, the KV cache, plus an adaptive cache of the hottest few thousand experts, about 700 experts per extra gigabyte of VRM. Learn from live usage. Tier 2 system RAM. All 24,576 experts pinned there. And here’s the key move. The CPU computes the cache mist experts in parallel with the GPU using AVX 512 kernels. Neither waits for the other. The second compute domain, not

09:59 naive PCI Express swapping. Tier three, [music] your SSD, that 28.8 GB on table lives there. A few rows red per token through the OS page cache. It barely touches compute. Then speculative decoding. The model’s own MTP head drafts up to three tokens ahead and one verification pass over all 48 layers, checks them. The author reports 2.4 to 3.2 tokens accepted per pass, a 1.6 to 1.8 time speed up, and it’s lossless.

10:30 Same model, same distribution. There’s also prompt lookup drafting, grabbing up to five tokens from repeated context, worth 6 to 11% on code edits. for prompts chunked prefill up to 8,192 token chunks with the next layer’s experts streaming over PCI Express while the current layer’s attention runs. The author claims over 2,000 tokens per second of prompt ingestion at 32k context and at 64k context and beyond. KV streaming moves the KV cache into RAM

11:03 with only the hot window in VRAM, freeing space for roughly 2,300 more cached experts, which the author says actually raises speed at 262K context from about 51 to 63 tokens per second. Long context gets faster. Wrap your head around that. Okay, the numbers. And I want to be extremely clear about sourcing here. Every figure you’re about to see was measured by Strata’s author on one machine. That machine, an RTX 570

11:34 with 12 GB of VRAM. A Ryzen 57,64 GB of DDR5 on Windows. There are no independent thirdparty benchmarks as of October 4th, 2026 with that giant asterisk on screen. They’re wild decode speed. At the most aggressive quant Q20, the author claims 93 tokens per second at 4,000 context, still 60 tokens per second at the full 262K context. The more conservative IQ3 quant claims 53

12:06 tokens per second at 4K. [music] The pruned coder variant which keeps only 256 of 512 experts per layer retains 91.3% of full swbench verified performance and fits in 32 gigs of ram claims around 55 tokens per second at short context. So depending on quant 53 to 94 tokens per second from a 125 billion parameter model on a GPU you can buy in a normal store. prompt processing. The author reports roughly

12:37 1,300 tokens per second at 4K over 2,100 at 32K and a newer engine version 0.1.36 reportedly pushes that past 2,600 tokens per second with fused int 8 tensor core kernels. And the headline comparison, the author and early Reddit users report the same model running around 11 to 15 tokens per second through llama.cpp on similar hardware versus 62 to 65

13:08 through strata. Hence the four times faster than llama.cpp claim. Is that a fair fight? Partially. LMA.CP’s support for this brand new architecture was admittedly rough at launch. Optimization PRs were still landing in early October. The honest version, Strata was about four times faster than a general engine’s immature support for a 10-day old model. That’s still genuinely impressive specialization works, but it’s not the same as llama.cpp is slow. Meanwhile,

13:41 GitHub stars 0 to roughly 8,700 in about 10 days with 770 forks and 218 open issues. The community is stress testing it hard. Time for the tale of the tape. 10 dimensions. Rapid fire. One. Single user decode speed. Strata claims 53 to 94 tokens a second [music] on a 125 billion parameter model on an RTX 570. Llama.cpp

14:12 reportedly manages about 15 on the same setup. VLM and Llama.CP are actually near parody at batch size one. Single stream speed was never VLM’s game. Round to straighta asterisked two [music] concurrency. Strata handles exactly one request at a time. Llama.cpp barely scales. Vlm 35 244 times. Llama.cpp’s throughput under load per red hat. Round of ELM, no