This Open-Source Engine Claims 2x Faster Than llama.cpp

Author: Better Stack · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: This Open-Source Engine Claims 2x Faster Than llama.cpp · Channel: Better Stack · 07:51 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, AI developer tools, Model inference and optimization, Local and open-source models

Summary

Magnitude is an early open-source inference engine that benchmarks kernel configurations on a user’s hardware and caches the fastest setup. The video tests its speed claims against llama.cpp and describes features aimed at local coding agents, including one-click connections to Claude Code through compatible local APIs. Results vary across hardware and tests: Magnitude approaches twice the decode speed in one benchmark, but independent comparisons also show it trailing other engines. The presenter sees potential in its hardware tuning and ease of use, while recommending caution for M5 users and production workloads.

Key points

Magnitude tests different kernel configurations on the local machine when a model is downloaded, then caches the fastest configuration.
The video distinguishes prefill speed, which measures prompt reading, from decode speed, which measures token-by-token generation.
Magnitude provides OpenAI-compatible and Anthropic-compatible APIs on localhost, with a one-click connection to Claude Code.
The engine is written in Rust and uses a numerical compiler called Seismic, with CPU, Metal, CUDA, and Vulkan backends.
Its agent-focused features include shared prompt caches across sessions, adjustable memory use, compressed KV cache, and loading and unloading models as needed.
In Magnitude’s cited M4 Pro test, decode improved from 30 to 57 tokens per second, while prefill was only 9% faster.
Other reported comparisons vary: some users found Magnitude slower than Rapid MLX or llama.cpp, including tests on M5 Max and RTX 5070 Ti.
The software is early, with a small curated model library, and the presenter could not find instructions for loading custom GGUF files.
The presenter considers Magnitude worth watching for M1-to-M4 users running coding agents, but advises M5 users focused on speed to stay with MLX for now.

Chapters

Start	Chapter	Summary
00:00	How Magnitude tunes itself	The video introduces Magnitude’s hardware-specific kernel benchmarking and explains how it differs from engines using broadly compatible pre-compiled kernels.
02:09	Testing the app and its agent features	The presenter demonstrates model setup, a side-by-side comparison with llama.cpp, and Magnitude’s one-click local connection to coding agents.
04:16	What the benchmarks show	Magnitude’s near-2x decode result is put in context alongside its slower prefill gain, differing test configurations, and mixed independent comparisons.
06:18	Limitations and who should try it	The video covers Magnitude’s early state and practical annoyances, then gives recommendations for users on different Mac generations and production workloads.

Quotes

Magnitude is a new open-source inference engine that claims it can run local models up to twice as fast as llama.cpp on Mac. — 00:00

It finds the fastest one and it caches that setup for your device. — 01:06

Claude Code is now talking to a model running locally on my Mac with no complicated setup. — 02:41

2x faster does not mean everything is twice as fast. — 04:47

Your local inference engine shouldn’t have to guess what hardware you’re running. — 07:19

Concepts and entities

Magnitude, llama.cpp, Ollama, MLX, Claude Code, Qwen, seismic, vLLM, SGLang, Better Stack

Full transcript (original language)

00:00 Magnitude is a new open-source inference engine that claims it can run local models up to twice as fast as llama.cpp on Mac. And the way it does it is actually pretty cool. Instead of using the same pre-compiled kernels for every machine, Magnitude benchmarks different kernel configurations directly on your hardware and keeps the fastest one. So, I wanted to see how much of a difference this actually makes and whether this is actually worth using over llama.cpp or MLX.

00:35 All right. So, what’s the basic thing here? Well, the problem Magnitude is solving is that engines like llama.cpp or Ollama need to run on a lot of different hardware. So, they ship pre-compiled kernels that work across a wide range of chips. That makes them portable, but those kernels aren’t necessarily the fastest possible version for the machine you’re actually using. Magnitude is approaching this a little bit differently. When you download a model, it tests different kernel configurations on your machine. Here’s

01:06 really what it does differently. Its kernels have different knobs it can adjust. When you download a model, Magnitude runs a bunch of variations directly on your hardware. It times them. It finds the fastest one and it caches that setup for your device. And according to it, right, it takes around a minute. And that part appears to be real. The question is whether it can actually make a difference. If you enjoy coding tools that speed up your workflow, be sure to subscribe. We have videos coming out all the time. All right. Now, there are two numbers we need to watch here. Pre-fill is how

01:37 quickly the engine reads your prompt. Decode is how quickly it generates the response one token at a time. And if you’re sitting there watching a coding agent work, decode is usually the number you actually feel. So, let’s race it. So, I’ve already downloaded the app from their GitHub repo and this is what it looks like live. You’re brought to the discover tab, which is where you want to be. Magnitude looks at the hardware in this Mac and recommends model that should fit. I’m going to grab Quinn 3.5 4B the Q8. I’m going to hit download.

02:09 Now, what does it do? It assesses the machine, it downloads the model, and it runs that tuning process. You can also download other models, too, as long as you have the space for them just here on the catalog. If we jump back here, I’m going to give it some time. It’s going to take time to run. And once it’s finished, we can actually compare it. Now, doing a quick side-by-side here, I have llama.cpp and magnitude. The same model, I’m going to give it the same prompt. llama.cpp is on the left, magnitude is on the right. Watch the tokens per second. And

02:41 whatever result we get here, keep it in mind because later the independent benchmarks get a lot messier. But before we get there, this is actually the part of magnitude that I really like. I can open connections, I can click connect Code, and I can restart it. That’s really it. And Claude Code is now talking to a model running locally on my Mac with no complicated setup. Magnitude does that by exposing both the OpenAI compatible API and the Anthropic compatible API on localhost. So, with one click, you can hook it into Claude

03:13 Code, Code X Open Code Klein Pi, and more. And that tells you what magnitude is really trying to be. This is not just another local chat app. It’s built around agents. Under this, the engine is written in a rust and sits on a numerical compiler they call seismic. It has backends for CPU, metal, CUDA, and Vulcan. And the agent side gets a few specific optimizations. Multiple agent sessions can share their prompt caches. Memory can grow and shrink as needed. The KV cache, basically the model’s

03:45 short-term memory, gets compressed. And models can load when you need them, then unload when they sit idle. So, this is pretty cool. They’re not just saying we made llama.cpp faster, this is rather more we tune the engine for your exact hardware, then optimize it around long-running agents. And that puts this in a rather unique position. llama.cpp in Ollama trade some speed for the ability to run almost everywhere. vLLM and SGLang are designed around data center batching. MLX engines can be

04:16 extremely fast, but they’re Apple only. So, Magnitude is a bit different. Measure the exact machine you’re running on, tune for it, and build around agent workloads. On paper, this sounds great. But now we get to the number Magnitude is using to prove it, the 2x. Because this is where things change. The original test used Qwen 3.6 35B A3B at 4-bit with a 64,000 of context on an M4 Pro with 48 gigs of memory, decode went from 30 tokens per second to 57. That’s

04:47 a 92% improvement, which is almost 2x. Now, again, that sounds good. But prefill, actually reading the prompt, was only 9% faster. So, right away, 2x faster does not mean everything is twice as fast. And just look at the benchmark itself. They pasted Moby Dick into the context up to 64,000 tokens and asked the model to repeat the final section. On Nvidia’s DGX Spark running that same test, the decode improvement dropped to

05:17 19%. Then, there’s another thing. The two engines weren’t configured the same way. llama.cpp was using a full 16-bit KV cache because its compressed cache path broke during their test. Magnitude compresses its cache by default. Then a lot of independent users started testing it, and this is where things get even less clean. On an M5 Max, one person got 175 tokens per second with Rapid MLX. Magnitude got 161. Another user found

05:48 Llama CPP running twice as fast as Magnitude on an M5 Max and on an RTX 5070 TI. Someone measured Llama CPP around 20 to 30% faster and now we have Magnitude almost doubling Llama CPP in one test while losing it in another. So, it kind of depends on what machine we’re running this on. Now, the founder has an explanation, at least for this part. Magnitude doesn’t use the M5’s new Metal Matrix hardware yet. So, apparently, they say an official MLX comparison is

06:18 coming soon. Now, again, this is extremely early as with a lot of these tools. It’s version 0.2.5. They shipped five releases in about four days and the entire engine was rewritten in September. Right now, the model library is about 15 curated models. Nothing below 4-bit and I couldn’t actually find docs for loading your own GGUF file. Now, a few annoyances here. There are things like Mac OS 13 or newer. Intel Macs only run on CPU.

06:49 Downloaded models live inside a hidden Magnitude folder in your home directory and deleting the app does not delete those models. So, you’ll need to clean that folder up yourself as you delete the models. And if a model has been idle, the first response can be slow because Magnitude has to load it again. So, are you going to install this? Well, if you’re running on an M1 through M4, you run coding agents locally and you want a really simple back and forth um Magnitude is pretty cool. We’ll see where it goes. Honestly, the connect button, how we connect Claude code,

07:19 might be the strongest reason to install it. If you’re on an M5 and all you care about is raw tokens per second, just stay on MLX for now. And if you’re thinking about using this for anything production-related, I’d wait for more independent MLX comparisons cuz we’re not quite there yet. Because after digging through all of this, I don’t think sometimes it’s faster, sometimes it isn’t. The idea is a lot simpler. Your local inference engine shouldn’t have to guess what hardware you’re running. It can just measure it and tune itself around the machine sitting in front of you. I’m Josh from Better

07:51 Staff. If you enjoy coding tips and tricks like this, be sure to subscribe. We’ll see you in another video. >> [music]