Edge0 35B Tested: 0 Tokens on a 4090, 187 tok/s the Normal Way

Author: Prompt Engineer · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Edge0 35B Tested: 0 Tokens on a 4090, 187 tok/s the Normal Way · Channel: Prompt Engineer · 11:17 · Watch on YouTube

Topics: AI evaluation and benchmarks, Prompt and context engineering, Model inference and optimization, Local and open-source models

Summary

The video examines Edge0, a preview framework that runs a quantized 35B-parameter model by streaming experts from disk, and checks its claims against the project’s documentation and benchmarks. The presenter could not get it to produce tokens on an RTX 4090: four attempts ended in different errors, consistent with the release currently targeting Apple Silicon. Running the same base model through llama.cpp on the 4090 delivered about 187 tokens per second, supported the full 262K context window, and passed several retrieval, coding, and tool-use tests. The verdict is that Edge0 may help Apple Silicon users with limited memory, while Nvidia users with 24 GB should use the conventional approach for now.

Key points

Edge0 packages the Qwen 3.6 35B A3B model in 4-bit form with a LoRA and a pre-router, for a total folder size of 19.7 GB.
Its key techniques are SSD expert offload, preloading experts predicted to be needed next, and a LoRA intended to recover quality lost through quantization.
The documentation’s quality results are self-reported; the reported average is 79.2 versus 83.2 for the FP16 base model, with larger drops in math and coding.
The model configuration routes four experts per token, while the base model configuration specifies eight.
Four attempts to run Edge0 on an RTX 4090 failed with different errors, producing zero tokens; the project currently targets Apple Silicon.
The same base model run with llama.cpp on the RTX 4090 reached 187.47 tokens per second, using about 17 GB of VRAM.
The 4090 test measured about 22.6 GB of memory use at the full 262,144-token context window, which fits on a 24 GB card.
The base model retrieved a hidden code from a long prompt, passed six interval-merging tests, and called two supplied tools in parallel.
The presenter recommends Edge0 for Apple Silicon users who cannot fit a 17 GB quant in memory, but advises Nvidia users with 24 GB to use the regular model setup for now.

Chapters

Start	Chapter	Summary
00:00	Edge0 and its memory-saving approach	The presenter introduces Edge0, describes its model and expert-streaming techniques, and reviews the project’s benchmark claims.
03:45	Testing Edge0 on an RTX 4090	The video explains that the release targets Apple Silicon and follows four failed attempts to run it on Nvidia hardware.
06:21	Running the base model with llama.cpp	The presenter benchmarks the same base model on a 4090, measures context memory use, and tests retrieval, coding, and tool use.
10:10	Limitations and verdict	The presenter weighs Edge0’s limitations and benefits, then gives recommendations for Apple Silicon and Nvidia users.

Quotes

The answer to that last one is 187 tokens per second. — 00:33

Every single run crashed during prefill with index error, small vector out of range. — 05:49

Four attempts, four different errors, and still zero tokens. — 06:21

generation at 128 tokens 187.47 tokens per second — 06:56

If you have an Nvidia card with 24 GB, skip it for now. — 10:43

Concepts and entities

Edge0, Edge Zero AI, Qwen3.6 35B A3B, RTX 4090, RunPod, MLX, MLX-CUDA, llama.cpp, Apache License 2.0

Full transcript (original language)

00:00 This is a 35 billion parameter model running on an iPhone 16 Pro. 8 GB of RAM, airplane mode on, and the overlay says 1.8 GB of memory peak. That clip comes from the Edge Zero team. It sits right on their hugging face page. So, I rented an RTX 4090 on RunPod, installed their framework exactly the way the readme says, and pointed it at the model. The install finished with exit code zero. Then, the very first command died with libmx.2. So, cannot open shared object file. I tried three different fixes after that. every one of them crashed in a new place. By the end of this video, you’ll know what Edge0ero

00:33 does, which of its numbers hold up, why it would not run on a 24 GB Nvidia card, and what the same base model does on that card when you run it in the normal way. The answer to that last one is 187 tokens per second. Now, what is Edge0ero? It’s a preview release from a team called Edge Zero AI, published on September 8th. The base is Quen 3.6 35B A3B. That’s Alibaba’s mixture of experts model with 35 billion parameters in total and about 3 billion used per token. It already has 3.6 million downloads on its own. Edge Zero takes

01:06 that model, squeezes it to 4-bit integers, and adds two small adapter files, a Laura and something they call a pre-outer. The whole folder is 19.7 GB. Apache 2.0 license. In 6 days, it collected 1,6424 likes against 809 downloads. That ratio is unusually high and it makes sense once you see who can actually run it. The trick has three parts. The first is SSD expert offload. A mixture of experts model has 256 small expert networks in every layer and each token only visits a few of them. So edge zero leaves the

01:39 experts on disk. Memory maps the files and reads an expert only when the router picks it. RAM holds the active pieces and the rest waits on the SSD. The second part is the pre-outer. Reading from disk is slow. So they trained a small head that guesses which experts the next step will need and starts loading them while the current step is still computing. The readme says that’s worth up to plus 59% decode speed. There is no table behind that number. The third part is recover Laura quantizing to four bits loses quality. So they froze the 4bit base and trained Allora to imitate the full precision model.

02:12 That’s how they claim to stay within 3.9 points of the original. And get this, the readme’s model table says 256 divide vite 4 k equals 4. The config.json in the same repo says num experts per token 8. I went into the source code to check and the edge dur 35b tier config hardcodes top k equals 4. So edge runs half the routed experts the base model was trained with. That’s a real speed and memory labor and the headline never mentions it. To be fair to them, their quality numbers were measured on this pipeline. So the cost should already sit

02:43 inside that 3.9 point gap. Now the benchmarks five tests edge zero at int4 against quen at fp16 aime 202686 forces 92.7 human eval 90.9 versus 95.1 GPQA diamond 79.8 versus 81.8 mm pro 81 versus 84.6 if bench 57.9 versus 61.7 average 79.2 versus 83.2 2. Two things about that table. The average hides an

03:13 uneven drop. GBQA loses two points while math loses 6.1 and code loses 4.2. Those are the two tasks people most often run locally and the page says it plainly. All benchmarks were run by us. I looked for an independent check on artificial analysis livebench and alamarina and found nothing. The blog post covering this model repeat the vendor table. Nobody independent has measured it yet. The speed numbers come from one machine, a Mac Mini M4 Pro with 24 GB 14.9 to 17.7 tokens per second and 2.9 GB of peak active memory with a footnote

03:45 saying short contexts. The framework’s own tier config budgets 3,400 mgabytes for this model. A little above the headline, the full 19.7 GB checkpoint still has to live on a fast drive and the operating system will happily fill spare RAM with page cache while it reads. Now, the part I actually wanted to test. The Redmi has one sentence that decides everything for most people watching. The MLX backend currently targets Apple Silicon. The GitHub repo is more direct. The CUDA back end is on the road map. MLX is Apple’s machine

04:15 learning framework. I don’t own a Mac and most people running local models have an Nvidia card. So, I rented an RTX 4090 on RunPod. Secure Cloud 74 cents an hour, 24,564 MGB of VRAM, Ubuntu 2204. I followed the quick start line for line clone create event pip install. It pulled mlx030.6 the Linux wheel and printed successfully installed exit code zero. Then I typed edge0-help import error lib mlx dot so cannot open

04:47 shared object file. The plain Linux MLX wheel ships without the library it needs. I check the source tree. There’s a backends folder with exactly one thing in it. MLX the CUDA folder the readme calls reserved doesn’t exist. MLX does have an official CUDA build now. So fix number one was installing MLX CUDA. Same version 030.6. That one worked on its own. Default device GPU zero and a test matrix multiply came back correct. So I pointed edge zero at the weights. It loaded set at 690 MGB of VRAM and a minute later

05:18 died in the first mixture of experts layer runtime error. QMM NYI. QMM is quantized matrix multiply. NYI means not yet implemented. The CUDA backend in that MLX version can run a 4-bit expert. Fix number two was the newest MLX 0.32.2 that got past QMM, ran for 3 minutes and 14 seconds, and then stopped with a message telling me to raise MLX CUDA graph cache size above 400. So fix number three was doing exactly that. I set it to 4,000 and reran the benchmark

05:49 and three chats with the pre-outer on and off. Every single run crashed during prefill with index error, small vector out of range. Peak VRAM was 3.9 GB and peak RAM about 10 GB and zero tokens came out. Fix number four, forget the GPU. MLX has a CPU built too, and this pod has 128 cores. I hit the graphics card and ran the official benchmark on MLX CPU. It ground away for 9 minutes and 38 seconds, climbed to 9.9 GB of RAM, and then crashed inside the expert streaming layer. Rad time error. There is no stream CPU 36 in current thread.

06:21 Four attempts, four different errors, and still zero tokens. I want to be fair about what this shows. Edge0 says in its own readme that it targets Apple Silicon, and I ran it on something else. None of these crashes or bugs in the claims. They do tell you who this release is for today, Mac owners. The iPhone demo is further out of reach still because the repo contains no iOS app and no instructions for building one. So, what does the same base model do on this card when you run it the ordinary way? I built llama.cpp CPP from source commit 41 ABBFD with CUDA for the 4090 and downloaded Anslo’s IQ4 XS quant

06:56 of Quen 3.635B A3B that’s also 4bit 17.7 GB on disk and 16.5 GB loaded LLBA bench two repetitions prompt processing at 512 tokens 6983 tokens per second generation at 128 tokens 187.47 tokens per second plus or minus 1.28 28. Peak VR RAM 17 GB. For comparison, Edge0’s own number on a Mac Mini is about 15. That’s around 12 times faster. The price is 16 1/2 GB of video memory, which is the exact thing Edge0 was built to avoid spending.

07:30 Now, context. Before running anything, I did the math from config.json. Only 10 of the 40 layers use full attention. Every fourth one, the other 30 are linear attention with a small fixed state. With two key value heads and a head size of 256, the KV cache costs about 2 kilobytes per token per full layer. From 8K to 131K tokens, that predicts about 2,400 megabytes of growth. Measured on the 4090 with Llama server, 17,360 megabytes at 8K, 17,864

08:01 at 32K, 18,536 at 65K, 19,880 at 131K. That’s 2520 megabytes of growth within 5% of the prediction. And at the full 262,144 token window, 22,568 mgabytes, the whole native context fits on a 24 GB card. That hybrid attention design deserves credit, and it helps edge zero’s long context memory, too. A window you can allocate still has to be a window the model can use. So, I built a hstack of 6,218 boring warehouse lock sentences, 460,129

08:36 characters, and hid one line 46% of the way in. The master override code for the Kolkata depot is Indigo Pengalin 77. Then I asked for the code at the very end. 106,63 prompt tokens pre-filled at 5,368 tokens per second. 20 seconds wall clock. It answered Indigo Pengolin 77. Exactly. Now, the prompts from Edge0ero’s iPhone video on the base model, what is on device AI returned 567 tokens at 167.9 tokens per second in 3.6

09:06 seconds. The Shanghai question in Chinese, 847 tokens at 167.5. The anime question in Japanese, 966 tokens at 169.4. All three read cleanly in their language. For code, I asked for a merge intervals function and ran six tests against it, including an empty list and touching intervals. It fought for about 18,000 characters, wrote 5,536 tokens in 33 seconds, and passed six out of six. For tools, I gave it two functions, weather and calendar. In one sentence, check the weather in Kolkata and put the review on my calendar for my

09:39 4 p.m. meeting tomorrow. In one turn, it called both tools in parallel. It turned 4 p.m. into 1600 and tomorrow into September 15th without being told the format. Then it wrote the answer from the fake results and made no extra calls. Edge Zero’s readme says tool uses weak in its preview and I couldn’t test their build. So, this is the base model only. One miss. I asked for edge zero’s RTX490 row from the official benchmark table. A row that doesn’t exist. It correctly said it didn’t know the model. Then it offered a typical range of 30 to 70 tokens per second for this class of

10:10 model on a 4090. I had just measured 187 on that card. It guessed and the guess was wrong by almost three times. Now, the problems in order of how much they matter. First, it runs on Apple silicon only. On an Nvidia card, I got three different crashes and zero tokens. Second, the quality numbers are self-reported with the biggest drops in math and code. Third, the routed experts are cut from 8 to four in code, while the headline talks about memory only. Fourth, the framework pins MLXLm0.31.0, which is a yanked release on Pi Pi. The listed reason is batched KV cache crosscontamination and batch serving is

10:43 one of their advertised use cases. And the good parts are real. Apache 2.0, zero. A clear limitation section that admits weak agent skills. An architecture that streams experts from disk instead of pretending RAM is free on a Mac with 8 or 16 GB where Qen 3.635b simply doesn’t fit. A 15 token per second version is a real option. So here’s my verdict. If you have an Apple silicon Mac with too little memory for a 17 GB quant, Edge0 is worth trying today and it’s the reason this project exists. If you have an Nvidia card with 24 GB, skip it for now. Run the regular QN 3.6

11:17 6 IQ4XS in llama.cb CBP 187 tokens per second, the full 262K context, and it found the needle. With 16 GB, you’ll want a smaller 3-bit quant or partial CPU offload, which I didn’t test here. With 8 GB on Nvidia, neither path works well yet. And the CUDA backend, Edge, promises is the one to watch. If you’re interested in reading further, I’ll link to this main page in the description below. Every command and every number in this video came from that runpod session, and the transcripts are linked, too. Thanks for watching.