70B on a 4GB GPU? I Tested It (2 Minutes Per Token)

Author: Prompt Engineer 48 · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: 70B on a 4GB GPU? I Tested It (2 Minutes Per Token) · Channel: Prompt Engineer 48 · 15:43 · Watch on YouTube

Topics: AI evaluation and benchmarks, Model inference and optimization, AI infrastructure and hardware, Local and open-source models

Summary

The video tests AirLLM’s claim that a 70-billion-parameter model can run on a GPU with less than 4 GB of memory. Using Qwen 2.5-72B, the test measured about 2.3 GB of peak GPU memory, but generation in 16-bit took about 121.5 seconds per token on the tested setup. A 4-bit AirLLM run was much faster after its files were cached in system memory, while llama.cpp with a 4-bit model was faster still, at about 3.4 seconds per token. The presenter concludes AirLLM is useful for running a model that otherwise will not fit, if you can tolerate slow output and have ample disk space, but it is unsuitable for interactive chat.

Key points

AirLLM’s README claims 70-billion-parameter models can run on a 4 GB GPU without quantization, but the README does not provide a speed figure for that claim.
AirLLM loads one layer at a time from disk; the tested 16-bit model used about 136 GB of layer files, and the initial split required roughly that much disk space.
The test used Qwen 2.5-72B Instruct on an RTX A5000 pod; the model download was 145.4 GB and the measured disk read speed was 601 MB/s.
Peak GPU memory for the 16-bit run was 2,391 MB according to Torch and 2,675 MB according to NVIDIA-SMI.
The 16-bit run took 121.5 seconds per generated token on average across the three-token run; the presenter notes this is one test on a network disk, not a universal speed.
AirLLM’s 4-bit mode initially failed because a size check in the code appeared to calculate the required space incorrectly; bypassing that check allowed the benchmark to run.
The 4-bit AirLLM run took 117.5 seconds per token cold and 24.9 seconds per token with a warm cache, likely because the 39 GB files fit in the pod’s available memory.
A llama.cpp 4-bit Q4_K_M run generated about 0.29 tokens per second, or roughly 3.4 seconds per token, using about 4.7 GB of GPU memory with seven layers on the GPU.
The presenter recommends AirLLM for occasional checks or overnight batch jobs when a model will not fit elsewhere, but not for chat or other interactive use.

Chapters

Start	Chapter	Summary
00:00	The 4 GB claim and the speed trade-off	The presenter introduces AirLLM’s memory claim and explains that the test will also measure its generation speed.
02:06	How AirLLM fits a large model on a small GPU	The video explains layer-by-layer loading from disk and estimates a token’s disk-reading cost before testing.
03:43	Test setup and memory measurements	The presenter describes the rented GPU pod, model download, disk measurements, and the method used to track memory and time.
06:21	16-bit results and the 4-bit cache effect	The 16-bit run stays under the memory limit but is slow, while the 4-bit run speeds up substantially when its files are cached.
10:34	Comparison with llama.cpp	A 4-bit llama.cpp run is much faster, with the trade-off that it requires substantial system memory.
12:05	Limits, reproduction, and verdict	The presenter clarifies untested claims, gives reproduction guidance, and summarizes when AirLLM’s slow, low-GPU-memory approach makes sense.

Quotes

So, the claim is true and it is also the reason you probably should not use this for chat. — 00:00

The memory claim holds and it holds with room to spare. — 06:21

If the question is, can you chat with it? The answer is no. — 07:27

If you have the RAM, llama.cpp wins by a huge margin. If you do not have the RAM, AirLLM is the one that still runs. — 11:34

Where it makes sense, you have a model that does not fit anywhere else, you can wait, and you have a lot of disk. — 14:42

Concepts and entities

Air-LLM, Qwen 2.5-72B Instruct, llama.cpp, NVIDIA-SMI, RunPod, quantização de 4 bits, cache do sistema operacional

Full transcript (original language)

00:00 This is a 72-billion parameter model and the GPU memory peak while it ran was 2391 megabytes. That is under the 4 gigabytes the project promises. Now, look at the next line in the log. Three tokens took 364 seconds. That is about 2 minutes for every single token. So, the claim is true and it is also the reason you probably should not use this for chat. I’ll show you exactly what I tested with the real logs on screen at the end. I’ll tell you the one setup where this makes sense and the setup where a plain 4-bit model is 35 times

00:32 faster. And before I measured anything, I wrote down a guess for the speed. I was wrong by a factor of two and I’ll show you why. All right, let’s go ahead and start with the repo. This is the Air-LLM repo. You can see it has about 36.9 thousand stars, about 3,800 forks, and the license here is Apache 2.0. The latest release is version 4.0.0 from the 5th of September. The project has shipped three releases since the middle of August. And the first paragraph of the readme says, “Air-LLM lets 70-billion parameter models run on a

01:04 single 4-gigabyte GPU without quantization, distillation, or pruning.” Then it goes bigger. Kimi K3 at 2.8 trillion parameters in under 4 gigabytes. QN 3.8 flash next 125 billion on 6 and DeepSeek V3 671 billion on about 12. Now, mind you, the readme says those were measured end-to-end on one RTX 4090, one RTX 3090, and one RTX 6000 Ada. Those are the projects’ own measurements. I could not find an

01:34 independent benchmark for any of them. Here’s the thing I checked. I searched this readme for tokens per second, seconds per token, latency, and throughput. Zero matches. There is no speed number for the 70-billion claim. The readme tells you the memory. It does not tell you the speed. One more detail. Scroll down to model compression. It says compression can speed up inference by up to three times and it is block-wise quantization. So, the headline without quantization is the default mode. The fast mode is quantized. I tested both and I’ll show you what each one does. And the Readme’s

02:06 own update list has two older speed notes. In December 2023, pre-fetching gave a 10% speed improvement and compression was advertised as a three times speed up. So, how does a 72 billion parameter model fit in under 4 GB? It does not stay there. RLLM splits the model into one file per layer on your disk. It loads one layer into the GPU, runs it, throws it away, and loads the next one. 80 layers one at a time for every single token. The Readme FAQ says the splitting step is very disk-consuming, so make sure you have

02:37 the space. Keep that in mind because it matters later. Now, let’s do the math because this is where my skepticism comes from. In 16-bit, this model is about 136 GB of layer files. To produce one token, every layer has to come off the disk. So, one token means reading about 136 GB. The GPU is not the bottleneck, the disk is. Now, here is a number I worked out myself, so treat it as my arithmetic. This model has 80 layers. 136 GB divided by 80 is about 1.7 GB per layer. So, the 4 GB headline

03:10 really means one layer on the GPU plus working memory. The peak I measured, about 2.3 GB, is roughly one layer plus the rest of the process. I assumed a fast NVMe drive around 3 and 1/2 GB per second. 136 / 3.5 is about 39 seconds. So, my prediction, written down before I ran anything, was 30 to 60 seconds per token. Write your own guess in the comments. I’ll tell you at the end who got closest. I did not run this on my laptop. My laptop has an RTX 4060 with 8

03:43 GB and about 120 GB of free disk. This model needs about 145 GB just to download. So, I rented a GPU pod and RTX A5000 with 24 GB for $0.27 an hour. That means the card is much bigger than 4 GB. So, I do not judge the claim by the size of the card. I judge it by the peak memory the process actually used. I logged two numbers, Torch’s own peak counter and NVIDIA-SMI. I picked Qwen 2.5-72B

04:14 instruct because it is not gated, so anybody can repeat this. It is 72.7 billion parameters, so it is not exactly 70. The download is 37 files, 145.4 GB. On this pod, it finished in 72 seconds. The disk is the part that is not fast. I measured it with DD, 251 MB per second writing and 601 MB per second reading. Remember that, 601. Install is one-line pip install AirLLM version 4.0.0 on Torch 2.8 with Transformers 5.19. Then,

04:47 the first run splits the model into layer files that took 361 seconds for the 16-bit version and it wrote 136 GB. So, you need roughly double the model size in free disk to get started. That is the gotcha the README FAQ was talking about and it is the first real cost of the 4 GB headline. I should tell you the first attempt failed. My first pod was an RTX 4090 in the community cloud. The download and the split worked. Then, when AirLLM touched the GPU, Torch crashed with quote “CUDA unknown error.”

05:19 I called the driver directly and she went in it returned 999. I restarted a pod, same error. I made a second pod on a different host, same error. So, it was not one bad machine. I deleted both and rented an RTX A5000 in the secure cloud. This time Torch printed CUDA okay and I had to download and split everything again. So, the numbers you will see all come from that third pod. Here’s how I measured. The script starts a monitor thread. Every second it asks NVIDIA-SMI for the memory used and it reads the

05:49 bytes this process has pulled from storage. Then it times the generate call with a plain stopwatch and reads torches own peak memory counter. Two independent memory numbers, one clock, nothing hidden. First claim memory, one prompt 20 tokens long, explain in two sentences why reading model weights from disk is the bottleneck. I logged the peak. The peak was 2391 megabytes by torches counter and 2675 by Nvidia-smi. The claim was 4 gigabytes, that is 4096 megabytes, so the memory claim holds and

06:21 it holds with room to spare. So, credit where it is due, the project said it fits and it fits. I want to say that clearly before I get to the part that is less nice. And the answer came out as real text. The model produced the words, quote, reading model weights, that is three tokens and I’ll come back to why I only generated three. Now, the number nobody printed one new token took 107.7 seconds. Three new tokens took 364.6 seconds, which is 121.5 seconds per token. My prediction was 30 to 60

06:53 seconds. The real number is about double the top of my range, so I missed. Why? Because the disk on this pod reads 601 megabytes per second, not the three and a half gigabytes I assumed. The monitor shows reads up to about 2 gigabytes per second at the peak and a median of 710. So, the disk is the bottleneck and the exact speed depends on your disk and on how much of it your operating system has cached. Now, why only three tokens? Because look at this, a normal answer of about 200 words is around 270 tokens. 270 * 121.5 seconds is 32,805 seconds,

07:27 that is 9.1 hours for one answer. So, if the question is, can a 4 gigabyte GPU run a 70 billion model without quantization? The answer is yes. If the question is, can you chat with it? The answer is no. And I want to be fair about what I tested. One prompt, three generated tokens, one run, one rented GPU, and a network disk. So, treat the 121 seconds as the measured number on this setup, not a law of nature. If you have a fast local NVMe, your number could be lower. I would love to see yours. Let me read the log slowly, so

07:58 nothing is hidden. Load and split 361 seconds. That happens once, then run one one token, then run two three tokens. Notice both runs print the same peak. 2391 then 2393. The memory does not grow with the number of tokens. That part of the claim is solid. Now, let me sanity check my own numbers. 136 GB bytes is about 146 billion bytes. Divide that by 121.5 seconds, and you get about 1.2 GB per second. That is faster than the 601

08:29 MB per second I measured with DD. So, prefetching and the cache are helping here, even for the 16-bit files. Without them, it would be slower than 121. Test three. The readme says 4-bit compression gives up to three times the speed. So, I ran the same model, same prompt, with compression set to 4-bit, and it refused to start. Look at this error. Not enough space. Model total size 481 GB, but the 4-bit files are only about 39 GB. So, where does 481 come from? I opened the

09:00 code. This is utils.py in the repo, lines 447 to 468. For 4-bit, line 458 divides the size by 0.2813. So, it multiplies by about three and a half instead of shrinking it. That looks like a bug. I bypassed that one check only in my own benchmark script, and the run went through. The split took 352 seconds and wrote 39 GB versus 136 for the 16-bit version. Now, the speed. The first token took 117.5 seconds. That is

09:30 the cold run, and it is no faster than 16-bit. It is actually a bit slower. Then, the three token run took 74.8 seconds, which is 24.9 seconds per token. That is almost five times faster than the 121. So, is the three times claim true? Here is what I think is going on. This pod gives the process about 50 GB of memory. The four-bit files are 39 GB, so after the first pass, they sit in the operating system cache, and the next token comes from memory, not from the disk. The 16-bit files are 136 GB. They cannot fit. So,

10:03 every token is a disk read. So, the speed up is real, but it comes from the files being small enough to stay in RAM. If your machine has the memory, great. If not, the cold number is the one you get. And remember, this mode is quantized, so the headline, no quantization, no longer applies here. Peak GPU memory in this mode was 3,059 MB by torch, still under 4 GB. In the four-bit run, the disk monitor shows a median of 449 MB per second. The files are smaller, so the disk has less to do.

10:34 The fair question is, what else could you do with the same hardware? So, I ran llama.cpp with the four-bit Q4KM version of the same model. That file is 44 GB. I kept only seven layers on the GPU, which uses about 4.7 GB of GPU memory, and the rest sits in system memory. The result, 0.29 tokens per second for generation. That is about 3.4 seconds per token. With zero layers on the GPU, it was 0.28, almost the same. The GPU hardly matters when the weights sit in memory

11:04 because memory bandwidth is the limit. Put them side by side, air-llm 16-bit, 121.5 seconds per token. air-llm four-bit with a warm cache, 24.9. And llama.cpp four-bit, about 3.4. So, llama.cpp is about 35 times faster than air-llm in 16-bit, and about seven times faster than air-llm in four-bit. A 200-word answer would take about 15 minutes on llama.cpp on air-llm 16-bit, over 9 hours. But, this is not a fair

11:34 fight. So, here is what each one costs. llama.cpp needs about 44 GB of system memory for this file. AirLLM needs almost no system memory and about 2.6 GB on the GPU, but it needs the layer files on disk. 136 GB in 16-bit and AirLLM keeps the original precision. While this llama.cpp file is 4-bit. So, it is a trade. If you have the RAM, llama.cpp wins by a huge margin. If you do not have the RAM, AirLLM is the one that

12:05 still runs. Could AirLLM be faster? I did not test this, so it is my reasoning, not a result. Every token reads all 80 layers once, whatever you do. So, anything that gives you more tokens per pass over the disk should cut the disk cost per token. For example, running several prompts together as one batch. I have not tried it, so I will not give you a number. The other lever is simpler. A faster drive or more RAM for the cache. The 4-bit run showed it. Same GPU, same code, and a warm cache took the time from 117 to 25. The readme

12:36 also claims Kimi K3 at 2.8 trillion in under 4 GB, and training a 125 billion model in under six. I did not test any of those. The Qwen 125 billion one needs about 360 GB of disk, which the readme says itself. So, treat those as the project’s claims, not mine. And the limits of my test, once more, 72.7 billion parameters, not exactly 71 prompt of 20 tokens, three generated tokens, one run each. If you run it on your machine, your numbers will be

13:07 different, and I want to see them. Put them in the comments with your disk and your amount of RAM. If you want to repeat this, here is exactly what I ran. pip install AirLLM, then auto model from pre-trained pre-trained pre-trained with the huggingface repo and a layer shard saving path on a disk with lots of space. For the 4-bit mode, add compression equals 4-bit, and install bits and bytes first. If you hit the space error I showed you, that is the line 458 bug, not your disk. Then call generate with a small number of new tokens. Three is enough to get a speed

13:38 number. Do not start with 200 unless you want to leave it running overnight. And write down your disk type, your RAM, and whether it was the first run or a repeat. Those three things explain most of the difference between your number and mine. What did all this cost? RunPod’s billing shows about 47 cents of pod time for the whole project. That includes the two pods that failed and the one that worked. So, you do not need a big budget to check a project’s claims. You need a few dollars, a plan, and the patience to repeat the download when a host breaks. Before the verdict, here is a scorecard of the claims I

14:10 could test. 70 billion model in 4 GB measured true. Without quantization, true in the default mode. Up to three times faster with 4-bit, almost five times with a warm cache, and slightly slower cold, and no speed number in the readme at all. The big claims about Kimi and the 125 billion model, I did not test. Let me put everything in one table. Memory, 16-bit, 2.6 GB by nvidia-smi. 4-bit, 3.2. llama.cpp, 4.7 with seven layers on the GPU. Speed, 121

14:42 seconds per token for 16-bit, 117 cold, and 25 warm for 4-bit, and 3.4 for llama.cpp. So, the answer, can AirLLM run a 70 billion model in under 4 GB? Yes, I measured 2.3 GB by torch and 2.6 by nvidia-smi. Is it fast? No, about 2 minutes per token in 16-bit on this setup. And I guessed 30 to 60 sec. Where it makes sense, you have a model that does not fit anywhere else, you can wait, and you have a lot of disk.

15:12 Overnight batch jobs, one-off checks, trying a huge model once. Where it does not, chat, agents, anything where you sit and wait for the answer for that. A four-bit model in llama.cpp is about 35 times faster. And the guess winner is whoever wrote a number closest to 121 in the comments. Thanks for guessing. It made me write mine down, too. So, in this video, we have seen what air LM really does on a small GPU with the numbers the readme leaves out. If you want to try it on your own system, the repo link is in the description, and

15:43 tell me what speed you get. I’ll see you in another one. Check out the next video.