INSANE: This 46 MB AI Model Runs 5x FASTER on My Laptop (Frozen Embeddings TESTED)

Author: Prompt Engineer 48 · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: INSANE: This 46 MB AI Model Runs 5x FASTER on My Laptop (Frozen Embeddings TESTED) · Channel: Prompt Engineer 48 · 05:42 · Watch on YouTube

Topics: AI evaluation and benchmarks, Data, databases and embeddings, Model inference and optimization, Local and open-source models

Summary

The video tests a method that replaces a large, learned language-model embedding table with a frozen 16.8 MB binary vocabulary code. In the reported experiments, the frozen-table models match or outperform the learned-table model on several benchmarks, with particular gains on rare words, while a ternary version runs about five times faster than an 8-bit baseline on the creator’s laptop. The creator also checks the kernel’s reported loss and tests adding new biomedical tokens, while noting limitations including weak results on control tokens and emoji prediction, and the study’s small scale. The video concludes that frozen embeddings could be useful for tiny CPU models, while pointing out licensing conditions and previewing work toward a shared table for text, images, audio, and video.

Key points

The video says Gemma 3’s 270-million-parameter model uses 168 million parameters for its embedding table.
The proposed method reduces the vocabulary to 512-bit sign codes per token, totaling 16.8 MB, and keeps the table frozen.
Two small linear maps connect the codes to the model body; input and output share the table.
On WordSim, the codes score 0.617 compared with 0.614 for the original float embeddings.
Across three model arms trained on the same data and steps, the frozen-table model ties or beats the learned-table model on several benchmarks; the learned table leads on ARC-E.
The frozen table performs better on rare words, while unseen token rows in the learned table remain random.
On the creator’s laptop, the 46 MB ternary model reaches 517 tokens per second using 63 MB of memory, versus 103 tokens per second and 471 MB for the 8-bit baseline.
The creator’s measured next-token loss matches the paper’s reported value of 4.495.
Adding 10,000 biomedical tokens without retraining hurts the learned table more than the frozen table, but the frozen table does not lead on new control tokens or emoji prediction.
The video flags the limited scale of the study, differences in code bases for the speed comparison, and non-commercial research and Gemma licensing terms.

Chapters

Start	Chapter	Summary
00:00	The embedding-table problem	The video introduces the claim that a large share of a small model’s parameters is devoted to its vocabulary and outlines a frozen-embedding alternative.
01:04	How the frozen vocabulary works	The method converts Gemma embeddings into compact sign codes and compares their word-similarity score with the original embeddings.
01:34	Training and benchmark results	Three model configurations are compared, with the frozen-table models matching or beating the learned table on most of the reported benchmarks.
03:07	Laptop speed and memory test	The creator runs the ternary model on a laptop, compares its speed and memory use with an 8-bit baseline, and checks the reported loss.
04:41	Limitations, licensing, and conclusion	The video discusses new-token results and study limitations, notes applicable license terms, and summarizes the case for frozen embeddings in tiny CPU models.

Quotes

63% of a small language model is not the brain, it is the dictionary. — 00:00

512 bits per token. — 01:04

The codes arrived already knowing the word. — 02:37

The ternary kernel is five times faster. — 03:38

For tiny models on a CPU, the evidence says a lot of it, yes. — 05:11

Concepts and entities

Gemma 3, embedding, BitNet, GPTQ, AWQ, WordSim, Lambada, HellaSwag, PIQA, ARC-E, llama.cpp, Zig, Shadow 250M, Shadow 50M

Full transcript (original language)

00:00 63% of a small language model is not the brain, it is the dictionary. In Gemma 3’s 270 million model, 168 million parameters do one job, they look up words. So, one researcher asked a rude question, what if that dictionary was never trained at all? This is embeddings are overpriced, a frozen 16 megabyte vocabulary for tiny models. I’m prompt engineer, I read the paper, pulled the repo, and ran the whole thing on my own laptop. Real numbers coming

00:31 up. First, where the weight lives. The embedding table is 62.6% of the model. The feed forward layers, where the thinking happens, are 26. And compression doesn’t fix it. BitNet, GBTQ, and AWQ all leave the embedding in 16-bit. Squeeze the body to ternary, and you still carry about 335 megabytes of vocabulary next to a 20 megabyte brain. The paper’s bet, a token doesn’t need 640 learned floats, it needs a code. Here’s the recipe. Take Gemma’s own embedding, center it, keep the top 512

01:04 principal directions, then throw away everything except the sign, plus one or minus one, 512 bits per token. 262,000 tokens * 512 bits, 16.8 megabytes, and it never trains again. Two tiny linear maps connect the codes to the body, together 650,000 parameters. That is all the vocabulary that learns. On the way out, the same codes score the answer. Input and output share one table and a fixed frequency prior called the

01:34 seal adds 1 megabyte. Does a frozen table still know what words mean? On WordSim, the codes score 0.617, the original float embeddings score 0.614. Now, the experiment. Three models, same data, same steps, same seed, only the vocabulary changes. Arm A, the normal learned table, 268 million trainable parameters. Arm B, the frozen table with a float body just 101 million tradable. Arm C, the frozen table with a ternary bit net body. Every weight is minus one

02:06 zero or plus one. Each arm trained on two billion tokens of fine web edu on one RTX 5090, about $18 an arm. The whole study under 75. So, what does the learn table actually buy? 0.2 nats of validation loss for 2.6 times the trainable parameters. But loss is not the test. On Lambada, the frozen table wins by 1.6. On HellaSwag and PIQA, it ties. Only ARC-E favors the learned table by 3.4 points.

02:37 That is the whole return on 168 million parameters one benchmark. The ternary model ties the learned one on Lambada and PIQA in a 46 megabyte file. Where does the learned table win? In the middle on words it has seen hundreds to thousands of times. On rare words seen 10 to 19 nine times, the frozen table wins by up to 2.3 nats. The codes arrived already knowing the word. Over 52,000 of Gemma’s tokens never appear in training. In the learned table, those rows stay random. In the

03:07 frozen one, they keep their meaning. Now, the part I care about, does it actually run on a normal laptop? The authors released a single C file for exactly that. So, I tested it. No C compiler on this machine. So, I installed Zig and built their kernel with the exact flags from the paper. The 46 megabyte weights file loaded and I ran their benchmark prefill 128 tokens, then generate 64. Result on my machine, 517 tokens per second in 63 megabytes of memory.

03:38 The paper reports 62 megabytes and 615 to 740 tokens per second on its Xeon. Memory matches almost exactly. Speed is a bit lower here. Then the baseline same model learned table 8-bit GGUF in llama.cpp, same eight threads. 103 tokens per second. The ternary kernel is five times faster. Memory tells the same story. 471 MB at 8-bit, 63 for the frozen ternary model.

04:09 Why can’t the normal model shrink? Every quantizer keeps that big table at 16-bit. 250 MB of the 4-bit file is just vocabulary. Is the kernel honest? I fed it there are 64 validation tokens. The paper reports a next token loss of 4.495. I got 4.495. Now, the trick I did not expect adding new words after training. The authors took 10,000 biomedical words and added each as a single new token. No retraining. The learned table collapsed 43% worse. The frozen table lost only

04:41 3%. Why? A frozen code comes from a rule. Average the pieces, take the sign, done. A learned table has no rule. It’s new rows start out random. And the paper is honest about what failed. Brand new control tokens and emoji prediction did not favor the frozen table. One seat per arm, 2 billion tokens which under trains the big table. The learned table’s edge probably grows with more data. The speed claim comes from one laptop and two different code bases. I have now

05:11 added my own laptop. The ratio held up. Also, read the license, non-commercial research only, and Google’s Gemma terms apply to the table. The lineage matters. The same author shipped Shadow 250M and Shadow 50M first. Neither was a control test. This paper is So, is the embedding overpriced? For tiny models on a CPU, the evidence says a lot of it, yes. 16 MB of fixed code matched or beat 168 million train parameters on three of four benchmarks. And next, the author

05:42 teases one frozen table on a text, images, audio, and video, still running on a CPU. Everything is released, weights, data, logs, and every evaluation answer. Links in the description. Remember the dictionary at the start? It turns out you may not need to teach it a single word.