NCM: The End of LLMs? Not So Fast (Local Tests)

Author: Prompt Engineering · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: NCM: The End of LLMs? Not So Fast (Local Tests) · Channel: Prompt Engineering · 14:27 · Watch on YouTube

Topics: AI evaluation and benchmarks, AI content generation, AI models, Local and open-source models

Summary

The video examines Stanford’s Next Concept Modeling (NCM), which generates text from coarse outlines to finer details rather than predicting one word at a time. The presenter explains NCM’s design and tests its speed and Sudoku performance against other models on a DGX Spark. NCM is faster for single requests in these tests, but slower when handling batches of requests; its Sudoku advantage also narrows when a diffusion model fills in its most confident cells first. The video concludes that NCM is a promising idea with notable limitations, including fixed-size outputs, errors with names and numbers, and a lack of large-scale testing.

Key points

Autoregressive language models generate text token by token, repeatedly reading model weights from memory.
Diffusion models generate text in parallel but refine every position repeatedly, which can require many passes.
NCM compresses text into hierarchical levels of learned codes, then predicts each level from coarse to fine.
The presenter reports that NCM’s published 31-times speed result compares it with diffusion models, not standard autoregressive models.
In the presenter’s tests, NCM matched the reported 86% score on hard Sudoku puzzles.
NCM cannot stream readable text like a typical autoregressive model because its output is not readable until the final pass.
For one request at a time, NCM was four times faster on translation and nearly twice as fast on news summaries in the tests.
With 64 requests at once, the autoregressive model was faster; the presenter attributes this partly to NCM doing more computation across its levels.
A diffusion model using a most-confident-cell-first rule came close to NCM on Sudoku, and more passes raised its score to 95% in the presenter’s experiment.
Limitations include fixed-size output, occasional name and number substitutions, a largest released model below one billion parameters, and evaluation-only code.

Chapters

Start	Chapter	Summary
00:00	What NCM claims to do	The presenter introduces NCM’s coarse-to-fine text generation and raises questions about its speed and Sudoku results.
01:31	Why language models and diffusion models can be slow	The video explains the memory demands of token-by-token generation and the repeated refinement passes used by diffusion models.
04:05	How NCM generates text	The presenter describes NCM’s hierarchy of learned compression codes and its transformer generator, then reviews the paper’s claims.
07:18	Testing speed and Sudoku performance	Local tests compare NCM with an autoregressive model and examine how NCM solves Sudoku, including its behavior under batch requests.
11:24	Sudoku results, limitations, and conclusion	A diffusion-model experiment challenges the apparent Sudoku gap, and the presenter summarizes NCM’s limitations and overall promise.

Quotes

it sketches the whole answer roughly first and then sharpens it all at once. — 00:00

a concept is not a human idea or sentence. It is simply a learned compression code — 05:10

nothing is readable until the very last pass finishes. — 08:49

When I sent one request at a time, NCM was four times faster on translation. — 09:19

sketching the whole answer before writing the exact words is a genuine step forward for single-user speeds. — 13:57

Concepts and entities

Next Concept Modeling (NCM), diffusion model, auto-regressive model, DGX Spark, H100

Full transcript (original language)

00:00 Look at how this new language model from Stanford translates a German sentence to English. After its first pass, the English is mostly gibberish, but 11 passes later, it snaps into a clean translation. This model is called NCM, which is short for next concept modeling. And the big idea is that it doesn’t write one word at a time. Instead of writing left to right like every chatbot that you use today, it sketches the whole answer roughly first

00:30 and then sharpens it all at once. And no, it’s not a diffusion model. And because the paper claims this runs up to 31 times faster and shows a tiny NCM model solving 86% of hard Sudoku puzzles that a normal language model of the same size almost would never be able to solve, a lot of people are calling it the end of LLMs. But when I actually read through the paper, I noticed two huge questions that the authors never tested. First, every a

01:00 single speed number in the paper is measured against diffusion models. Not once do they test whether NCM is actually faster than normal auto-regressive models that generates text word to word. And second, on those hard Sudoku puzzles, I wanted to know if the 86% score really comes from the new NCM architecture or from something much simpler. So, I set up the release models on my own on my own DGX Spark, built a normal LLM of the exact same size, and

01:31 even trained my own diffusion model to test both claims head to head. By the end of this video, you will know exactly how NCM writes an answer, why it is four times faster for a single use, yet slower on a busy server, and what actually happens when you test the Sudoku claim. Let’s start with the problem NCM is trying to solve, which is why normal language models are slow in the first place. Today, every chatbot that we use today, including GPTs and Geminis of the world, uh writes its

02:03 answers token by token. Because they are auto-regressive in nature, the model writes the first token, adds it to the text, and only then it predicts the second token. That means a thousand token answer takes a thousand steps in a row. Now, the expensive part is not the math, it’s the memory. On every single one of those thousand steps, your GPU has to read the model’s entire set of weights out of memory just to produce one more word or token. For a typical local

02:33 model, writing a long answer means moving terabytes of data through your GPU. That means the speed you see on your own machine is mostly decided how fast your memory is, not by how fast your GPU can do math. If you have ever wondered why a local model writes slower than you can read, it is the main reason. So, the obvious fix is to stop writing one word at a time and write a lot of words at once. And this is what diffusion models do,

03:04 which I have covered on this channel before with models like Mercury from Inception and Diffusion Chama from Google. A diffusion model starts with a whole page of blanks, fills in a rough guess for the entire answer at once, and then goes back over the full page again and again until the text is clean. But, diffusion has a catch. Every single pass has to touch every position in the answer. And getting good quality usually takes 64 passes or more. If you cut

03:35 the number of passes to save time, the quality drops quickly because the model skips refinement steps it needs. So, diffusion basically gives you parallel writing, but you actually pay for it by redrawing the whole page dozens of times because it has to touch every single token. And that sets up the exact question this new Stanford paper is trying to answer. Can you write in parallel without paying for all of those full-page passes? To do

04:05 that, NCM borrows an idea from a image generation. Some image models first draw a tiny, blurry version of the whole picture, and then each step adds a sharper layer on top. That way, uh they never have to decide what the eyelashes look like before they know where the face is. NCM applies the same coarse-to-fine idea to text. It writes a rough sketch of the entire answer first and fills in the exact words la- later.

04:38 To make that work, it uses two pieces. The first piece is a compressor, which learns to store the passage of text as a stack of levels made of codes. The top level only has handful of codes, which hold a blurry outline of the whole passage. Every level below it gets longer and stores only the missing details. Finally, the bottom level has one code for every word. And when you add all of the levels together, the compressor can rebuild the original text

05:10 almost perfectly. Now, this is where the name Next Concept Modeling comes from. Each of these codes come from a fixed list of about 4,000 learned codes. And the paper calls each code a concept. So, a concept is not a human idea or sentence. It is simply a learned compression code, and the model predicts the next level of concepts from top to bottom instead of predicting the next word from left to right. That brings us to the second piece,

05:42 which is the generator. And the interesting thing is that the generator is just a normal transformer, which we have seen in everyday large language models. The difference is that instead of predicting the next word, it predicts every code in a level at the same time in a single pass, and then moves down to the next level. So, a summary model with nine levels take nine passes total, whether this summary is short or long. And that is why the German sentences at the start of the video took 11 passes,

06:13 because the translation model has 11 levels. Now, here’s the big difference between NCM and a diffusion model. A diffusion model redraws every word position on every single pass. NCM’s top level only have a few codes. So, its early passes are tiny and almost free. And each new level only adds the detail it is responsible for. When the authors tested this against diffusion model on a single H100, the numbers looked huge. On summarizing news articles, NCM matched

06:46 the quality of strongest diffusion model while running 31 times faster. And then came the results that got everybody excited, uh which is hard Sudoku. A normal word-by-word model or auto regressive model of the same size uh solves only 1% of the hard boards, because once it commits to a wrong digit uh early on, it can never go back and fix it. A tiny NCM model solved 86% of those exact same boards.

07:18 But, as I went through the paper, I noticed two things that most of the coverage have actually missed. First, on actual text tasks like translation and open-ended writing, the plain word-by-word model in their own tables still wins on quality and produces text that reads better. And second, every single speed number in the paper is measured against diffusion models. Not one of them is measured against a normal token-by-token auto regressive LLM. Okay, so the two questions that

07:49 actually matters for anyone running models locally were still wide open. Is NCM faster than a normal LLM of the same size on the same hardware? And why did it really win on Sudoku? Now, you might be thinking diffusion models are usually faster than auto regressive models, but does it actually hold in this case? Because NCM uses a very different mechanism. Since the authors released their code and trained model, I set them up on my DGX Spark and run three tests

08:19 to find out. Test one is how the board evolves pass by pass. And this was a great learning experience. First, I checked whether the paper’s number hold up, and they actually do, which is a great news. On hard Sudoku, I got almost the exact same 86% score. And then, I printed out the board after every pass to see how it actually solves a puzzle. NCM it does not fill in the blanks one square at a

08:49 time. It redraws the whole board on every pass. On the early passes, it even gets the starting clues wrong before snapping the whole grid into a place at the end. And that shows an immediate catch for normal LLMs. You can’t stream an NCM answer the way you would stream text from normal auto regressive models because nothing is readable until the very last pass finishes. Okay, the second test that I ran was how

09:19 does it compare to an auto regressive model that generates text token by token. So, I was looking for a model of exact size and shape as the NCM model. And then, give both of them the same prompt and ask both for the same number of words. When I sent one request at a time, NCM was four times faster on translation. On news summaries where the prompt is a long article, it was close to two times faster. So, seems like it’s actually faster than the normal auto regressive models, which

09:50 is great. But then, I sent 64 requests at once, which is what a real server handling many user would do. And the results completely flipped. The normal auto regressive model beats NCM on speed. And the reason goes right back to the memory versus math bottleneck we talked about On one request at a time, your GPU spends most of the time waiting for on memory to load the weights. So, doing only nine or 11 passes is a huge win for

10:20 NCM. On 64 requests at once, the GPU loads the weight once for the whole batch and spends all its time doing doing math. And across all the levels, NCM actually does more math because it processes about four times more positions than the words it writes. On top of that, in the release code, NCM runs the whole prompt through the model again on every single pass, which is why the long news article hurt it the most. So, in my tests with these

10:53 small models, NCM is a real speed-up for one person running a model on their own GPU, and it slows down for a busy server. Now, this is important because if you want to use these models for agentic tasks, which are going to be running multiple agents simultaneously, NCM would slow down, at least in my experiments. Now, the third test was the Sudoku mystery. Now, for this, I trained my own diffusion model with the exact same size

11:24 as NCM, on the exact same training puzzles, and tested it on the exact same hard boards. When I ran it the standard way, where it fills in cells in a random order, it it solved about a quarter of the hard boards, right in line with what the paper reported. But, I wanted to experiment a little bit more. So, when I change one rule at the test time, which is basically instead of filling cells at random, I had it fill the cell

11:55 it is most confident about first, it actually is pretty close to NCM. Now, the funny thing is, when we let the diffusion model take more passes, the score actually climbed up to 95%, right? Now, keep in mind, these are simple experiments that I did with these models. But in my experiments, this kind of means that the big Sudoku gap in the paper comes mostly from the order the cells are filled in. But there is still a very novel idea

12:25 from NCM. Now, what NCM still has it going for it is that it gets the coarse-to-fine built for free. Uh well, the diffusion model needs that extra decoding rule. So, I think we probably need to figure out a way to combine both of these techniques. Now, a few other limitations of uh NCMs that I have noticed. Uh the first one is that every NCM model writes into a fixed-size box. So, even uh one-liner

12:55 answers still runs through every level, and essentially you’re paying the full cost. The second one is names and numbers because the compressor sometimes swaps them when it turns text into code and back. And the third one is the scale. Uh the biggest model the authors train is smaller than 1 billion parameters. Uh so, we do not know yet if this holds up at the model sizes you actually run at home. Think about

13:25 uh 27 billion or even like hundreds of billions of parameters. And the last one is the released code. So, right now um it is for evaluation only. There’s no training code, and the built-in option that caches the prompt is kind of broken in my experiments. Okay, so here’s kind of the bottom line. NCM is faster than normal LM of the same size when you send one request at a time. Uh it could be actually up to four times faster. But it’s actually slower when you try to batch many requests

13:57 together. I’ll highly recommend to test the it out. It’s a really, really neat idea. But there are some obvious limitations. Because sketching the whole answer before writing the exact words is a genuine uh step forward for single-user speeds. Now, if you have been experimenting with the diffusion models or other ways of generating text in parallel, please share what you found in the comment section below. I’m putting together more hands-on tests of new model architectures running on local hardware.

14:27 So, if you’re interested in that, make sure to subscribe to the channel. And if you want to see how diffusion model actually runs on your own GPU, watch this video next. Anyways, I hope you found this video useful. Thanks for watching, and as always, see you in the next one.