Niko Strata MUDOU O JOGO da IA local com o Qwen 3.8 Flash Next

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Niko Strata MUDOU O JOGO da IA local com o Qwen 3.8 Flash Next · Channel: AI ProgBr · 15:43 · Watch on YouTube

Topics: AI agents, Model inference and optimization, AI infrastructure and hardware, Local and open-source models

Summary

The video tests Nico Strata running Qwen 2.5 Flash Next locally on two Intel Arc B770 GPUs, comparing Q4 and Q2 quantizations, memory use, generation speed, and prompt processing. The Q4 setup loads the full 262,000-token context and averages around 60 tokens per second, with faster output on code-heavy prompts and cached text processed almost instantly. The Q2 version can run on one card, freeing the other for a different model, but using both cards improves code generation and prompt-processing speeds. The presenter concludes that the best setup depends on speed, model quality, cost, and use case, and proposes a future comparison with other Qwen models.

Key points

The initial test uses Qwen 2.5 Flash Next from Unsloth with 4-bit quantization on two Intel Arc B770 cards, each with 16GB of VRAM.
The Q4 configuration loads the full 262,000-token context, using both VRAM and system RAM.
Generation averages about 60 tokens per second in the initial test; code-heavy output sometimes reaches higher speeds.
The presenter attributes potential performance limitations on Intel cards partly to software that may be less optimized than CUDA-based setups.
Reusing the prompt cache makes repeated processing of the same text appear nearly instantaneous.
The IQ2XS Q2 version can fit on one card, leaving the other available for another model or modality, but uses more system RAM.
With the Q2 model on both cards, code generation reaches 85 tokens per second in one test, while prompt processing exceeds 2,000 tokens per second.
The presenter has not yet compared the Q2 Flash Next setup directly against Qwen 3 27B and suggests that comparison as a future video.

Chapters

Start	Chapter	Summary
00:00	Setup and Q4 model configuration	The presenter introduces the local setup and explains how Qwen 2.5 Flash Next is loaded across two Arc B770 GPUs and system memory.
01:34	Q4 generation and performance tests	Initial prompts, code generation, and profiling reveal generation speeds that vary with the kind of output.
08:22	Prompt caching and single-card Q2 test	The presenter demonstrates fast cache reuse, then tests the more quantized Q2 model on one card.
11:33	Q2 prompt processing and two-card test	The video measures Q2 prompt processing, adjusts the context setup, and compares performance with both cards in use.
13:37	Trade-offs and next comparisons	The presenter weighs the Q2 results against other model options and outlines a future speed-and-quality comparison.

Quotes

we averaged 60 tokens per second, which isn’t bad at all, but it could be better. — 02:07

it starts responding instantly because it reuses the cache. — 09:25

we achieved an average of 58 tokens per second. — 11:33

we are getting over 2,000 tokens per second of prompt processing. — 14:41

Concepts and entities

Qwen3.8 Flash Next, Unsloth, Intel Arc B770, MTP, cache, Hermes, Three.js, Qwen3 27B

Full transcript (original language)

00:00 Today we’re checking it out—the most requested, most awaited, Nico Strata, running here on my setup with the two Arc B770s. This engine became very famous for running Qwen 2.5 Flash Next on basically any setup, as long as you have a graphics card with 16GB, though some are even running it with eight. And provided you have enough system memory, the computer’s memory, the RAM. Basically, it uses your graphics card just for the main experts and uses the rest of the memory for the

00:31 rest. Let’s go to the screen so we can check out everything we’ve managed to do here so far. I’ve already got everything configured and loaded, so let’s go. Which model are we running? We are running Qwen. 2.5 Flash Next from Unsloth with 4-bit quantization. It’s their smaller 4-bit model, and we managed to load it completely here. For this, I’m using the two ARC B770s. Each one of these ARC B770s has 16GB of VRAM. So the total is 32GB. We can see here that 22,321 experts are on the cards, okay?

01:03 So, it’s occupying 50GB there and the rest is in the RAM. So, we’re using about 45%of this system’s memory here . In this case, our PC has 32GB, soon it will have 64GB, but for now, we only have 32. Even so, I managed to load the full context of this model. This model has a context window of 262,000 tokens, and we managed to load it all. And the processor on this machine we’re using is an AMD with 16 cores. But let’s get to what matters; let’s talk about speed. Let’s start here with a simple

01:34 question, or maybe not. What should I look for when buying hardware to run LLMs locally? Let’s hit enter here. And up here, we’ll see the speed we’re achieving. We’re getting around 50 tokens per second on average. At the end, we’ll have a clearer view of that. We can see here, check the CPU load. Our CPU is working as well. So the CPU will also play a role in this generation because we aren’t just using the cards. In this case with Nico Strata, look, it’s finishing up the generation. We can see that our card isn’t even close to its power limit. So

02:07 , the two cards aren’t working at their maximum capacity. And it finished right here. We can see a summary right here on the monitor. And look, we averaged 60 tokens per second, which isn’t bad at all, but it could be better. Some people watching this video today can probably run it faster with less powerful hardware. And I was a little surprised, too. I believe the reason for this is as follows. When a new engine comes out, it’s usually built and tested on NVIDIA cards with CUDA, and soon after, the community

02:38 starts making adaptations for AMD, Intel, etc. Since we’re running a setup with two Intel cards, that’s a trade-off. Like I said, the cards weren’t running at their maximum power. So, there’s likely room to extract a bit more power, a bit more performance, but for that, we need to work on the software. In my last video here, for instance, I showed that I tweaked the software to run the Qwen 3.827 B on two cards in EXL format. So, it would

03:10 probably be possible to tweak it in this case as well. I’m not sure if I’ll do that, because usually when I run this model, I run it on my Mac, and on the Mac, I’ve been getting 90 to 100 tokens per second, sometimes a bit more . Now, there is one thing that might change my mind, and that’s the profile. If we look here, we got 47 tokens per second. But an important thing about profiling is that if you send just a sentence, a short text, or a paragraph —something very brief—it will likely be low. To see the real profile,

03:40 we need a larger body of text. And that’s what I’m going to provide it with now. I’ll start a new chat here, watch. I’ll tell it to summarize a paragraph and paste my text here. And here we’ll be able to see it better. I can go over to the monitor because we can see it reading the prompt right here. So it’s already reading at 900 tokens per second, which is about what the Mac does. It increased a bit, then a bit more; now it’s already higher than my Mac, at least higher than the M3 Ultra. 100 tokens per second. I haven’t been able to reach that on the

04:11 Mac M3 Ultra yet, but it’s not that far off. In this case, I think I prefer the 900 to 1000 tokens per second of the Mac, since it has a much higher token generation rate. In this case, it does 90, which would give 50%more tokens per second in generation. And in this case, I think it would be worth waiting a little longer for it to read the prompt , but it would respond much faster. But once again, in theory, with two ARC B70s I would have 1200 GB/s of

04:42 bandwidth there. Each one has around 600 GB/s, so the two together would have a ceiling of 1200. We wouldn’t get 1200, but comparing it to the Mac which has 800 GB/s, maybe we could get something close. So, it’s probably possible to extract more performance from this setup. Look, now it has started generating tokens per second, right? And probably, since the context is a bit fuller now, we’ll have a little less than 60. It will be around 50, 55. I think it’s already done. We ended up with 49. It only made a small

05:13 paragraph here. Now I’m going to ask it to do the following: create a website based on this book. Usually, these new models, when they are creating something with code that is a bit more predictable, they are also a bit faster . So here, look, we are around 40 tokens per second. Let’s see if when it starts writing the code, it will speed this up, look. Look, now that it’s in the code part, we can already see higher numbers there, sometimes hitting 65, 70 tokens per second. Look, in this part here where it’s listing, it hit 80

05:45 tokens per second, right? When things are more predictable like this, it gets it right more often. This has to do with MTP too, right? The MTP is a layer that generates several tokens at once and the larger model just accepts them or not. So when you have something that has less variation, there’s a higher chance of being accepted. You manage to accept more tokens per pass, right? So you end up increasing the speed as well . Look, here it is writing the code for us. We are now seeing an average of 70 tokens per second, more or less. So, in this case, it even hit 95 there, and then I start to change my opinion,

06:16 right? If I can get around 80, 90 for code, it hit 100 tokens per second there and I can get 100, 1300 tokens per second of profile, then maybe here we have an excellent option, even better than the Mac, right? especially if we consider the cost today. A Mac with 256 GB, I don’t know if you can even buy the M3 anymore, but to give you an idea, the M5 costs around 15,000 Canadian, right? The M3 Ultra would be

06:49 quite a lot. So if you could find it for 12, or 10,000 Canadian, it would be a steal. The two ARC B70s have gone up a lot in price, okay? But they cost around $ 4,000 Canadian. When I bought them it was much cheaper, you could get two for less than 3,000. Wow, I just noticed, I picked up a completely random book. I had no idea what I was putting here, but that’s fine. Look, in this part he’s doing, Jason, the speed goes quite high there, look. When he’s writing text, it drops a bit, look. In

07:21 this part, which is more code, code, code. It’s hitting 100 tokens per second at times. Look, 70. Now it’s back, right? Because it’s writing text again. Finished. Look, 25,000 tokens at an average of 66 tokens per second. But the thought process part, which had very little code, really dragged down that speed. If it were 100%code that we were talking about here, we would get a much better speed, okay? So, in total we had 25,000 tokens here, which changes my view a bit. So, if we can

07:51 get an average for code of 80 to 90 tokens per second, that’s an excellent result, especially considering we’re running a Q4 version, 60 tokens per second. Actually, if we were using it for non-code, for something that has nothing to do with code, then 60 tokens per second is also enough, right? It’s not that bad. So, I don’t know, now I’m in doubt, I don’t know if when I use this model I’ll use it more on this setup or more on the Mac. Another test I’m going to do is the following. I’m going to take that initial prompt we gave. I’ll create it with the entire book text, everything correct,

08:22 everything identical. I’ll create a new chat here and send it, look, exactly the same text again, because I want to see if it will reuse the cache and how long it will take to process the same text. This is very important because when we’re using a code agent, we have that initial load, right, which will take a little longer. Depending on your code agent, it puts a lot of data into the system prompt for skill calls and so on. Open code, depending on the provider’s model, can range from 7,000 to 15,000 tokens. When we talk about

08:54 cloud code, for example, cloud code sometimes injects 50,000 tokens. All of this will also serve as context for the model, for better or worse, but it takes up a good chunk of the context window. And you’ll usually feel it most during the first load, but afterward, with each message, you’ll only process that specific message. If you are reading a project file, at that moment it will only read that file. If the file isn’t that large, you won’t feel that prompt processing as much. And once it has read it, if you have to reread it, it needs to reuse the cache. And then it’s almost instantaneous. That’s what I want to see here, look.

09:25 If I send this prompt, the result here has to be very fast, look. There wasn’t even time to blink, so it was instantaneous. If we look here at the monitor, we’ll see that the prefill token here was 38 tokens per second, because basically it didn’t process anything, right? It was instantaneous there, look. I’ll do it again here so we can see, okay? I’ll paste it here and, look, no cuts at all, it starts responding instantly because it reuses the cache. Got it? But there is a way for us to improve this performance

09:56 almost like magic, meaning, by giving up a little bit of precision and quality, that is, by moving to a more quantized model, to a Q2 quantization, for example. And in my setup, Q2 will have an advantage, because I’ll be able to run it on a single card. And that is what we are going to do now. Now we are running with the 3.8 Flash Next IQ2XS. This one is a more quantized version, theoretically dumber as well, but it will have the advantage of fitting on a single card. We use more RAM, it’s actually using quite a bit more here,

10:28 right? But when we look at a 64 GB machine, perhaps this isn’t a problem. The main advantage is that now we have a free card; we could load another model onto this card. We could load an image, video, or audio model, right? A music generator, or a generator for various other modalities. What we want to see now, too, is whether we will have any speed gain or not. It’s going to be the same thing. I’ll start here with a simple prompt. Tell me a story. And here we can already see that we are at a rate of about 40 tokens per second . So 43 tokens per second. Next test,

11:02 create a snake game with Three.js. Here we’ll see if the speed gets a little better with code. I think asking for Three.js was a mistake here because it’s thinking too much. Just do the code, man. I’ll do the following: I’ll stop, start again, and just ask for the snake game. I told it, hey, make a snake game, don’t think, just write the code. And there it’s writing the code for us now. It’s getting a good rate here, right? Let’s see the average at the end, look. With little thinking and a lot of code, we achieved an average of 58 tokens per second. The next step here will be to test the summary of our

11:33 book. Summarize a paragraph, and here we will monitor it, and we are getting around 1200 tokens per second, look. So , up to 50,000 tokens, it’s roughly 1200 tokens per second. And we see that with each pass here, it slows down a little bit. It will finish around 1000 to 1100 tokens per second. Look here, it’s finishing processing, at a rate of about 1080 tokens per second, 1073 tokens per second to process 111,000 tokens. An important thing here is the

12:05 following. To be able to load the full context of 262,000 tokens, we are loading fewer experts in the VRAM. So, even if it’s a smaller amount, we end up loading fewer experts here, and this has a performance cost. There’s nothing in my chat here and the model started responding. I think it’s my Hermes agent asking something. A message arrived here from Renato, and he is exactly configured to use this model. Renato is my Hermes agent, okay? For those who don’t know. Look, I tried to

12:35 decrease our context window here, okay? We didn’t get much more VRAM space, it’s almost at the max, 24 GB, it lowered the RAM consumption a little bit, but we didn’t get a big speed gain either. So, for us to get any speed gain, we’ll have to use both cards. Let’s test running both cards, the Q2 version. Look, now we have the Q2 version here with 262,000 tokens loaded on both cards. And we can see that the two cards are no longer near the limit, okay? There is plenty of space here. We

13:06 are also using much less memory and have 24,000 experts on the VR. Let’s check the speed here. Basically, the speed is almost the same, 54 tokens per second. Let me try with code here. I’ll ask it not to overthink and go straight to HTML. It went straight to HTML. Now, yes, we are getting excellent instantaneous speed here. Let’s see at the end. There was a problem here because it stopped generating out of nowhere, but it generated 6,000 tokens at 85 tokens per second. So here we had a real gain in speed. The problem is that we are using

13:37 two cards here with a Q2 version. And so we need to consider several things. If you saw the other video I made about this setup, we can run a Qwen 3.8 27B Q4 on one card or the FP8 version on two cards. So, if I’m going to use two cards, is it better for me to run a Qwen 3.8 FP8 or is it better to run a Qwen 3.8 Flash Next Q2? I don’t know, I haven’t done that test yet, but in principle I think I would go with the 27B. Let me ask it to continue here, look. Does it think it finished?

14:09 Actually, it had almost finished, right ? This version isn’t very good, though. It just needed to close the, see? This one here is a Q2 version. It just needed to close the script there. But well, theoretically it works, right? At least the clicks and such. Let’s see if we have any prompt processing improvement here. So, we are achieving here, look, a good improvement in prompt processing. We have a big advantage here; using both cards with the Q2 version, we are getting over

14:41 2,000 tokens per second of prompt processing. The card is almost at its limit here, look. especially the first card here, but this speed here is incredible, okay? Awesome. It finished here with a speed of 55 tokens per second. On the side is a summary of what we saw here today. We started by looking at the Q4 version, which needs two cards to run. Then we saw the Q2 version running on just one card. Then the Q2 version running on two cards. And probably a month from now these

15:11 numbers will all be outdated, because as I said, it’s a matter of the field evolving here. The big question is whether it’s worth it or not. Actually, this gives me an idea for a new video: Qwen 3 8B Flash versus Qwen 3 27B. We’ll run Qwen 3 8B Flash in Q2 on one card, Q4 on two cards, and Qwen 3 27B we run in Q4 on one card, and FP8 on two cards. Then we can check the speed and the quality. And it’s good because I’ll also decide how I’m going to use

15:43 this setup of mine. But that’s it. If you liked this video, leave that hype, it helps a lot. If you want to see the other videos of this setup, I’ll leave two links right here. Now, if you want to take a look at my course, just click right here. Alright? That’s it. I’ll leave it here, see you in the next video.