Which is The Best Qwen3.8-27B?

Author: Sam Witteveen · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Which is The Best Qwen3.8-27B? · Channel: Sam Witteveen · 21:00 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, AI models, Local and open-source models

Summary

The video compares three Qwen 3.8-27B fine-tunes—Thinking Cap, Swift 1.5, and QwenPi—that aim to use fewer reasoning tokens while maintaining accuracy. It explains how each model was developed and the trade-offs in their goals, with QwenPi focused on the Pi coding-agent harness and the other two targeting broader reasoning efficiency. In playback tests, all three often finish faster than the base model, but their relative speed and quality vary by task. The host concludes that Thinking Cap and Swift are more general-purpose, while QwenPi is especially suited to coding tasks.

Key points

Longer reasoning chains can increase both inference time and cost; reducing tokens can also improve generation speed.
Qwen 3.8-27B offers low, medium, and x-high reasoning effort, with x-high as the default described in the video.
Thinking Cap aims to shorten x-high reasoning conservatively; its reported average is 37% fewer tokens with accuracy decreasing from 86.6 to about 85.8 across 12 benchmarks.
Swift 1.5 targets up to 58.5% fewer thinking tokens and emphasizes long-horizon agentic and coding tasks; its training data is also published on Hugging Face.
QwenPi is fine-tuned for the Pi coding-agent harness using successful Pi sessions and reinforcement learning focused on reasoning efficiency at low and medium effort.
The host’s demos show that no model wins every prompt: relative speed and outputs differ across reasoning, coding, math, logic, and SVG tasks.
QwenPi’s evidence is specific to use inside Pi, and the video notes that its results have not been shown to transfer to other harnesses.
The host characterizes Thinking Cap and Swift as more generalist models, while QwenPi appears more suited to coding.

Chapters

Start	Chapter	Summary
00:00	Why reduce reasoning tokens?	The host introduces the trade-off between lengthy reasoning chains and faster, less costly answers.
02:12	Qwen reasoning effort and its costs	The video explains Qwen 3.8-27B’s reasoning-effort settings and how shorter traces can affect speed and accuracy.
04:22	Thinking Cap	The host describes Thinking Cap’s background, conservative training objective, reported token savings, and intended use.
09:12	Swift 1.5	The video covers Swift’s approach to reducing overthinking and its emphasis on coding and agentic tasks.
11:19	QwenPi and the Pi harness	The host explains QwenPi’s training for the Pi coding agent and notes the limits of evidence beyond that harness.
15:01	Comparative playback tests	The host runs the models on several prompts and shows that speed and results vary by task.
19:20	Takeaways and model fit	The host summarizes the variation between fine-tunes and recommends viewing QwenPi as coding-focused and the others as more generalist.

Quotes

one model flies there to get the answer as fast as possible, and the other one drives around the coast to take a scenic route. — 00:00

The target was one thing, how many thinking tokens it takes to reach the same answer — 06:30

Qwen Pi is Qwen 3.8/27B fine-tuned specifically to work inside of the Pi harness. — 11:50

you probably should try each of them out for your specific use case. — 19:20

Concepts and entities

Qwen3.8 27B, Thinking Cap, Swift 1.5, QwenPi, Pi, BottleCapAI, UkisAI, Tomáš Mikolov, vLLM, Terminal-Bench

Full transcript (original language)

00:00 Okay, so this is the same question going into two versions of the same model. same weights underneath, similar answer at the end, But one of them has used a lot more thinking tokens to get there, And the other has used hardly any in comparison to get there. So on my machine, that’s a difference between eighteen seconds and ninety seconds. And if you think about it, that’s sort of the problem with reasoning models now, right? They get the answer, but they take a long time to get there. it’s almost like one model flies there to get the answer as fast as possible, and the other one drives around the coast to take a scenic route.

00:34 and you’ll see this in ways that are often not totally bad. They’ll do things like they restate the question, they check, they check again. They say to themselves, “Wait, let’s reconsider.” And every one of those tokens is time that you’re spent waiting for this answer and of course, money that you’re paying for the inference for this now in the past, I’ve talked about OpenAI focusing on reducing the number of tokens between the 5.0 model and the 5.5 model. well, they’re not the only ones that have been focused on this, right?

01:06 Most of the major labs have been looking at how do they take these long chains of thought and reduce them but still keep the accuracy. And the cool thing is that there are also teams trying to do this with open source models. So in the past, I covered the Thinking Cap model for the Qwen 3.6-27B. And no surprise, I’m gonna cover their new model in this video. But also, I want to look at some of the other teams that have taken on the Qwen 3.8-27B and fine-tuned it to think less. so on top of Thinking Cap, we’ve also got QwenPi, which is sort of

01:39 like a specialized version with less thinking for coding agents. Got Swift 1.5. They all claim fewer tokens for about the same accuracy, but it seems they all go about it in really different ways. And it also seems that They’re probably really good at different things. So this video, I want to look at how each one was made, then I’ll run all of them against the base model and see if the claims actually hold up. All right, so before we do the fine-tunes and look at those, you need to know a bit about the sort of thinking and reasoning on Qwen 3.8

02:12 that it already gives you out of the box, and they call it reasoning effort. It has three settings, low, medium, and x-high. and amazingly out of the box, x-high is actually the default. So if you’re serving with vLLM, you actually set this in the chat template and you can see that when we set that reasoning effort, that’s going to affect how long the chains of thought are that’s coming back. Now, I talked about this when I made the video about serving Qwen 3.8 27B and I got to say that model is still one of the best open models, that you can get and serve

02:44 on reasonably small hardware for some uses the 3.8 Next is probably better, But that’s gonna move you into the realm of needing a bigger GPU or needing multiple units in there Now back to the 3.8.27B. The whole reasoning effort there is you are basically telling the model to try less hard So yes, you can get less tokens out of it, but when you actually turn that reasoning down it’s often gonna cost you in accuracy and if you’re running this locally on one GPU, you’re really going to be decode bound.

03:15 So the thinking or reasoning tokens there match pretty much directly to seconds for you getting your final result out. if you can have those tokens, you’re roughly halving the weight that you’re going to have to wait for your final answer out secondly, these models have multi-token prediction heads for speculative decoding, and that actually works better on shorter traces than longer traces and we actually see this in some of the reported results. So Thinking Cap’s model card actually reports that speculative decoding goes from about 2.6 tokens per step at X high to over 3

03:50 tokens per step at medium and low. So you can actually think about this as thinking less doesn’t just mean fewer tokens. Each token comes out faster as well. Alright. So the big question here is what are these fine-tunes actually trying to do, and are they going to get you the savings without any sort of accuracy hit here? Alright, so here’s the family tree. Everything starts from Qwen 3.8 27B. On one branch of the tree, you’ve got Thinking Cap from BottleCapAI. Their first Thinking Cap model was the one that I covered a couple of months back And

04:22 that was built on the Qwen 3.6 model then. And you could say related to this was the UkisAI Swift model. So when they built their first version of Swift, part of that recipe was a transfer component from the ThinkingCap model. So you can sort of think of Swift of having some of the ThinkingCap DNA in it and being related, now, on a completely separate branch here, you’ve got Qwen Pi, which is playing a different game totally. I’ll get to that as we go through each of the models. But let’s start with ThinkingCap. so Bottlecap AI is a lab in Prague, and One of the co-founders

04:55 there is Tomáš Mikolov. so at the time that I recorded the original video about the first Thinking Cap, I didn’t actually know that until one of you pointed it out in the comments. And so for those of you who don’t know who Tomáš Mikolov is, he was actually the first author on a very famous paper Word2vec. So this is one of the key papers that really changed, natural language processing. The whole idea of having vectors that could represent words and then could be used for things like classification, like a whole bunch of these things. at the time Mikolov was working at Google, from memory, he was also

05:28 working on a lot of the ideas for Seq2Seq, which all of these were very early precursors to the transformer idea of sequence in, sequence out. now after one of you had pointed that out, I actually reached out to him to send the video and message him were they gonna do one for 3.8? And I found out they’re working on an interesting project that hopefully I can cover in the near future But clearly I wasn’t the only one giving feedback that the community wanted a 3.8 version of the Thinking Cap models. So I think they reallocated some of the compute that they were using

05:58 for their other project to this And just a very quick side note here, I would say that We really should be grateful for startups like this. his startup is not some billion-dollar startup in San Francisco that’s training their own frontier models to compete with OpenAI or Anthropic, et cetera Bottle Cap is a startup that’s raised a tiny fraction of that money compared to some of these bigger ones and most likely has a fraction of the compute that those more famous startups actually have. So the fact that they’re allocating compute to do something like this for the community, I think is very cool.

06:30 All right, so how was this actually made? Well, BottleCap didn’t publish the actual training recipe, but they’re clear about the objective here. The target was one thing, how many thinking tokens it takes to reach the same answer and they mentioned that the aim was deliberately conservative here. They didn’t try to make the model smarter, to teach it anything new, to change how it talks. They haven’t tried to add any knowledge in here or even improve the instruction following or safety or any of those things. Those are meant to come through totally untouched So it’s trained at X-high, and

07:01 this time they put more focus on much harder sort of benchmarks and tasks. So things like math, reasoning, long context, agentic tasks, et cetera you can kind of think of this is teaching the model when it’s done rather than teaching it to be brief. So the headline here is 37% fewer tokens on average across 12 benchmarks With the average accuracy going from 86.6 to sort of 85.8. So it’s just a tiny percentage less accuracy there. So for example, on the GPQ8 Diamond, This goes from an average of 12,800

07:36 thinking tokens on the original version down to 7,300 on the Thinking Cap one. And interesting on some of them, like The long context result is a really interesting one. On their long context retrieval benchmark, it uses 39% less thinking, but the accuracy actually goes up a couple of points there But the chart I think that is most useful, in their blog post about this is this one because it ties Straight back to where we set the reasoning effort. If you take the base model and just drop it to medium, you remove about

08:07 half the thinking, but you give up about nine points of accuracy. Thinking Cap at X high removes 37% for under a point of accuracy So suddenly that x high level gets really useful there. All right, so what’s this model actually good for? it’s basically just a drop-in replacement for the Qwen version at x-high. So anything that needs general reasoning, knowledge questions, long documents, et cetera all of those things you can basically just drop this model in. Perhaps the only downside I would say here would be the agentic traces.

08:40 and not because they’re not shorter. They’re shorter, But it seems that the reasoning only drops by about 11% there now interestingly, the other thing here is that this version is made for X-high. if you want a version that’s made for the medium or low thinking and optimize for that that’s something that they’re actually selling to enterprises and along with that, the other change here is the actual license. So it’s gone from being Apache 2 for the 3.6 version to this one being a small business plus personal use kind of license Which totally makes sense if they’re

09:12 trying to make some money to actually cover their compute costs, et cetera all right, so next up is Swift 1.5 from Ukis AI, and this one’s quite a lot more aggressive. So this one is going for up to 58.5% fewer thinking tokens on the Qwen 3.8/27B. They built it in two steps. So for Swift 1.0, they were really looking for specific tokens in Qwen’s reasoning that seemed to trigger overthinking And you could think of some examples

09:44 of those are where things like it says, “Wait, let me ask, what the user wants.” “Hmm.” And then, you know, asks itself something. “Let me reconsider.” And then they fine-tuned the model to penalize those tokens rather than just penalizing the length. And then they added that to the original version of the ThinkingCap model. For 1.5, though, they took the model and scaled up the reinforcement learning and on-policy distillation. With their main focus being on long horizon agentic and coding tasks. Now the cool thing here is that not only have they published the models, the

10:16 training data is also on Hugging Face which is really good to see here You can see it in the coding numbers here. On their model card, LiveCodeBench goes from just under 77% for the base model with around eleven thousand two hundred tokens out to actually an increase of accuracy of eighty one point seven percent with only eight thousand four hundred tokens out. So the other interesting thing here is that they also claim that the savings that they’re getting hold at every effort level. At low, it actually scores slightly above the base model with about 29%

10:48 fewer thinking tokens practically too, this one has probably the most ways for you to run it. there are standard LlamaCPP, GGUFs for this. there are small ones to fit on small GPUs. There’s MLX there’s an NVFP4 version, and even an AMD build here and they even have like a free research API for this with no API key if you wanna just try this out Now that headline of the 58% fewer thinking tokens probably turns out to be a bit less when you start looking at the mean thinking reduction across

11:19 this, but it’s still very impressive here. Another thing that they’ve focused on is that they claim that the base model loops and fails early Which they’re claiming that their version doesn’t do. now the license for this one is their own custom license, which is free up to a million dollars in revenue. After that, you need to basically contact them. Alright. So that brings me to the third one, which is kind of the odd one out here And incidentally, this is the only one of the three that actually has an Apache 2 license. So this is QwenPi. To understand it, you first need to know about what Pi is.

11:50 Pi is an open source coding agent that’s very deliberately minimal It has a very limited set of tools for doing read, write, edit, and bash and last time I checked, it even has a system prompt that’s under 1,000 tokens. so Qwen Pi is Qwen 3.8/27B fine-tuned specifically to work inside of the Pi harness. And I wanted to cover this one because I think this is something we’re going to see a lot more of where people fine-tune a particular model for a very specific harness And it totally makes sense.

12:22 If you can get the model and the harness to work together And the model can be primed for what kind of inputs it’s gonna see, what kind of inputs it should generate out You’re probably gonna end up with a model that’s a lot more suited to a very particular vertical harness there And this idea is something that I’ve been running tests on recently with a number of vertical harnesses in the co-scientist area So I feel like this model shows a little bit of where that direction is actually going. So how was this made? first, it’s basically supervised fine-tuning on real Pi sessions,

12:52 but only the successful ones. So the model learns the whole workflows, how to look at code, how to make a change, how to run tests, how to fix what’s broken, et cetera. Secondly, they make sure in those tasks that they actually check for working code versus broken code separately themselves. Thirdly, they’re doing reinforcement learning with GRPO and a custom reward for reasoning efficiency there. So obviously the goal there is to basically reduce the long chains of thought. Interestingly, that reward is aimed at the low and medium settings while X-high

13:23 is left to focus on getting it right the other interesting thing that’s mentioned in here is that they actually pick the checkpoints based on actual agent results, not just based on training loss. So you could imagine that as the model gets better and the RL kicks in to reduce that training loss, that doesn’t always lead to the best agent runs. So they’re much more focused on getting this to actually work for agent runs than purely just for some LLM training goal now my guess here is a big part of why this works is that Pi is a minimal harness

13:56 When it’s only got a few tools and a really tiny prompt like that, every session looks reasonably similar. and you can imagine that that generates really nice data to actually fine-tune on So the idea with this one is different. ThinkingCap and Swift are trying to make your X high shorter. Pi is actually trying to make the low and medium effort settings actually good On their numbers, Pi’s medium setting matches the base model’s x high on Terminal Bench with about 41% fewer output tokens. they’re trying to reduce the number of tokens you’re using by actually

14:28 getting you to use the lower and the medium levels by increasing their intelligence up to the high level. Now, obviously the weakness for this one is that it’s a very narrow use case All the evidence comes from using it inside of Pi. From what I’ve seen, nobody has really shown that this carries over to other harnesses if you wanted to use open code or you wanted to use other things, et cetera. and on the general sort of reasoning tasks, like the GPQA, the base model Still actually wins at the medium level. so in many ways I’d say this is sort of like a community testing effort that

15:01 perhaps isn’t totally proven yet, but is certainly very promising here so let’s jump in and actually take a look at some of the outputs that we’ve got here Okay, so Each of these models is running on a Dell Pro Max tower here, and they’re running on an RTX Pro 6000 with standard vLLM here. I haven’t really optimized any of that in here. So thank you to Dell for sponsoring the compute for this It really makes it possible for me to be able to benchmark these models out Okay, so let me set up the demo here. What I’ve basically done is I’ve gotten Claude, Fable, and Opus to

15:34 generate a bunch of questions or prompts that, they’re gonna create a large amount of thinking tokens or reasoning tokens, et cetera. and then what I’ve done is I’ve gone through and actually rendered out the inference for each of these models and saved it with exact timestamps so that we can look at all four of them playing back at once here. So you can see here we’ve got the, original Qwen 3.8-27B. This is technically the fine-tuned model that they released of their

16:05 base model, but this is sort of the standard, version of the model. I’ve got the Qwen Pi one, I’ve got the Thinking Cap one And I’ve got the Swift one in here. So you’ll see that if I come up and I pick something like this, I’m actually playing back at 16 times the speed, that it was, recorded. And you can see, sure enough, here, this is now, Thinking Cap has done, it’s completed. it’s got the correct answer. we’ve now got Swift 1.5 done, completed with, again, with the correct answer.

16:38 We’ve got the Qwen, Pi model done with the correct answer. And finally, at some point, we will get the original 3.8 27B looking at this one also with the correct answer there So in this case, each of the models has given us the exact same, answer back. But we can see very clearly in this case that ThinkingCap was way shorter, right? With only, four thousand tokens. You can see that even the inference speeds are roughly the same, second is, the

17:08 Swift, model, and third is the Pi model, and then finally is the long version. So all three of them are doing much better than the original version. Let’s try it with a different prompt Okay, so again, you can see, these are all off. and we can see in this case, the Swift model actually, was fastest, followed by the Pi Thinking, followed by the original model, And in this case, Thinking Cap actually came last, right? Now, I will point out that this one is more of a sort of coding task, with this.

17:43 And remember that the Pi model is made for coding tasks and tends to specialize in that kind of thing Alright, If we look at another one, like this, we can see that sure enough, in this case, the Swift model did really well, followed by Thinking Cap. The Pi model took longer than both of those, but still was way shorter than the original one Alright. If we give it sort of a logic puzzle to go through, you see in this case the Thinking Cap 1, Swift came second, followed by The base model and then Pi

18:14 If we give it another mathematics one, we can see in this one that Thinking Cap won. Swift came second, Pi came third, and then the original model was quite a long way behind that And you see if we translate these into time, that we’re getting huge savings with some of these models. The ThinkingCap 1 is just a lot faster, a lot more concise in what it’s actually producing there Another example, again, ThinkingCap first, followed by Swift. and in this case, the base version actually beat the Pi version So

18:45 just to finish up, I’ll show you one that, is kind of a bit surprising. in this one, you’ve got, the Swift model actually doing slightly better. we’ve then got the base model coming second, followed by Thinking Cap, and finally the Pi model in here. For me, the interesting thing here is that even though we’ve got the same prompt, for making the SVG, each of them is making a slightly different SVG, you know, coming out of it And I would say this is sort of the perfect example of where you probably should try each of them out for your specific use case.

19:20 because just as the SVGs change, in quality in here, you will find that other answers that don’t necessarily always have an exact answer also can change quite a lot from model to model in here While in my test Thinking Cap seems to be winning most of the time, the variance can be really different on different kinds of questions. And I think that’s got a lot to do with just how people are doing the fine-tunings here whatever they make the model focus on and use the RL on, it’s going to get good at that.

19:52 and I think For each of these three fine-tunes here, they’ve gone for slightly different, data curation and perhaps curriculum learning in the way that they’ve actually put these together. So if you remember back, I also talked about that the Qwen Pi model was actually trained to do better on medium. And the thing I find is that sure enough, this model actually is a lot faster out on medium than the original base model. it doesn’t mean though that it’s always gonna be faster for all of them. often the Thinking Cap 1 will still be faster, for some of these.

20:25 And I find that even the Thinking Cap 1 tends to be a little bit more accurate at a lot of the different, tasks and stuff when you have it actually set to medium in this particular use case here, you can see, though , when we compare this, that the, Pi model, is actually being faster than both of them when we’ve got the setting on medium there so anyway, let me know in the comments if you’ve actually had a play with these, how you found them out, which one of the three is your favorite. I would say that all three of these are actually very good. In general, they all tend to do better than the base model for

21:00 a lot of the particular tasks. Now, The Pi 1 seems to me to be definitely more suited to coding kind of tasks, and it really shines at those The Thinking Cap and the Swift one are much more sort of generalist models So anyway, as always, if you like the video, click like and subscribe. and I will talk to you in the next video. Bye for now.