Nanbeige4.3 33B: Great Benchmarks, But It Failed My Tests Royally

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Nanbeige4.3 33B: Great Benchmarks, But It Failed My Tests Royally · Channel: Fahd Mirza · 08:05 · Watch on YouTube

Topics: AI evaluation and benchmarks, Software engineering with AI, Model inference and optimization, AI models

Summary

Fahd Mirza tests Nanbeige 4.3, a 33-billion-parameter model advertised as Claude-level, by asking it to fix a food-costing bug in a restaurant application. Despite spending more than an hour, the model fails to identify the bug and instead reports numerous issues that were not present. It also struggles for over 45 minutes to generate a simple animated HTML scene, repeatedly thinking without completing the task. The reviewer concludes that the model is not production-ready and does not live up to the Claude-level claim.

Key points

Nanbeige 4.3 is a 33-billion-parameter model from Nanbeige LLM Lab, part of Boss Zhaopin.
The model is presented in Hermes Agent as a Claude-level model, which the reviewer sets out to test.
The first task is to fix a restaurant application’s food-costing bug involving meat yield calculations.
The dashboard incorrectly shows low food costs because it calculates less meat to buy than the amount served.
The model runs locally with vLLM and uses just under 88 GB of VRAM in the demonstrated setup.
The reviewer shows benchmark charts in which Nanbeige 4.3 appears to outperform similarly sized rivals, while noting some exceptions in occupational tests.
After more than an hour, the model has not found the bug and has reported 14 or 15 issues that were not present.
For a second test, it fails to complete a simple single-file HTML animation after more than 45 minutes.
The reviewer calls the model unusable in its current form and says it is not production-ready or Claude-level.

Chapters

Start	Chapter	Summary
00:01	Introducing Nanbeige 4.3 and the restaurant bug	The reviewer introduces the model and sets up a real-world test: fixing a food-costing error in a restaurant application.
02:46	Local setup and benchmark claims	He explains the local vLLM setup and VRAM use, then reviews benchmark results presented by the model’s lab.
04:55	The model fails to find the costing bug	After lengthy reasoning, Nanbeige reports multiple nonexistent bugs but does not identify the application’s actual issue.
06:29	A second test: generating an HTML animation	The model also struggles to produce a simple animated HTML file, leading the reviewer to call it a failure.
07:34	Final verdict	The reviewer says the model is not production-ready or Claude-level and invites viewers to share their thoughts.

Quotes

It says that it is a Claude-level model — 00:35

the system thinks we buy less meat than we serve — 02:15

it couldn’t really find the bug — 05:57

This is not production ready, let alone cloud level model — 07:34

Concepts and entities

Nanbeige4.3, Nanbeige LLM Lab, Claude, Hermes Agent, Wildfire, vLLM, Mass Compute, Gemini 4, Muse Glimmer, Gemini 3.8, Fahd Mirza

Full transcript (original language)

00:01 This is Nan Beach 4.3 in 33 billion parameters. This is from Nan Beach LLM Lab, the AI team inside Boss Zhaopin, which is a big Chinese hiring platform. We have been covering their models for quite some time from day zero, as you can see. Primarily, their claim was always that they can beat larger models with their smaller models. But this time, their claim has even got bigger. Let me quickly show you what they are

00:35 saying in Hermes Agent, as I already have loaded it. It says that it is a Claude-level model, and that is exactly I want to test out how close this model is to Claude, because we have seen that claim a lot. So, what I’m going to do, I’m going to give it a real-world application. So, this application is Wildfire, a Surrey Hills restaurant serving crocodile, kangaroo, emu, venison, camel, wallaby, rabbit, and much more

01:07 exotic meats. Its kitchen OS costs every plate from recipes and supplier prices, reconciles stock against delivered, and streams live order from the floor tablets. Exotic meat loses a lot of weight when it is trimmed, boned, and skinned. Crocodile tail keeps only 58% for example. Taste really good, by the way. Wallaby shank 46% and rabbit 44%. So, the kitchen must buy more meat than

01:39 it serves, and the costing engine is supposed to handle that. Now, look at this dashboard. Food cost is 18.9% labeled excellent, way under the 32% target. Margin Watch shows zero dishes over target. Every bar is green. but there is a huge bug. So, you see, for example, if you look at this wallaby shank. So, let me click on this wallaby shank here. Now, if you look at this chart, 220 g on the plate, 46% yield, yet only

02:15 101 g to buy. So, the system thinks we buy less meat than we serve, which makes every food cost look far too low, and this restaurant is going to go down just because of this. That is what I’m going to do. I’m going to ask this model to go and fix this bug, and then tell me after testing thoroughly. This is Farhat Mirza, and I welcome you to the channel. Now, first up, let me quickly take you back and show you how exactly I got it

02:46 installed. So, this is vLLM, as you can see, serving the model locally with this command. And at the moment, it is consuming just under 88 gig of VRAM. The reason why it is so high is because that it is using this context length. You can increase it uh sorry, decrease it to get this lower, but I don’t think so that in full precision, you would still need around 60 to 62 gig of VRAM. Also, if you’re looking to rent a GPU on very good price, you can find the link

03:18 to Mass Compute in video’s description. And if you don’t know how to install vLLM and prerequisites, just search on my channel. And don’t worry, I’m not going to go away without telling you about this greatest of all time newsletter at farhatmirza.substack.com, which you must subscribe today, please. Okay, while the model works through it, let’s talk a bit more around this model. And by the way, there is also a base model, and this one is fine-tuned from that base model. So, look at this image

03:52 first. The lab has put its model against three rivals of similar size, Gemini 4, Muse Glimmer, and Gemini 3.8 in 27 billion. And all eight tests shown that teal bar is the tallest. The gaps are huge on the hard coding and terminal tasks where some rivals barely register. The only close races to the phone where Gemini 3.8 is just behind. So, you know, look at this and this really shows that this model is way, way better than the

04:25 rivals. Now, this one tests real job, 35 occupations from IT and engineering to finance, healthcare, and law. Overall, Nan beats scores well above Gemini 3.8 and far above Muse Glimmer. It wins big in jobs like web admin and lawyers, roughly doubling Gemini’s score. It is not perfect, though. Gemini still wins in a few like healthcare managers and personal financial advisers. But the gap is not that far. Anyway,

04:55 let’s go back and check what is happening. Sorry, here. Let’s wait for it to finish. So, it’s been around 20 minutes and I just wanted to show you this output as the model still works through our first test. You can see that it has thought a lot, a lot when I say. Now, what it it is doing, it keeps just sort of compacting the context because it is thinking a lot and then it moves forward a bit, then thinks a lot, then moves

05:26 forward a bit. Now, even Claude’s Haiku model is way better than this. And I haven’t given this test to Claude Haiku. I might do that if it takes more time. But so far, there is no way it is a Claude-level model what I am looking at even in this thinking, which is not really good enough for me. Going is way more better. Even [snorts] Music Glimmer is quite good when it comes to thinking. So, yes, the task is hard, but then

05:57 again, it’s a 33 billion dense model, so we expect better. Okay, so it’s been more than an hour, and it couldn’t really find the bug. So, what it has done so far, it has found 14 or 15 different bugs, whereas the application had only one bug, which it couldn’t ever find. Just going in circle, thinking, you know, very wildly. So, meanwhile, even the crocodile meat has cooked fully. So, I’m just going to cancel that, and this test test is a

06:29 fail. So, let’s now try out code generation. Okay, I needed to show you this. So, for the next test, what I was trying to do is to ask it to write the single HTML file, which is going to simulate a string of Mexican banners for, you know, they’re hanging across a plaza, some bright colors, and some doves and stuff. And they should just sway. So, it is not that hard a simulation at all. And this is what it did. So, it has a run

07:02 I’ll just go and show more more than 45 minutes now. Now, instead of completing it, what it did, it just hit it max output tokens and the whole context length of and it has thought a lot, as you can see, just going in circle. Uh really really struggling to create a simple HTML file, even the cascading style sheet, all the JavaScript function, really struggling a lot. So, I don’t think so this model is

07:34 really usable um at all in this shape and form. I really tried my best. I I it’s been around 3 hours almost for me to just getting this thing to work. This is not production ready, let alone cloud level model, and I’m just showing you as is. So, it is going to start again and then think like crazy for another hour, which I don’t really have time for. We have lot of other models to test and go for. So, this one is a big no-no. Let me know your thoughts in the

08:05 comments, too. And again, thank you for all the support.