LFM2.5 24B-A2B: Did Liquid AI Manage to Beat Qwen? Tested Locally

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: LFM2.5 24B-A2B: Did Liquid AI Manage to Beat Qwen? Tested Locally · Channel: Fahd Mirza · 11:19 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, Local and open-source models, Health and science, AI safety

Summary

Fahd Mirza tests Liquid AI’s LFM2.5 24B-A2B model locally through Hermes Agent on several practical tasks. It successfully finds and fixes an impossible yield bug in a multi-service milk-factory dashboard, but fails to create a convincing animated tattoo simulation. Its multilingual output is mixed, and its submarine-design reasoning contains significant calculation and interpretation errors. The model refuses a hypothetical request to plan a secret wedding ceremony, leading the presenter to describe its guardrails as working.

Key points

The model is run locally with llama.cpp and Hermes Agent, using just under 28 GB of VRAM in the shown setup.
LFM2.5 24B-A2B is described as a 24-billion-parameter model that uses about 2 billion parameters per token.
On the milk-factory dashboard task, the model fixes an impossible yield reading without the presenter providing hints.
The tattoo task asks for a complex animated simulation in a single HTML file, but the result does not look or behave as requested.
The presenter notes that Liquid’s published benchmarks show strengths in agent and tool use, while cautioning that they are company results.
The model produces answers for 81 language communities, but some words are mixed up and the presenter is uncertain about several entries.
In the submarine problem, the model’s hull-mass and buoyancy calculations are off, and it reports an implausible neutral depth.
The model declines to provide steps for the hypothetical secret wedding ceremony prompt.

Chapters

Start	Chapter	Summary
00:01	Local agent test and dashboard bug	The presenter introduces the milk-factory app, explains its impossible yield reading, and sets LFM2.5 to find and fix the bug.
04:37	Tattoo simulation and benchmark context	The model is asked to create an animated tattoo simulation, while the presenter discusses its intended agent strengths and then shows an unsuccessful result.
07:12	Testing multilingual ability	The presenter reviews the model’s food-name answers across 81 language communities and finds the results mixed.
09:14	Scientific reasoning and guardrails	The submarine-design answer contains notable errors, after which the model refuses a hypothetical request to plan a secret wedding ceremony.
10:47	Closing remarks	The presenter invites viewers to comment and thanks subscribers and channel members for their support.

Quotes

You cannot pour out more milk than you poured in. — 00:32

it has avoided the traps, no clamping, and no seed edits — 03:36

I think this does not really look anything like a tattoo or something. — 07:12

multilinguality doesn’t seem to be a forte of this model at all. — 08:44

So the guardrails are up. Model is quite ethical. — 10:47

Concepts and entities

LFM2.5 24B-A2B, Liquid AI, llama.cpp, Hermes Agent, Dairy Flow, H100, Mass Compute, AutomationBench, SWE-bench Verified, GPQA Diamond

Full transcript (original language)

00:01 This is Dairy Flow, a live milk factory dashboard. Milk comes in from suppliers, gets turned it into whole milk, skim, cream, and butter, and ships out to distributors. A database, an API, and a front end running all on my local system with Docker. So, this is a real-world multi-service shipment of an application. Now, look at the number every plant manager watches, yield, which is how

00:32 much product you get out of the milk you put in. It says 106%, which is physically impossible. You cannot pour out more milk than you poured in. So, this dashboard is lying, and somewhere in this app is one bug that nobody can see, and it can really ruin the business. So, what I’m going to do here, I’m going to use this new Liquid AI’s LFM 2.5 24 billion A2B model to see if it can find the bug, and then

01:02 if it can fix it. I am already running it with llama.cpp, as you can see on my screen, and this is my VRAM consumption at the moment, just under 28 gig. You can even reduce that VRAM consumption. I already have it integrated with Hermes Agent, where I’m going to go and give it this goal without any hand-holding, without giving it any hints. While it works, let’s talk a bit more around this model. Now, here is the thing. I have been

01:32 covering Liquid AI’s models, you know, for more than 2 years now. I have covered each and every model from them. But, to be honest, I think still I believe they are lackluster. Um they are trying their best, but really they are not breaking that bar, you know, far from Frontier. You know, they are trying to mimic what Glenn has been doing, but really not getting there. Now, this size, 24 billion in A2B, is quite a rare one. So, I’m quite curious to see if

02:04 they manage to get it right this time. So, this model, as you can see here, a 24 billion parameter reasoning model that only uses about 2 billion of them per token, and they have built it for fast agents that run locally, and that is what exactly we are doing. Now, before I take you back to that um terminal, let me show you something very interesting. This is a post trading recipe where specialist models are

02:34 trained first, each with its own fine-tuning and reinforcement learning, then their skills are distilled into one generalist. And the last step is agentic reinforcement learning inside real agent harnesses, and that is why I’m using one of the code harnesses. So, this model has practiced inside the very harnesses we are testing it in. So, let’s see how it performs. Also, this is Wahab Mirza, and I welcome you to the channel. If you want to help out the channel, please become a member, and

03:05 please subscribe to this free weekly AI newsletter at wahabmirza.substack.com. As you just saw in my VRAM consumption, I’m using um H100. If you’re looking to rent a GPU on very good price, you can find the link to Mass Compute in video’s description. Okay, so it is still working through it. Let’s wait for it to come back, and we will check out the result. And there you go, the model has come back, and you know what? The reasoning and the result looks quite promising because

03:36 uh look, before I show you the result, this is important to know because this is uh what varies from model to model. You see, it has avoided the traps, no clamping, and no seed edits, and it has done that with minimal thinking possible, but yet it hasn’t really compromised on any sort of looping and reasoning depth, which is quite good. It has taken its fair bit of time, around 6 to 7 minutes, but didn’t overthink. Anyway, the proof is in the pudding, so

04:06 I’m just going to go back and do control F5. There you go. Looks awesome. So, everything is green now and the yield has dropped below 100, which looks good. So, the bug has been fixed. Not only that, you can see that all the processing runs are now good enough and all the side silo levels are good. Plus overall yield and storage used, plus yield by run is under 100. So, I think first result it has passed it with

04:37 flying colors. For the next test, I’m asking the model to build a tattoo on my thigh or wherever. So, I’m just going to give it this prompt. While it works through it, let me show you the prompt again. So, now this prompt is quite hard as you can imagine. We are testing whether the model can build a complex animated simulation, procedural skin, a moving tattoo machine, and a layered artwork in one self-contained HTML file. So, not

05:07 only a coding build is being tested, lot of other things as you can imagine. And the idea is from a Japanese Izumi dragon coiling around the thigh with crashing waves, storm clouds, and peonies, the style, lot of other things. So, let’s see what model does here. While it does that, let’s talk a bit more around these benchmarks very quickly. So, this is the story where the model wins. Liquid’s own chart shows that this model

05:38 is pulling clearly ahead on the agent and tool use test like automation bench, live claw, and the tool sandbox. And it also leads on following instructions. The competition is with coins 35 billion A3B model, which is not only a bigger model, but a really celebrated model. Um so, that still holds the edge on the heavy coding web browsing and deep science test like Sweet Bench verified and GPQA diamond and few others. So, this model

06:09 is built to be a fast reliable agent that calls tools and follows orders, not a know-it-all and that is exactly the job we just gave it in Hermes. So, keep in mind that these are companies own published results, so I really don’t fall for these benchmarks numbers. We only check what is um shown in our results. So, let’s wait for this thing to come back and then we will go from there. And quite rapidly, model says that it

06:40 has produced this tattoo.html and this is what it has done. Like dragon is there, various things are there, whatever I asked it to do, it is telling me that it has done it. So, what I’m going to do, I’m going to open this HTML in front of you. Please don’t judge my thigh. So, let me show you my tattoo. There you go. It actually looks like my thigh, but I don’t know what that is. So, I’ll just replay, nothing happens, doesn’t matter, pause, I’ll just increase the speed, replay.

07:12 Where is the machine? Uh So, I think this does not really look anything like a tattoo or something. Maybe if I would draw it, it would look like that. So, you can already tell that model has failed miserably. Okay, let’s do another test. Now, for the next test, we are asking the model to name the favorite food of 81 language communities in each language’s own script to see how far the multilingual ability

07:42 really goes down to rare languages and pure gibberish. And this is what model has come up with. Please check out your language and let me know if this is correct or not. I’m just checking it out and you know what? I think I would also leave the answer the final answer to you guys, but mostly the big languages I think it has got it right. Uh with correct scripting and believable national dish each time. I know this could be quite subjective, too. So, for English, for example, they say

08:14 it is fish and chips. Maybe New Zealand and Australia, but not sure. Uh Mandarin Chinese and then this is Hindi. Not sure what that means. So, please check your language and let me know. Um Urdu biryani, yeah, very close. I would say this is Arabic. Um and various other languages. I’m just going to scroll down. And this is actually quite interesting, but let me know your thoughts. But I can tell that few of the languages it has just broken down. Uh some of the words are mixed up. So,

08:44 multilinguality doesn’t seem to be a forte of this model at all. So, you see this is what it has given me. I’m not really sure about the food which it it has given us. And I know that we have um viewers from almost every language which is mentioned here. So, I’m just going to show it to you once again. And I am not sure again.

09:14 So, for Indonesian nasi goreng, maybe it is correct. So, let me know, please. And now let’s do a scientific reasoning question with a touch of math. This prompt is a deep sea submarine design problem and test whether it can chain pressure, hull strength, and buoyancy together and catch that the sub would hover halfway down instead of reaching the bottom. I already have given this to model. Model has produced answer and model got the first stretch

09:45 right. Like about pressure, wall thickness, the thin wall check and the outer volume. Um this looks pretty good. Now then it started slipping. Like a thin shell shortcut gave a hull mass of 607 kg instead of 663 kg which turned the net buoyancy into plus 68 kg instead of plus 14 kg. So the difference is huge. It also has missed a very key insight reporting a meaningless 27 km neutral depth instead of the bear sub hovering

10:17 at about 4,089 4 m. So I think again an average response. I believe what they have said in their model card that this is not a know-it-all model. For scientific reasoning math, maybe you know, day-to-day work is okay, but don’t expect it to, you know, hit it out of the park on complex reasoning and math. And now finally I’m going to do a situational test totally hypothetical just testing models guardrails and ethical boundaries. So this is a prompt

10:47 which I’m going to give the model. Um it needs to plan a secret wedding ceremony. I’ll and you can read it through. Let me not uh don’t make me read it. So let me run this. So it needs to provide me clear steps and all that. So model cannot help me with that. So the guardrails are up. Model is quite ethical. All good. Let me know what do you think about this Liquid’s new model. Please put it in the comments again. If you want to help out the channel, please become a member from

11:19 this link. By the way, there’s a difference between a subscriber and a member. Members are what keeps the lights on. So very grateful to them. Also grateful to subscribers, but really membership just takes it to the new whole level. Thank you for all the support.