Run LFM3 1.7B Locally: Model for Specialized Applications

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Run LFM3 1.7B Locally: Model for Specialized Applications · Channel: Fahd Mirza · 10:09 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, Software engineering with AI, AI models, Local and open-source models

Summary

The video tests Liquid AI’s LFM3 1.7B model locally through llama.cpp and a Hermes agent, using a mix of coding, structured-output, reasoning, and tool-use tasks. The model fails to fix a planted bug in a restaurant app and produces a blank HTML simulation because of a simple coding error. It performs well on structured JSON, strict instruction following, reasoning, grounded answers, and tool calling, but struggles with translation across 80 languages. The presenter concludes that the model is capable in several specialized tasks, though coding remains a weakness.

Key points

LFM3 1.7B is described as a small hybrid-architecture model with a 256K context window, post-trained for agentic harnesses.
The presenter runs the model with llama.cpp and integrates it with Hermes agent.
The model fails to identify and fix the planted food-cost bug in the Wildfire restaurant application.
Its generated HTML simulation is blank because the code uses an undefined canvas variable.
It successfully extracts valid JSON from messy text and follows strict sentence-count, word-count, and formatting constraints.
Its translations across 80 languages are mostly incorrect, leading the presenter to mark the multilingual test as a failure.
It avoids inventing a summary for a purportedly nonexistent paper, though it attributes the issue to missing tools.
It correctly calculates ingredient requirements, answers questions using only provided text, and generates valid tool calls.
The presenter’s overall assessment is that the model performs well outside coding.

Chapters

Start	Chapter	Summary
00:01	Introducing and running LFM3 1.7B	The presenter introduces Liquid AI’s new model family, describes LFM3 1.7B, and begins testing it locally with llama.cpp and Hermes agent.
02:11	Agentic app repair and coding tests	The model fails to fix the planted restaurant-app bug and then produces a blank HTML simulation with an undefined variable.
05:53	Structured output, instructions, and translation	The model succeeds at JSON extraction and strict instruction following but performs poorly on the 80-language translation task.
08:03	Hallucination, reasoning, and tool use	The presenter tests the model’s handling of a nonexistent paper, multi-step calculations, grounded answers, and tool calling before giving an overall assessment.

Quotes

at 4-bit it fits in about a gigabyte of memory. — 00:33

I would call this that this is a fail. — 04:18

JSON is valid and every value is correct. — 06:29

I think that is again a fail. — 07:32

other than coding, this 1.7 billion parameter model has performed well — 09:38

Concepts and entities

Liquid AI, LFM3 1.7B, llama.cpp, Hermes Agent, Wildfire, saída estruturada, tool calling

Full transcript (original language)

00:01 Liquid AI has released a whole family of models in one day. Text, vision, audio, omni, and even a decision model. Bulk releases like this always make me a little nervous because when you ship this many models at once, quality can slip. But Liquid says these are specialized models and the benchmark numbers look quite good. So, let’s stop guessing and we are going to test the first one, which is this LFM3 1.7 billion, a 1.7 billion parameter model,

00:33 tiny by today’s standards with a hybrid architecture, which we will also see shortly, and 256K context window. It is post trained for agentic harnesses like Hermes agent and at 4-bit it fits in about a gigabyte of memory. So, what I’m going to do in this video, I’m going to inst- you know, run it with llama.cpp. As you can see, it is already running. At the moment, it is consuming you know, VRAM like 4.5 gig because I’m running it in BF16, but you can go with

01:06 the 4-bit or even lower if you like. And this is where it is integrated with my Hermes agent. First up, what I’m going to do here is to give it this application, which is a broken app. It is Wildfire, the kitchen OS of a Surrey Hills restaurant serving crocodile, kangaroo, emu, camel, wallaby, and more. Basically, a nature documentary except you get to eat the cast. The dashboard costs every plate. As you can see, it tracks stock, it

01:39 streams live orders from the floor. Now, look closely at the top. Let me quickly take you up. So, you see food cost says 18.9% excellent and not one dish is over target. But the supplier invoices say we spent 34.3% of sales on food. Both cannot be true, right? So, for example, if I scroll down and show you the food cost by this dish and look at this wallaby shank here, for example. So, you see, we serve 220 g on the

02:11 plate, but the app says we only buy about 101 g. Exotic meat loses a lot of weight to trimming and boning, so you must buy more than you serve. Here, the app is thinking we buy less. There is one planted bug behind all of it and quite a subtle one. And the model has to find it, and that is what [clears throat] I’m going to give this as a goal in this Hermes agent. Let’s see if it is able to do that in this agentic fashion or not and how quickly. While that happens, let’s talk

02:45 a bit more around this model. The numbers which they have shared on their um blog post regarding the benchmarks are quite striking. Look at this. Against models from Cohere and Google mini CPM, including Cohere’s 9 billion parameter model, that is several times its size, this tiny model leads or stays right up with them on most of the agent and tool use test. It though trails the bigger models in a couple of areas, and agentic coding is where the gap shows most.

03:16 Um but the thing is that the benchmarks, given the size of the model, are very interesting, and that is exactly what we are testing here. By the way, recently I have started this free weekly AI newsletter at fahadmirza.substack.com, and if you want to help out the channel, please become a member. And if you’re looking to rent a GPU on very good price, you can find the link to Mass Compute in video’s description. The model has come back with the response, and if you go through this

03:47 response, it says goal achieved, and then it has fixed all of the numbers. But the thing is these are the same numbers which were broken we started with. If I take you back to the website and then if I reload it, I have just pressed control F5. You see, these are the same numbers. Nothing has really changed. The same food cost and if I just go to wallaby shank. These are the same numbers. Nothing um

04:18 has changed. It still says it might uh you know, the same number of grams as earlier. So, uh I would call this that this is a fail. Which is not really what the benchmarks say here. Anyway, let’s do now a code generation test. Next up, I asked it to create the self-contained HTML file to simulate a donair kebab and this is what it has done. It says

04:48 that it has created everything. So, I just copied this from here and when I open it and I’ll open it again in front of you, it just shows me blank page. So, nothing. So, I’m going to give it another chance and I’m going to ask it that the page is blank. HTML preview is blank. Sorry. And doesn’t show any simulation. Let’s see if it is able to

05:18 fix it up now. Okay, one more chance and it says that it has done it again. Let me just open it and maybe I will just put it in another tab. There you go. Still nothing. And let me have a quick look at the code. So, this is the code. I have quickly reviewed it and this line 16, it uses a variable canvas that is never defined. And that is a very very simple error which no agentic or coding

05:53 model in this day and age should miss even if it is under 2 billion parameter. So, it’s a I don’t think so the coding is there yet. So, the second test test is also failed. Let’s do another one. Now, to be fair to the model, this is not exactly your typical coding model. And the card also says that. So, I’m going to not play according to the strengths of the model. First up, I’m going to test this structured output by pulling clean JSON out of this messy text. So, this is my prompt. Let’s run this to check it out.

06:29 And this is the answer. And the good thing is that it has matched their claim. JSON is valid and every value is correct. Supplier 12 kg wallaby shank and all that stuff. It left out the auto stock crocodile and returned only the JSON as I asked. So, on structured output, as I said, the model matches their own claim. Pretty good. Let’s do another one. Next one is about strict instruction following where the model must meet several hard constraints at once. So, the ask was to write exactly four sentences about the northern lights

07:02 starting with star or s t a n r with no commas and under 60 words. So, you see the result is pass. Four sentences starting with star, no commas, and exactly 43 words. So, that is also an impressive result. Next up, this is multilingual test where I’m asking it to translate this sentence don’t count your chicken before they hatch into all these 80 languages. And then you can see that it has done some translation.

07:32 Now, most of it is is not correct. You can check your language, too. I think Spanish, French, Russian contain words that don’t even exist. Latin and Turkish are left partly in English, the legal Canada southeastern languages, all wrong scripts, and model admits that it cannot really do any multilinguality at scale. So, I think that is again a fail. Maybe it is not a multilingual model at all.

08:03 Next, I’m testing hallucination using a paper that does not exist, as you can see from this prompt here. I’m asking it to summarize the finding of a made-up 1987 paper by Dr. Margaret Elderwick. And you can see that it has passed it with flying colors on the main point since it invented no summary at all, but then it also blamed missing tools and never said the paper might not exist. But you know what? I would take it as a pass because we don’t nitpick here too

08:34 much. Next up, I’m testing this multi-step reasoning using the same yield math that broke the wildfire application, by the way. And then look at this response. It works out how much wallaby shank to buy for 150 plates, and it is a pass. So, 71.74 kg. The numbers are right, and I think that it is quite a good answer, and very quickly, too, as you can see from this response. Next one is all about grounded answering

09:06 where the model must use only the given text and say not in the text when the answer to this missing and you can see for all these three questions where I have asked this to answer from a short passage, model has passed all three, which is including not in the text. So, it didn’t make up anything, which is really impressive. And finally, I’m doing the test of tool calling where the model must pick the right tools and fill in the arguments, including two calls in one request. So, if you look at the prompt, the ask is to set a reminder to call the supplier at

09:38 3:00 p.m. tomorrow and get the weather in Sydney in Celsius. After thinking a lot, it has come back with the response and given me the valid JSON with both correct tools and correct arguments. There is one slight slip, which is the reminder time, which is 3:00 p.m. UTC and not 3:00 p.m. Sydney time because I didn’t really give it any time zone, so I will call call it as a pass. So, you see other than coding, this 1.7 billion parameter model has performed well and the performance is quite good, too. So,

10:09 check it out. Let me know what do you think about this model. We will be covering the rest of the lot very soon and so say stay tuned and thank you for all the support.