Amos 1 Small: 69.5% Accurate, Zero Tokens, 5 GB VRAM

Author: Prompt Engineer 48 · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Amos 1 Small: 69.5% Accurate, Zero Tokens, 5 GB VRAM · Channel: Prompt Engineer 48 · 08:16 · Watch on YouTube

Topics: AI agents, Task and workflow automation, AI evaluation and benchmarks, Local and open-source models, Multimodal models

Summary

The video tests Imos 1 Small, an open-weight model that returns probabilities for typed questions instead of generating text. The presenter checks its benchmark claims, then runs it locally on an RTX 4060 laptop with 8 GB of VRAM, where it uses just over 5 GB and responds in roughly a fifth of a second. Tests cover insurance claims, potentially destructive agent actions, image questions, support-ticket routing, and a few non-English examples. The presenter sees promise for high-volume classification with human review for uncertain answers, but cautions against relying on it alone for severity ratings, precise photo boxes, or tasks that require written responses.

Key points

Imos 1 Small returns probabilities for predefined answers, such as yes/no, a choice, a score, a quote, or a point on an image.
The model card reports 69.5% on 2,000 decisions across four workflows; the presenter notes that it trails the listed comparison models.
The model card says ratings are its least reliable answers and describes the model as English-only.
The local setup used an RTX 4060 laptop with 8 GB of VRAM; the model used just over 5 GB.
Across 30 repeated requests, the median response time was 187 milliseconds; the fastest was 157 ms and the slowest 215 ms.
In the agent-run test, the model identified the production database deletion as destructive, but rated its severity as serious rather than critical.
On one photo, it identified cats and roughly located a remote control, but the bounding box was loose and had 0.49 confidence.
It routed 9 of 10 self-written support tickets correctly; the presenter suggests sending low-confidence cases to a person.
Three simple non-English versions of one ticket were classified as billing, but the presenter says this does not establish broader non-English performance.
The presenter recommends it for high-volume typed decisions, not for generating explanations or making certain decisions without human review.

Chapters

Start	Chapter	Summary
00:00	A model that answers without writing	The presenter introduces Imos 1’s probability-based answers and explains the model sizes and intended use.
01:01	Checking the model card and hosted demo	The presenter reviews the model card’s benchmark results and limitations, then demonstrates how changing evidence shifts an insurance decision.
03:04	Installing and benchmarking the local model	The model is run on an 8 GB laptop GPU, and the presenter measures its memory use, response times, and sensitivity to evidence.
04:38	Testing agent actions and image questions	The model flags a destructive database command but underestimates its severity, then answers cat questions and roughly points to a remote control.
06:12	Support-ticket routing and final assessment	The presenter tests ticket routing and a few non-English examples, then describes suitable uses and cautions before concluding.

Quotes

It did not write a single word. — 00:00

The answer is a number, and the number reacts to the evidence. — 02:32

The median is 187 milliseconds. — 04:06

10 tickets is a small sample, so don’t read it as an accuracy number. — 06:43

Anything that needs to write, explain, or reason in words. — 07:45

Concepts and entities

Amos 1 Small, Amos 1, Decision Index, Apache License 2.0, RTX 4060, Python, UV, PyTorch, NVIDIA-SMI, RunPod

Full transcript (original language)

00:00 An AI agent just deleted a production database. This model read the whole run and coded the exact command that did it. Confidence 0.9997. And look at the usage line here, output tokens zero. It did not write a single word. It answered in 380 milliseconds on this laptop. Hi, I’m prompt engineer. Let’s see how a model answers without talking. This is Imos 1 from imos.zero.ai. It’s a family of open weight models and the idea is a bit different from a chat model. You hand it a case that can be text, JSON, or a

00:30 photo, and you hand it a list of typed questions. Each question is a yes, no, a pick one, a pick many, a score, a quote, or a point on a photo. And what comes back is a probability for every answer you allow. No sentence to parse, no broken JSON. There are four sizes. Large is 27 billion parameters, medium is nine, small is two, and mini is half a billion. Mini is the default on the hosted API. We are going to use small because the model page says it runs on an 8 GB GPU, and that is exactly what I

01:01 have in this laptop. The point is that your code can act on the confident answers and send the uncertain ones to a person. So, it’s built for volume, things like insurance claims, support tickets, and agent supervision. Now, before I trust any of that, let’s read what the page actually claims. This is the model card for Imos 1 small. The headline table is typed decisions. 2,000 decisions from four workflows it never trained on. Imos 1 small gets 69.5%. Look at the rows below it. Microsoft

01:31 Decision 1 is at 79.5, and that one has closed weights. Decider 1 is at 76.8, but the card itself says that number is from its makers, not rerun. Claf from Cloudflare is at 72.5, so small is behind all of them on this table. The decision index public part says 31.92, and the card says the official ranking also uses private tests that only the board maintainers run. So, treat that number as partial. And the limits are written right there. Seven points behind medium weaker on broad knowledge,

02:02 ratings are its least reliable answers, and English only the license is Apache 2.0. That’s a clean license, and I’ll take an honest limit section over a hype page any day. One more claim to check later, 48 milliseconds. Hold on to that number because I’m going to measure it. Before I install anything, the page has a playground, so let’s use it first here. I picked the insurance claim case on the right. You can see a probability bar for each question. The question here is, “What should happen to this claim next?” Right now, the top answer is ask

02:32 for report at 58%. The policy says, “Third-party claims need a police report number.” Now, I select that one sentence, the policy line, and delete it. And you can see the top answer flips to medical team at 82%. 713 tokens became 693. That’s the whole product in one gesture. The answer is a number, and the number reacts to the evidence. This runs on their server, so the speed here tells me nothing about my own machine. The model page also says the small model sleeps after two idle hours and takes

03:04 about 2 minutes to wake. So, if you try the hosted small model, and it hangs, that’s why. Okay, let’s bring it local. Now, the local run, mind you, everything from here on is running on this RTX 4060 laptop with 8 GB of VRAM. First, the weights, it’s 4.24 GB with HF download. Then, I make a Python 3.12 environment with UV because the package pins Python 312 and PyTorch 2.14. I install torch for CUDA 13, and then I install the serve folder that comes inside the model

03:35 repo. And then, we start the server. Loading weights takes about 6 seconds, and if you look at NVIDIA SMI, it sits at just over 5 GB used, so it fits on 8 GB with room left. Then, the Readme’s curl example. One thing tripped me up here. After the server sat idle for a while, the first call took 6.9 seconds, and then it was back to normal. And when I call localhost, each request took about 350 milliseconds. The same request to 127.0.0.1 took about 120. I think Windows is

04:06 trying IPv6 first, so on Windows, use the IP address. Test one is the example from their readme, and invoice charge twice and three questions is the customer asking for a refund. 0.969, which team billing 0.957, how urgent today 0.941. Now, the speed. I send that same request 30 times. The median is 187 milliseconds. The fastest is 157, and the slowest is 215. So, that 48 millisecond claim is a server-side number. And on my laptop, it’s about

04:38 four times that. The very first request took 1.6 seconds because of the cold start. Then, the insurance claim with a plumber’s invoice sentence in evidence attached is 0.967. I cut just that sentence, and it falls to 0.294. The pay without review question moves from 0.307 to 0.192. So, it reads the evidence, not just the topic. Test two is the one from the start. Three sequel steps from an agent that was told to clean up staging tables. The third step is drop database production customers. Destructive action

05:10 0.91. Page a human 0.874. And the quote, exactly that command, in 380 milliseconds. But here’s where I want to be honest. I also asked for a severity, harmless to critical. It put 60% on serious and only 22% on critical deleting a production database with no backup and no approval, and the model says serious. So, for severity levels, I would not let it decide alone, which matches the card saying ratings are its weakest. And when I remove the drop database step, the destructive answer is

05:41 0.59, and page a human is 0.42. It’s not sure, and it says so. That’s useful because that’s the case you send to a person. Test three, a photo, two cats on a couch, the standard Coco test image. Is there a cat? 0.977. Is there a dog? 0.033. Fine. Then I ask it to point at the remote control. It gives me a box and you can see the red box here. It lands on the remote on the right, but it’s loose and it clips the cat’s head and its confidence is 0.49. So, pointing

06:12 works roughly, not precisely, on this one image. One photo is not a benchmark. So, the card’s 97% on Coco is still a claim. I haven’t repeated. And the photo question took about 1 second. Test four is the one I care about most because it’s the real use. 10 support tickets I wrote myself, three teams, billing, technical sales. I send them one after another to the server. Nine out of 10 are routed correctly. The whole batch took 2 seconds, about 203 milliseconds each. Now, look at the one it got wrong. Ticket nine, can I get a quote for

06:43 annual billing across three departments? I labeled that sales. It said billing, but its confidence there is 0.688, the lowest in the batch with the next lowest at 0.783. So, that’s exactly how the card says you should use it. Set a threshold, say 0.8, and this one goes to a human instead of the wrong team. 10 tickets is a small sample, so don’t read it as an accuracy number. It’s a demo of the confidence doing its job. One last check. The card says English only, so I took the same double charge ticket and wrote it in

07:13 Spanish, in Hindi with English letters, and in Devanagari script. All three came back billing with 0.952, 0.919, and 0.924. So, on this one simple ticket, it worked. Does that mean the card is wrong? No. Three sentences of the easiest kind of ticket don’t prove anything about harder cases, and the authors only tested English. I’d try your own language on your own data before relying on it. So, who is this for? If you’re building something that has to make many yes, no, or pick one calls like routing tickets or flagging

07:45 risky agent runs, and you want a number per answer with no text parsing small on a single 8 gigabyte card is real. 5 gigabytes of VRAM about a fifth of a second. Who is it not for? Anything that needs to write, explain, or reason in words. Anything not in English, and I would not use it alone for severity ratings or tight photo boxes. Next, I want to try the medium model against this same test file and see if those seven points show in the agent case. Let me know in the comments if you want that. If you’re looking for cheap and reliable GPUs, I use RunPod, and the

08:16 link is in the description. So, in this video, we ran almost one small on a laptop and measured it. If you’re interested, go ahead and try it on your own system and let me know how it goes. I’ll see you in another one. Check out the next video.