Holo4: A Model That Clicks, Codes and Calls Tools

Author: Sam Witteveen · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Holo4: A Model That Clicks, Codes and Calls Tools · Channel: Sam Witteveen · 18:00 · Watch on YouTube

Topics: AI agents, AI models, Local and open-source models

Summary

Holo4 is an open-weight model designed to choose between using a graphical interface, writing code, and calling tools or APIs within a single agent workflow. The video explains how the model fits into an agent loop, reviews its model variants and benchmark results, and describes training with interactive tasks and reinforcement learning. Demonstrations show it switching between GUI actions, code, and MCP tools depending on what is available. The main caveat is its non-commercial license, though its benchmark trajectories are publicly available.

Key points

Holo4 is trained to select among GUI interactions, code execution, and tool or API calls rather than relying on a separate model for each.
In an agent loop, the model decides what to do, while the harness executes the action and returns the result to the model.
GUI use can work with interfaces that have no API, but may be slow, fragile, and costly in context.
The model family includes Holo4 27B dense, Holo4 35B MoE with 3B active parameters, and Holotron Nano.
The video reports that the 27B dense model performs substantially better than the MoE variant on several visual agent benchmarks.
H Company trained the models using interactive environments, supervised fine-tuning, and reinforcement learning to reward effective action choices.
The demonstrations show Holo4 using a GUI and code, switching to an MCP tool when one is available, and handling a code-only task.
The model is available with a non-commercial license; shipping a product requires using H Company’s API or contacting the company.
H Company has published trajectories behind its benchmark scores, which can be examined for research and learning.

Chapters

Start	Chapter	Summary
00:00	Why agents need to cross interfaces	The video introduces the limits of agents specialized in either visual computer use or API and tool calling, and presents Holo4 as a model intended to bridge those workflows.
03:10	The three action lanes	The video compares GUI interaction, code execution, and tools or APIs, explaining why Holo4 is trained to choose among them.
06:19	Models, benchmarks, and compute	The video outlines Holo4’s model variants and deployment considerations, then discusses benchmark results and the reported advantage of the dense model over the MoE version.
09:27	How Holo4 learns to choose	The video describes H Company’s interactive task factory, supervised fine-tuning, reinforcement learning, and work on the agent harness.
12:11	Demonstrations across GUI, code, and tools	Examples show Holo4 navigating a website, combining GUI actions with code, using an MCP tool when available, and completing a code-focused task.
15:51	License, trajectories, and takeaways	The video discusses the non-commercial license, the release of benchmark trajectories, and why agents that choose between interfaces may be important.

Quotes

The harness actually does all the doing — 02:40

The big difference here, though, is that they’ve trained this model to do all three together — 05:17

So the training itself is supervised fine-tuning on about 127 billion tokens. — 10:36

this is actually a non-commercial license, right? — 16:22

Concepts and entities

Holo4, H Company, Holo4 27B, Holo4 35Ba3b, Holotron Nano, Qwen, Nemotron 3 Nano Omni, High Agents, MCP, Harness, Agentic task factory, reinforcement learning, OSWorld, Hugging Face

Full transcript (original language)

00:00 Okay, so imagine you’ve built an agent that can use a computer. It looks at a screen, it can click, it can type, but now you drop that into a server with no screen. it’s just gonna get stuck. It’s got nothing to look at. It was built for visual kinds of things. If you flip it around, if you’ve got an agent that’s great at calling tools and APIs, but you then ask it to pull a report out of some old app or something like that, that has no API, it’s also stuck, right? It doesn’t have the ability to look at the UI. It doesn’t have the ability to actually act on that and so far up

00:32 until now, people have generally tended to make models to do one side of that line or the other. So either being able to do the visual kind of stuff, and act on things, click on things, that kind of thing, or being able to actually do the API related stuff, the non-visual kind of stuff Unfortunately though, the problem is that real work doesn’t stay on one side or the other of that line more and more as people using agents to do a whole variety of different tasks, it often means that it’s gotta click through some legacy UI.

01:03 It’s then gotta be cleaning a CSV file. It’s then gotta be posting it to Slack or to some other API And right now, the way that people have sort of handled this is you tend to have lots of different models. You chain models together. So you’ve got one visual model that you force everything as screenshots. You’ve got another model which is doing your orchestration and dealing with MCPs, et cetera So in this video, I wanna talk about a model from a company in Paris called H Company and they’re the lab behind Holo Models. They’ve just released Holo4, And that’s what I want to go through here.

01:34 This is basically an open weight model trained to click, to write, to run code, to call MCPs, to do API tools, et cetera. And most importantly, to be able to orchestrate and choose between those actions as it goes along And I kind of feel there are two really interesting parts here. There’s the model itself, which we’ll look at, but the other part is how you train a model to make that choice and I should point out that they’ve actually got a family of Holo models here. So we’ll have a look at those. We’ll have a look at how they fit into an agent, how they’re built, and

02:06 also perhaps where they’re falling short, et cetera Alright, so let’s recap what an actual agent loop is you basically give the agent a task. The model looks at what’s going on and it decides on one action, right? Then the harness, which is just the code around it, executes that action. It does the click, it runs code, it calls the tool. And then obviously it captures what happened, whether that’s a new screenshot, whether that’s just some return from an API, and it feeds that back into the model along with the memory of the previous steps around it

02:40 And this loop just goes round and round, right? This is fundamentally the base of what’s going on here I guess the key thing here is that the model itself never actually touches anything. It only decides. the harness actually does all the doing And that matters more and more nowadays. When we look at benchmarks, we see that models can score really differently depending on the harness that are actually running with it Now for Holo 4, they’ve actually optimized this to work with three main lanes of sort of actions here. The first are GUIs, right?

03:10 Graphical user interfaces. The model looks at a screenshot, then says, “Click here, type this,” and you can kind of think of it as the agent using the computer a web device, a browser the way that you do it. it works on anything that a human can use, but it’s gonna be slow, and it can often break when someone moves a button or changes the UI, et cetera. The second lane is code. So the model writes a script, runs it in a sandbox or a shell, and it gets an output back. That’s fast, that’s precise, but it only works when you can actually write some

03:42 software to do the particular task, right? it can’t just easily write a Python code to know which button to press on a visual output. the third lane relates a bit to the second lane, but this is where you’ve got tools. So MCP servers, APIs, that kind of thing. The model calls a function and it gets back structured data. and in many ways this is the most reliable of the three here. but obviously the challenge is it only works if there is an MCP or an API or some kind of tool that is actually gonna do what you want it to do now

04:14 the challenge has been for many people running agents, especially local agents, that these three different lanes often require either different models or different setups with your agent harness. If you’ve got something that’s doing screenshots, that’s going to eat, a huge amount of context compared to just JSON files or a few lines from a terminal output, et cetera and for exactly that reason, most people have tended to ignore running something that could actually do visual stuff just because it’s often not practical to do locally and going back to the example I talked about in the intro,

04:47 something like exporting a report from some kind of legacy app through a GUI. you want it to go through an e-commerce site and find different things that are not available through their API then you wanna take that, clean it up for a CSV file with a Python script, and then post it to an API like a Slack, et cetera is really the type of goal that Holo-4 is sort of working on here. They talk about this as being working across interfaces it’s not just that it supports MCP as a feature, it’s trained to pick the right lane

05:17 between those images, code, or tools And in many ways, that’s what sets it apart. The earlier Holo models that they already trained were using function calling. The big difference here, though, is that they’ve trained this model to do all three together Graphical user interfaces is a first-class option, code is a first-class option, and tools are a first-class option. They’re not just add-ons now the big question for me was Would you use this as your only model or alongside of another one? and the answer is perhaps kind of both.

05:48 H runs all their benchmarks with Holo as the sole model, doing both the planning and the acting But you could also put this underneath a bigger model for doing orchestration or a different model if you wanted to do that H actually has their own harness and SDK called High Agents, which runs their hosted agents and can plug into things like Claude Code over MCP. so your main model plans the job and hands the screen-heavy work to Holo And I do think that may make sense for certain use cases where you just want

06:19 to use this for doing the GUI stuff, for doing all the computer use agents kind of stuff All right, so let’s look at what each company actually shipped here. They’ve got two main models here, the Holo4 27B, which is a dense model, built on the Qwen 3.8 27B, and then they’ve also got the Holo4 35Ba3b, right? So that’s basically a mixture of experts model with 3B active. That one is actually still built on the Qwen 3.6 model Along with those two models, they’ve also released Holotron Nano, which is the same recipe for

06:51 actually making the first two, but applied to NVIDIA’s Nemotron 3 Nano Omni there And so I made a video about that Nemotron 3 Nano Omni, a mouthful of a name there. but you can check that out if you want to see more of some of the reasons why people are using that smaller model. All right, so the weights for all of these are on Hugging Face. They’ve got a bunch of different quantizations, et cetera. They’ve also got their own API if you want to run it through that. The full bfloat 16-bit version here is 54 gigs. So I’m gonna be running this on my Dell Pro Max machine

07:22 with an RTX Pro 6000 in here. So a big shout-out and thanks to Dell for sponsoring the compute for this one of the challenges that you wanna think about wherever you’re gonna choose to run this is that screenshots fill up your KV cache very fast. So you need to make sure that you’re leaving, plenty of headroom specifically for that Okay, so if we look at the benchmarks in here, we can see that they’re actually comparing, both the 27B dense version and the mixture of experts version and the original Qwen 3.8 27B, along with a bunch of

07:53 models that were sort of frontier perhaps a week, two weeks ago. And I got to say, I feel for a lot of these companies when you basically benchmark what you’re about to release and then the frontier labs go and release something completely new but honestly, I feel like the frontier ones are more just for a guide of what sort of state of the art for this thing. Obviously, none of those we can run locally, and obviously if you’re paying for API usage with images on those, that can get very expensive very quickly So if we look at things like OS World, they’re really not that far behind the frontier models.

08:25 In fact, you can see the 27B here is actually a little bit ahead of what now is the old GPT-5.5 model Now, when it comes to OS world, yes, they’re quite a bit behind But again, don’t just look at the accuracy scores here. Really pay attention to the costs, right? Because while Opus 5.5 and Astra are coming out on top, you can see that the cost of running those compared to the 27B is like seven times more here the other big takeaway I would say from the benchmarks here is it’s

08:55 really kind of fascinating just how far behind the mixture of expert model is. So when we look at something like OS World, we can see that the 35B MoE is scoring less than half what the 27B dense model is and while it’s not as dramatic as that for the other benchmarks in here, there really is a clear advantage to using the dense model for these kinds of tasks. and I guess it makes more sense that if we compare sort of active parameters, we’re comparing 27 billion to 3 billion and it just does seem that for images,

09:27 and getting representations out of images The 27 billion dense model just does that so much better here All right, so just quickly, how do they actually train the model to choose a latent like this? ’cause this is one of the parts that I find really interesting here. So H Company built what they’re calling an agentic task factory. It’s a set of pipelines that build like interactive environments and tasks that can be checked automatically. So things like screenshots of real websites, open source software using specific web apps, specific MCP servers, and even things like desktop environments

10:02 and the interesting thing there is that some of those environments are kind of hybrid, meaning that you could take the state of that, for example, like a website or something, and you could actually do it through either a GUI way by looking at the website through screenshots, or you could do it perhaps by an MCP way or by using some code to look for something on the website. so over many of these different tasks then, the model can actually solve the task by clicking or by calling a tool And then reinforcement learning sits over all of that and rewards the model for whatever works the best in that situation.

10:36 so that’s how the model learns that clicking through five menus is perhaps gonna be the slow way, when there’s an API sitting there And also vice versa, that perhaps using a screenshot to get a bunch of detailed information when there is no detailed API available is going to be the way that the RL reinforces it and makes the model better at things like that. So the training itself is supervised fine-tuning on about 127 billion tokens. then creating two separate RL experts which get merged back into the one model on top of that, the other thing that they did was rebuild their actual

11:09 harness to actually fit in with this They were then able to basically tag sort of what failed and have engineers review those that look at like how could we improve the harness to actually work with this kind of output from the model. so it’s interesting here that some of the biggest wins were a reliable memory that can track sort of hundreds of steps and it can relate those steps together perhaps in a better way because it’s been trained on these three different lanes Alright. So that same recipe was used for the Qwen 3.8 dense model, the 3.6 MOE

11:40 and the Nemotron And I think it’s pretty clear that what they’re really focusing on here is a really strong post-training pipeline rather than over customizations for any one specific model. They’re really setting themselves up here for when the Qwen4 models come out and Alibaba has actually recently confirmed that there is going to be a Qwen 4 27B. So I think everyone’s gonna be looking forward to that. and you could imagine a Holo 4.5 or 5 that actually improves on all of this even more All right, so let’s take a quick look and demo at how this actually works

12:11 Okay, so you can see here this is going to use the GUI element of this. So it’s basically going to the Hugging Face page. we’ve given it, the task of open the Holo-4 collection, and list out the model’s, precisions. And you can see that sure enough, it’s gone to, different elements there. It’s actually finding the collections, the Holo-4 collection, and it will go on, for quite a while to do this. But you’ll notice at the bottom that everything, going through here is, actually a GUI task in there.

12:42 so let me just stop this one, and let’s look at some of the other ones. I’ve built a UI that’s using a little mini harness, at the back end, And that harness, we can give it an MCP if we turn it on. We can give it, access to tools, access to code, et cetera. So I’m gonna try going through, the first task. . so here it’s basically got a made-up, admin panel so that this is running locally. You can actually see that this is a list of orders there. and it’s using the GUI in this one.

13:13 We’ve got the MCP turned off, so it doesn’t have the ability to use the MCP, anything here. So it’s literally having to navigate that, webpage and go through and get stuff. Now, it’s getting it, and then it’s actually using code, to put it all together. All right? So, you can see finally it comes back with the answer out. In this case, we can see that it basically downloaded the CSV file. It then ran Python on the CSV file, it then gave us our answer back. Now, it started out with the GUI there.

13:44 If we look at the bottom, we can see it went one, two, three GUI steps, then it went to code, and then it gave us our final answer out. let’s look at the exact same thing though now if we give it an MCP on. So now we’ve got, the MCP is actually on, so it doesn’t need to go to all the UI stuff now, right? It can actually just go and say, “Hey, we’ve got a tool, export orders,” and it will go through. it’s going to basically get, the same, kind of thing, and we’ve got the, answers out at the end, as well here, going through this.

14:18 And you can see that obviously that was a lot less steps, and none of them were GUI in this case. It was basically just a, the tool, and then code and then output there If we wanted to basically just do everything, where it’s, doing various clicks as it goes through, you can see it did a click on the overdue, it did a click on the export CSV file. Again, we had multiple GUIs there, and then it’s running code before it’s going to go back to the model and give us our final, response out. here’s an example that I put together where, it’s basically now just using code.

14:50 so it’s able to go through a file, pull out different information, reconcile the two different order exports here, and list any mismatches that it’s finding in there And the main thing, to notice here is that it’s just using code this time. it’s basically just putting that all together, and then at some point here, it’s gonna give us our final result out. And you can see actually We’ve got some environment issues going on there where it’s actually ran into some, problems with that You can see that eventually it’s starting to use Pandas, and then

15:20 finally we get our answer out. Now, it took quite a long time there, right? That was, a minute 37 seconds. I’ve obviously cut some bits out for the recording. but you can see that it’s done each of these steps, and that’s the whole thing with this model is that It can choose to just do the GUI and use just the graphical user interface and then process, those screenshots and then work out where to actually click, on the UI. It can do the code like we’ve seen in this one, and it can use, tools like

15:51 the MCP, if they’re available and it sees those in the tool repository going into the model, et cetera. Okay, so just to finish up, another cool thing that they’ve published are the actual trajectories, of what’s going on in here. So you can actually come in and have a look at how these trajectories actually went Now the big question to sort of finish up on is should you use this? well, I certainly think it’s worth checking out here. Unfortunately, the biggest issue with this, I would say, is the license. that while it’s open weight and you can check this out, this is actually a non-commercial license, right?

16:22 even though the Qwen base model underneath this is Apache 2.0, and even though their Holo 3.1, models were Apache 2.0, unfortunately this doesn’t fall into that category so you can experiment, you can prototype, you can do research with this. But if you want to ship a product, you’re gonna have to go through H’s API or you’re gonna need to contact them directly Now, one thing that I do think they deserve some credit for is that every trajectory behind these benchmark scores that they’ve put out are actually open. So you could actually go through and learn a lot from those if you were

16:56 trying to make your own model for doing this so for me, the takeaway here is that agents choosing between clicking, coding, calling, using GUIs, those sorts of things, is where things are heading. Holo-4 shows us that that ability comes from these training environments, comes from reinforcement learning, comes from basically incorporating the model to work with a specific harness as well it also shows us some interesting things about the dense versus MoE gap, in there Anyway, as always, I’ve put the links to the blog and the models

17:27 and even the trajectory viewer in the description there let me know in the comments what you think. I’m really curious to hear from you guys how many of you are actually doing sort of computer use or browser agents, things like that versus just sticking to basically tools, code, MCPs, APIs, that kind of thing I’ve covered some browser use agents in the past and I’ve certainly been pitched a bunch of them, but I haven’t always been convinced how actually useful they are over just using a normal model and a custom harness. Anyway, as always, if you found the video useful, click like and subscribe, and

18:00 I will talk to you in the next video. Bye for now