How to Build Your Own AI Agent (Like PewDiePie)

Author: Hugging Face · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: How to Build Your Own AI Agent (Like PewDiePie) · Channel: Hugging Face · 23:46 · Watch on YouTube

Topics: AI agents, AI models, Local and open-source models

Summary

The video explains PewDiePie’s Odysseus AI agent harness and shows how an agent uses a language model, context, and tools in a repeated loop. It distinguishes running an application locally from running the model locally, and outlines the privacy, cost, and hardware considerations of local models. The presenter then explains how a smaller model can be fine-tuned for tool use using conversation traces, and discusses possible sources of those traces. The video closes with pointers to Hugging Face resources and tutorials for building and fine-tuning an agent yourself.

Key points

Odysseus is an LLM-agnostic agent harness with support for models, tools, memories, skills, and other features.
An agent sends a user request and context to a language model, executes any tool calls the model requests, and returns the results to the model.
An agent harness manages the system prompt, tool definitions, relevant memories, files, and conversation transcript.
An application can run locally while sending requests to a cloud model; a locally running model keeps the data on the user’s hardware.
Local models can offer more privacy, offline use, and control, but require hardware and electricity and are not necessarily free to run.
Hugging Face and llama.cpp can help users find and run models compatible with their hardware, including quantized versions.
A smaller model that struggles with tool calling can be fine-tuned for a specific agent harness using suitable conversation traces.
The presenter suggests using public agent traces or traces generated with another model, and stresses removing credentials before sharing traces.

Chapters

Start	Chapter	Summary
00:00	Introducing Odysseus	The presenter previews the video and tours Odysseus, its models, tools, memories, skills, and interface.
04:18	How an agent harness works	The video explains the agent loop, tool calls, and the context an application provides to a language model.
10:06	Running models locally	The presenter distinguishes a locally hosted application from a locally running model and explains the reasons to use local inference.
12:47	Finding models for your hardware	The video shows how to explore models and use hardware compatibility information and llama.cpp to choose and run one.
17:04	Fine-tuning a model for tool use	The presenter discusses why a smaller model may need fine-tuning and how conversation traces can provide training data.
21:37	Training resources and wrap-up	The video points viewers to a fine-tuning masterclass, recaps possible training approaches, and closes with advice on sharing anonymized traces.

Quotes

an agent is essentially just a loop or two calls to a language model. — 04:18

On local, however, that LLM, the language model, the model itself is running in your machine. — 11:43

running a local model does not mean that it is free — 12:15

You can train them on your specific tasks and make them even better — 19:16

Concepts and entities

Odysseus, PewDiePie, Hugging Face, Qwen3.5 9B, Mistral Large 4, llama.cpp, OpenRouter, Docker, MCP, Unsloth, DeepSeek, GLM

Full transcript (original language)

00:00 Good morning everyone, how’s it going? So PewDiePie is doing AI now and I figure that this is a great moment and a great opportunity to explain to you exactly what he is building, how he’s doing it and how you can do the same thing. Because I figure that a lot of you might be inspired by the things that he’s building and some of them sound a little bit too complicated and too complex for you to actually take on. But they’re actually very approachable if you start from the beginning. So today I’m going to give you a very quick overview of everything that PewDiePie is building and how you can build yourself something similar in your own home.

00:35 So what we’re going to be doing today is we’re going to be taking a look at what is Odysseus, the agent harness that PewDiePie is building. And then we’re going to take a look at what even is an agent harness and how these things work from scratch and how you can make your own. After that, we’re going to be taking a look at local AI because he’s trying to run a local model in Odysseus and it’s not very straightforward apparently. So what he’s doing is he’s trying to fine tune or to train a Qwen 3.5 9 billion parameter model to run locally and to work correctly with Odysseus.

01:06 And I’m going to be telling you a little bit more about how you can do that yourself, how you can train a small model to run specifically for your own harness. So yeah, without any further ado, let’s actually get right into it. So first, let’s take a very quick look at what is Odysseus. So it’s basically some sort of a ChatGPT, but it is LLM agnostic, which means that it can run with any LLM. It has supports for other tools through MCP servers and it has a bunch of other tools that I’m going to show you in just a bit. And in order to install it, it’s a bit technical.

01:38 So you have to clone the repository from GitHub and you have to mount the Docker container. And then once that is running, it will be running. Everything will be running inside the Docker container in your machine. So once that is done and everything is running correctly, you will see a page that looks a little bit like this. I’m just going to zoom in a little bit. So this is Odysseus. This is its landing page. And right here, you can select your model. You can add new models right here. I have myself already added my authentication with OpenRouter.

02:09 And I have downloaded a couple of local models also to test this. And yeah, so basically just select which subscriber, which subscription or which provider you want and then log in. The ChatGPT subscription I think is not running correctly right now. It just doesn’t work. But I think the other ones do. At least OpenRouter does. And llama.cpp does as well. Here I have the loader. Here I have the added models that I have already. So I have, as you can see, a Qwen model. And I have my OpenRouter integration. Some other stuff that you have right here.

02:40 Here’s your list of chats. You can search the chats. You have a bunch of tools, actually. You have an email client in here, apparently. You have a brain, which automatically extracts memories from your interactions with it. It also automatically extracts skills from your interactions. Kind of like a Hermes or OpenClaw kind of way. And you can also add memories or skills manually. You can compare different models outputs. You have a nice little cookbook right here, which is supposed to scan your hardware and tell you which local models are best for running in your machine.

03:13 We have some sort of a filtering problem right here. Because, as you can see, EmbeddingGemma is listed as the top recommendation for my hardware. Which is, of course, not a model that would power a chat application. Consistence is an embedding model. But a really nice idea. I really much like it. You have, yeah, a bunch of different things right here. You have deep research. You have gallery. You have some sort of a library of documents and chats and research. Notes. Tasks. And you can also change the theme. So it’s pretty cool. I’m not a fan of the window, this window, like, layout system.

03:46 But there you go. Let me just show you now how it works with the model. So I’m selecting Mistral Large, just for the demo. Large 4. And let me just ask it, hello, please introduce yourself. And let’s take a look at it. So hello, I’m your AI assistant here to help with a wide range of tasks, blah, blah, blah. And something funny that it somehow searches the web on every single interaction, even when I just say hello. So it’s searched for hello by Adele and latest news from Hello magazine. Why not? So there you go. Yeah, everything deserves very good documented sources.

04:18 So that is what Odysseus is. But how did he even build this? Let’s take a look at how you can build your own harness and what even is an agent harness. Okay, so let’s get right into that. All right, so let’s go very quickly over what an agent harness or an agent in general is. The first thing to know is that an agent is essentially just a loop or two calls to a language model. And it works like this, like you see right here on the left. And by the way, this is a tldraw diagram developed, generated using AI.

04:49 So you can totally just build your own diagrams using ChatGPT or Claude to generate these diagrams. All right, so what happens is that you send an input to your agent, right? And an agent is the software that wraps around your large language model. So that input is sent to your language model and the language model reads all your context, all the system prompt and the definitions of the tools, etc. that are sent to it. And it decides how it’s going to answer your question. If the question is something that can be answered directly with text, then it might just spit out the answer.

05:21 If, on the other hand, it requires some additional tools, so for example, search the internet, it requires to run a program or call an external service, then it will call what we call a tool. Tools are these programs or functions that are available and they can do a bunch of different things, like basically anything that you can do with code, you can put it inside a tool call. And then, once the agent returns a tool call, your application, in this case it would be Odysseus,

05:53 but it could be ChatGPT, Claude, or your own harness in this case, will execute that tool call and will return the results to your model. And then your model, after seeing the results of the tool that it called, it will be able to reason again and figure out if it needs to call another tool or if it can finally give you an answer. So that’s essentially the diagram that you have right here. And when I told you that the model receives a lot of context on the first call that you, I mean, on every call that you send to it, what I mean is this right here.

06:23 Usually, the context or all of the material that is sent to a language model is defined by the harness that is built around it. In this case, it would be Odysseus, but different applications have different ways to manage the context. Usually, what you have in the usual context of a coding agent or a general purpose agent would be a system prompt that would tell the agent or describe what its initial goals or purpose are. ChatGPT has a system prompt.

06:54 Claude has a system prompt. By the way, in case you’re not familiar with it, actually, the prompts for Claude are open sourced in the Claude website. Feel free to take a look at them. They’ll be linked in the description as well if you’re interested. And after that, you have the tool definitions. In other words, the list of all the tools available so that the agent knows what tools to call. And that’s essentially what dictates what happens right here. So the model will read its tool definitions and from there decide whether or not it requires to make a tool call or not.

07:24 Sometimes also, inside the context window, we have the memory and the files available also in that session. Memories are usually memories extracted from previous conversations or which come from other resources that the user may have shared with the assistant. And those, by the way, are extracted in different ways. Different services do it differently. There are other videos right here on the channel where we go over memory and how it works. But just know that many different agents do this in a different way. And, of course, they are not appended to the system prompt every single time

07:57 because every interaction requires different kinds of memories, right? So the agent remembers something relevant to the conversation that it is having. And after that, usually there is the transcript, which essentially means all of the entire list of all the messages that have been sent back and forth to the language model. As you can see right here, the first one might be something like fix the failing test from the user. Then the assistant just sends out a reasoning message first. Then it makes a tool call, as we mentioned before. Read a file, for example. And then from there, it gets the tool result.

08:29 Then maybe we’ll do another tool call. Then it gets a tool result. And then finally, it sends over the final answer. And all of this, every single square that you see right here is actually a message. And all of them are persisted and sent over to the language model on every interaction so that it has the entire context of the conversation that it is having, okay? So essentially, that is how an AI agent works. And the wrapper around this minimalist explanation is the application that coordinates all these calls.

09:00 So it coordinates how the user sends the first request. In this case, for example, in Odysseus, it would be this nice little input right here. But you may also have a voice input, for example, for other agents. And you have also the application that is capable of defining how the tools interact with the language model. So the language model can, of course, interact with the tools and update them. But in the case of Odysseus, and I think that that is actually a very interesting way to build this, you can also interact with the tools manually. So for example, in this thing, if it’s the calendar, you can create events yourself.

09:33 So I think that this is a very interesting way to see how tools can actually be their own applications. And this is something that we’re starting to see more and more with MCP servers that allow you to connect to existing applications such as Notion, Airtable, etc. So the agent can modify and run actions on your applications. But the applications still have a user interface that you can use yourself. So really interesting stuff right here. And that is essentially how an AI agent works. But now let’s talk about how the model is chosen.

10:06 Because in this case, of course, the demo I showed you, I was using Mistral Large. It’s a very large language model and very powerful. But if you wanted to run that in your own machine, then that model would be too large. And that’s essentially what PewDiePie was trying to do. He was trying to build an assistant and also the local model to run the assistant with it. So let’s take a look at how you can run these kinds of applications with your own local models. All right. So let’s take a look at how to run these AI models locally and what it actually means to run a model locally.

10:38 Because, I mean, again, this is really for beginners. But when PewDiePie says that Odysseus is running locally, it actually is running locally in the sense that it is running in a Docker container inside of your computer. So if you share this URL with someone else that is not connected to your computer, they will not be able to see this Odysseus application because it’s running locally. That does not mean that the model is running locally itself. And that is the difference right here. So the application usually lives in your machine. And whenever you call the language model, in the example I showed you, I called Mistral Large 4, which is one of the latest AI models recently released.

11:13 And your data leaves your application and goes to whichever inference provider I was using in that case. I was going through OpenRouter and that was routing to the official Mistral API, if I remember correctly. So that data left your application, went into the internet, went into the service that ran the language language model, and then returned to you the response. And that’s essentially what happens on every single iteration of the agent loop, right?

11:43 On local, however, that LLM, the language model, the model itself is running in your machine. So the data never leaves your hardware. And that’s why a lot of people prefer local. Because even though companies, and especially European companies, protect your data and take care of your data, sometimes they even encrypt it so that not even them, they can look at your data, at your messages that you send your application. The data still leaves your computer. And for some people, that is a problem. And there are, of course, many different reasons to go for local.

12:15 For example, it becomes, as I mentioned, it makes everything more private. It also works offline because everything is running in your machine. There is no bill per token. However, do keep in mind that running a local model does not mean that it is free because you’re still running it yourself. And you’re paying for the electricity that comes with running a model, which is increased cost. But it does give you more control over the model that you want. You can even customize it and fine-tune it, as we’re going to see in just a moment. And it is actually even pretty fast for small and very self-contained jobs.

12:47 Okay. And the way to do that, and that’s why PewDiePie mentions Hugging Face so much, is because here is the largest repository of open models for you to take a look at and download. And in this case, for example, we’re taking a look at models that would power an AI assistant. And in that case, you can go right here to models, you go to filter by text generation, and you can even filter them by their size. I’m going to tell you a little bit more about their size. But in general, depending on your machine, you should be able to run models that are about probably smaller than 24 billion parameters, probably smaller than 32.

13:19 It really depends on your hardware. I’m going to tell you a little bit more about this either in this video or another one. But here you have the list of all of the recent models that have been available. There is, for example, Kolibri 1, which is a recent model developed in Germany, which are, for example, Qwen 3.8 27B. That should run in most consumer hardware, at least if you run it with some quantization. And that is essentially what is going on right here. So what happens is that he goes to Hugging Face, he selects a model that is useful for him, and he downloads it and runs it locally. There is actually a very straightforward way to run your models locally if you go to llama.app.

13:54 Llama is the engine that allows you to run a model in your machine. There are many, of course, available, but llama.app. Llama.cpp is my favorite one right here. And it is actually a nice little application that also helps you find the models that are adapted to your computer. So right here, you can just install it, you can download it, and it will install a nice little menu right here like this. And it will have a recommended for your computer section right here. So you can select the models that are the best for your particular machine.

14:26 And you can also see the models that you have already installed. And once you have downloaded this and it runs correctly, then you can run them either in Odysseus or any other AI agent that you’re running. In my case, I’m running here with Llama. And let’s suppose that I want to figure out which other models are available for me in my machine and which ones my machine can run. So I’m going to go right here to my profile. And in Hugging Face, I’m going to go right here to settings. I’m going to go right here to hardware. And what’s going to happen if you have not done this before is you’re going to have this empty.

14:57 But here you’re going to select which computer you have. In my case, it’s an Apple M4 Max with 36 gigabytes. Let’s take a look at it. And here I have my score. And this is especially important because now you can take a look at the models that you’re the most interested in. So for example, let’s suppose that you go right here to llama.app. Take a look at the most popular models that people run locally. And let’s suppose that you want to try Qwen 3.8. So what I would recommend is you click on it. You go to the Hugging Face page. And right here, you will see your hardware compatibility chart right here.

15:28 And here you will see the level of quantization that is the best for your particular machine. In this case, it recommends that I use Q4. And if this model is a bit too large for my machine, I can always go a little bit below. So for example, let’s suppose that I want something smaller. I can use Qwen 3.5 small. You can go right here. And as you can see, my machine can even run the BF16 version, which is the version that is not quantized, like the full version of the model. Right. And then you can select it and then you can run it on your own applications. If you are using Odysseus, then you’re going to go right here.

15:59 You’re going to go to cookbook and then you’re going to take a look at Qwen 3.5. And in this case, remember that the one that was suggested for me was 3.5 0.8 B. And this is the one that you would download and run. Okay. And if you’re running other agents, don’t hesitate to use those too. Okay. And there you go. That’s essentially what’s going on. And here’s the final diagram again. So you have the user request. Sends the message to your local model this time. Then your local model is able to either call a tool or not.

16:32 If it holds a tool, just like before, your computer runs the tool in your computer, then sends it back to your local model and then go back right here. If there is no other tool call, then it gives you the final answer. Okay. And this is the model that we swapped. And if you wanted a cloud model, you can also do that. But what PewDiePie is building right here is something local completely. All right. All right. So now that we understand a little bit better how local models work and how they run in your machine, let me tell you a little bit more about what PewDiePie did and how he is training his own model, because that’s what he says he’s doing right here with Ajax.

17:04 And we have a nice little hint right here. He says that Ajax is a fine-tuned Qwen 3.5 9B. Now, what is going on here? What does that even mean? And how can you do that yourself? So basically, as I showed you before, you can go right here to Hugging Face. You can find the model that you’re interested in. In this case, it’s Qwen 0. Nope, not this one. It is B. And this one is the version that he’s likely running right here from Unsloth. And as you can see right here, since I already added my hardware, every single quantization runs in my machine.

17:37 Even the BF16 version right here, which is the full model without any quantization. So all its capabilities at 100%. And this is probably why he chose this model, because it is capable of running in pretty much any machine on full BF16. Now, what I assume that happened is that he was running Odysseus, just like that, with Astra on the OpenAI API. And he figured that it was working great, that it was doing its job perfectly, and everything was working great. And he wanted to switch to a local model.

18:08 And when he switched to Qwen 3.5 9B, he realized that the agent actually stopped working reliably, and it was missing the tool calls. Remember, as I mentioned before, that when you send a question to your agent, it is supposed to select or not a tool. And Astra was doing that reliably, but the new one was probably not selecting the right tools, or maybe not even being able to call the tools. Now, what I would have done from there, and what was probably, and I don’t know if he actually tried it, was to actually refine the tool definitions that he had in his agent right here.

18:45 Because he could have probably made it easier for smaller models to catch on those tools. But what happened is that, essentially, he figured out that this small model was not good enough yet. So he figured that he can fine-tune it or train it for his specific harness. And that’s what he says right here. And this is important, because open models can be trained on top of the existing weights that are uploaded to Hugging Face. So that’s the fun part about open models. You can download them from Hugging Face.

19:16 You can train them on your specific tasks and make them even better, even if they’re small, like 9 billion parameters. But how do you do that, and how is he doing that? So in order to train a model like this, you’re going to need data. And in this case, since we’re trying to train a model to use tools correctly, then we’re going to need histories of conversations with the same agent where it runs the tools correctly. And I suppose that what was happening right here is that he saw that Astra was doing it perfectly. And he tried to use those transcripts, as you see right here, those existing traces of the agent to train the smaller model.

19:52 And he probably faced an issue when he figured out that OpenAI actually does not ship the entire generation of the language model to you over the API or even over the subscription. They actually encrypt the reasoning tokens. Because remember that an agent or a language model first reasons and thinks about the answer that it’s going to give you, and then it gives you the answer. And those reasoning tokens are very important if you’re fine-tuning a model. So what he figured is he tried to hack his way into extracting the contents of those reasoning traces.

20:26 He got banned from OpenAI twice, and that’s what we learned from this video. There were, however, easier ways to do it, probably. You could have probably used a non-OpenAI model, such as an open weights model, such as DeepSeek or GLM, and used those traces to train the smaller model. And that’s something that you could do. If, however, what you want to do is train a model for general tool calling, even if it’s smaller than 9B, for example, you could be fine-tuning a 0.8B parameter model, what you could do is go right here to Datasets and filter by only traces.

21:01 And here you will have a bunch of public traces of AI agents that people have uploaded to Hugging Face so that you can fine-tune your small models on these traces right here. And, by the way, if you are running your own agents and have your traces in your machine, you can totally upload them to Hugging Face. Either put them on public or on private if you just want to back them up. And if you put them on public, of course, without licking your credentials, be sure to filter out your credentials before you make it public, you will help other people train their small models so that they can run tool calling assistance from their machine.

21:37 Okay. Actually, what I want to show you is that there is a complete masterclass on fine-tuning agents right here in the YouTube channel. Let me show that to you in a moment. And it is right here in the Hugging Face channel. If you take a look at, I mean, the link is going to be in the description anyways, but it’s part of the post-training agents playlist that you see right here. And here you can see that you have five videos. If you want to do what I mentioned right here, you can start from the video number three, which is training agents. And shows you precisely what I told you that you could do right now, right here,

22:09 which is training small models with the traces from a bigger model. And that’s essentially what’s going on. So to recap, there are two things that he could have done. PewDiePie could have either fine-tuned his smaller model with the traces that are already available in the Hub right here, and following the tutorials that I just linked right here. Or he could have run his same Odysseus agent, and he could have gathered all the traces from the agent and used those to train his smaller model. And in order for the traces to have the full reasoning tokens,

22:40 he would have had to use another model, not necessarily OpenAI, because they encrypt the reasoning tokens, right? And if you are going to be using the traces from the Hugging Face Hub, you’re probably going to have to clean and format them before. This is, of course, mentioned and covered in the masterclass I mentioned. And then you just fine-tune it with whichever technique you select. That’s also covered in the masterclass I mentioned. And then you will have your complete agent harness made by yourself, or Odysseus, running with a model that you fine-tuned yourself. And that way you will be able to compare the before and after of your small model

23:15 before fine-tuning it and after fine-tuning it. That’s why it’s always a very good idea to keep those traces and to help the community by anonymizing them, removing the credentials, and putting them in your public repo so that other people can use them to also train small models. So there you go. That’s essentially what we have learned today. It’s very straightforward. And again, this is a very high-level overview of what PewDiePie was trying to build and what he did. I hope that I have at least given you the starting point or the directions to follow if you want to do this yourself.

23:46 If you want more in-depth tutorials on how to do these things, please do let me know. I can make more videos about this if you’re interested. Do definitely take a look at the masterclasses and the videos that I linked in the description and that I mentioned right here. They are made by some of the most amazing people and very well-respected experts in this field. So yeah, thanks a lot for watching. It has been a pleasure as usual and I’ll see you in the next one.