How to build your own agent harness?!

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: How to build your own agent harness?! · Channel: LowSpecLabs · 11:50 · Watch on YouTube

Topics: AI agents, Software engineering with AI, AI developer tools, Local and open-source models

Summary

The video explains how to build an agent harness around an API, emphasizing a simple, modular structure rather than one large script. The creator recommends keeping the core agent loop separate from tools, logging, configuration, knowledge-base functions, and other modules. A live walkthrough shows how to gather BookStack API documentation, plan a Python CLI harness with safety confirmations, and have an AI implement and test it. After manual testing reveals a connection error, the creator asks the model to improve its troubleshooting guidance, then demonstrates the harness creating a knowledge tree from the wiki.

Key points

A basic working harness can often be generated quickly with an LLM; decide how much additional time is worth spending on refinement.
Keep the main agent loop independent from tool handling, logging, approvals, configuration, and knowledge-base functions.
Avoid putting all agent logic in one giant script, which is harder for AI tools to parse, edit, and troubleshoot.
Use separate modules for functions such as goal tracking, file management, search, uploads, and logging.
Gather and save API documentation before asking an LLM to design a harness around it.
Ask the model questions during planning and implementation so you can catch questionable design choices early.
The demonstrated harness uses Python, an interactive CLI, and confirmation safeguards for destructive actions.
A fresh session can implement a saved plan with a cleaner context, while a local inference backend is available for testing.
Manual testing uncovered a BookStack connection error, prompting an update to explain the failure and guide users through configuration.

Chapters

Start	Chapter	Summary
00:00	What an agent harness is and how much to refine it	The creator introduces harnesses, sets expectations for MVP development, and recommends limiting refinement to a reasonable amount of time.
01:33	A modular harness structure	The video outlines the agent loop and shows why its tools, logs, configuration, and other functions should live in separate modules.
04:38	Planning a BookStack harness with AI	The creator gathers BookStack API documentation, asks an AI to propose a plan, and chooses a Python CLI with safeguards for destructive actions.
08:47	Implementing and testing the harness	The saved plan is implemented in a fresh session, tested, and revised after a manual run exposes a connection error.
11:20	Running the completed harness	After receiving connection guidance, the harness runs against the wiki and creates a knowledge tree of its pages.

Quotes

My recommendation is don’t spend more than a couple hours on it — 01:03

create one giant Python or bash script that you try to hold all the agent logic in. — 03:06

This gives you a chance for you to catch any weird design choices or weird implementation choices before the code’s complete. — 07:47

we’ll go with safe diff confirmation guard rails for page overwrite and delete actions. — 08:17

That is how you build your own agent harness. — 11:50

Concepts and entities

agent harness, BookStack, Google Antigravity, Gemini, LM Studio, llama.cpp, Python, MCP

Full transcript (original language)

00:00 How to build your own agent harness. Welcome back, boys. This is another episode of LowSpec Labs. In our last video, we covered the idea of wrapping your API or whatever tool or environment you want around the model and giving the model a limited set of tools in order to let local AI to help local AI perform better. Now, most modern LLMs can build you a Python or, you know, whatever preferred language you want harness

00:33 within a couple hours, right? It’ll get you a decent working MVP pretty quick. From there, the amount of time investment depends on how well you want the harness to work. If a couple hours of effort makes the model perform significantly better, in my mind, it’s worth it, right? an extra 10 to 12 hours of effort for maybe like another five to 10% performance gain isn’t worth it for me. I’ll just pay for Deep Seek. So, you have to decide how much time you want to take finetuning your harness. My

01:03 recommendation is don’t spend more than a couple hours on it, especially if it’s only supposed to be calling one API. This video is more or less to teach you the common pitfalls and mistakes I ran into while building various agent harnesses. Turns out there’s kind of a right and wrong way to build the harness. Who would have thought? Am I an expert? No. But I can tell you what not to do based off of my own mistakes. To my right is what you would consider your

01:33 standard agent harness loop. Right? You have your agent loop. Within that agent loop, you’ll have the harness or you have the agent and the agent will be used to plan and decomposition task. It’ll then move from there to call tools and then it’ll read the output of its tool and either make a further plan or go from there. Generally, you want this loop to be independent, right? You want the main loop to be different from your tool call handling, from your logging,

02:04 intelligation, from any knowledge base or MCP or whatever you set up, even from user approvals. You want the main agent loop to be isolated. If we were to look at the bookstack folder from our last video, please go ahead and check that out to stay in the no. But inside of script/bookstack, we actually have our chat, which is our main agent loop. We’ve got docs because uh I do pass docs into and out of book stack. That’s where I store them. We’ve got logs, so we can

02:35 log tool calls, chat, and history, and errors from the harness. We’ve got a requirements.ext text to set up our Python environment. We have a setup.sh and subscripts. And subscripts, that folder is what holds the rest of the modules for this harness, including tool calling, user requests, the configuration, processing, the knowledge base, and the logging and tele. Right? So, basically, you want to structure your folder as simple as possible. What

03:06 you do not want to do, and this is from my own failures, is create one giant Python or bash script that you try to hold all the agent logic in. That is going to make it very hard for whatever AI you’re using to parse and edit and troubleshoot it. By going with the structure I showed you, like for instance, let’s enter that folder for subscripts. If we were to do a ls/subscripts, you’ll see here we have a goal tracker to track our goal. File manager, create

03:37 docker, search, build, scratch, upload. These are all the different subscripts that add additional functionality and features to the harness. Note how they’re kept separated, at least logically, inside of this folder structure. And then we have the main chat.py at the top level. The end result is that you have a script which contains the main core of your agent loop and all the additional stuff like tool calling, your knowledge base, uh how it processes

04:08 your goals, your configuration and settings, logging in and telemetry are all kept in separate scripts so that you can modify and tweak them independently without you having to break your entire loop. If you put everything in one giant script, you can still do that. is just going to be very difficult and eventually you’ll enter a situation where you want to break out that monolithic script. So, how do you actually build your agent harness? Well, you’re going to need AI or you’re going to need to know how to code. I am going

04:38 with AI because AI is a much better coder and a much faster coder than I am. The wild part about this is you don’t even need the smartest AI to build out the agent harness. I’m pretty sure documentation has been out there for more than enough time now that most AIs trained on the general search area of the internet itself will be able to knock this out no problem. Claude, GPT, Deepseek, Kimmy, all of these can get

05:09 you your agent harness built in about 30 minutes to an hour. At least your MVP, you know, your bare working product. From there, you might want to take a couple hours to tweak it and refine it, or maybe a couple days if you get hooked. For the purposes of this video, we’ll be using AGI since Google basically gives away Gemini and French for free nowadays. So, let’s do clear. As you can see, we have anti-gravity pulled up here, the CLI version, which is honestly the only version of it I would recommend using. Then we’re going

05:40 to create a new folder and we’re going to walk through recreating the book stack harness that I demonstrated in the last video. First, you want to make sure that you have your documentation. Then we’re going to CD into our folder. Let’s do cd slash testbook stack. Then we’re going to start our agent harness, which in our case is going to be agy or Google’s anti-gravity. I know I’m using anti-gravity. That’s wild. It’s okay. it uh it works just fine. I would prefer

06:11 Opus or Chad GBT models but I am out of tokens. So the ADI So AGI and Google I go they give away inference for basically free. So we’ll ask it to search the web for information about the bookstack API. Then we’ll ask it to save what it found to a markdown file. And that is the first pass. Before we even have it write the harness, we get information about the API and we save it

06:45 locally. And that took about 60 seconds. This is one of the things I like about Gemini 3.7 and 3.8 Flash. It is a fast model. But it did the web search. It created api.md. Let’s go back and then let’s cd2 slash testbookstack ls. And we do have api.md. If we were to do a quick catap.md, you can see up here we have all the contents of the bookstack or the

07:17 bookstack API, right? It gives us lots of useful information. This is going to be necessary because now we’re going to use that API spec to propose a plan. You don’t have to use this exact prompt, but the general drift is you want to get your API spec and then ask your smarter LLM to create an agent harness that wraps around that API spec. Something I also like to do when I’m having AI build stuff is to stop and ask me questions at

07:47 least during the planning phase and oftentimes during the implementation phase. This gives you a chance for you to catch any weird design choices or weird implementation choices before the code’s complete. We’ll just walk through the options for this. For this, uh, Python’s good enough. We don’t need TypeScript or Node. Which interface surface should the agent harness expose? For our interfaces, we’ll go with an interactive CLI/Rebel.

08:17 How should the tools be abstracted for LM usage? We’ll go with the recommendation and which reliability and safety harness feature should we build in and we’ll go with safe diff confirmation guard rails for page overwrite and delete actions. Right? So we want confirmations of certain destructive tool calls. There we go. The cool thing about Google anti-gravity is it’s very quick. Like look at that. All right. It’s already got the plan ready to go. Bookstack

08:47 agent harness architecture and implementation plan. >> So now it’s asking if we want to implement that plan. We’re not going to implement it yet. Save that to working as plan MD. We’re instead going to save it as plan MD. Start a new session and then have the model start with a fresh context window on implementing our plan. We could also do that just by telling anti-gravity to launch a sub agent, but I like to keep things separated in different conversations because it helps

09:18 keep things a little bit more organized. I went ahead and started a new session. We’re going to tell it to read plan.md and implement it. Use this lm studio backend and model to test during implementation. So if you can have LM Studio kind of ready and waiting to go in the background or llama CPP or onslaught or whatever inference engine you’re using and then we’re also going to give it the

09:48 model and then we just hit enter and we let it run. Anti-gravity is incredibly fast. I’m probably going to give this like 15 to 20 minutes. All right, it’s very good. It’s about 15 minutes later and as you can see it was able to code up the project and launch the execution environment and start launching some tests. If we were to spin down clear and list the contents of this folder you’d see our bookstack harness folder our projecttoml our tests and then our api.md. What we

10:19 can do is make dur docs and we’ll move our docs after the ais has finished its implementation. Let’s wait for it to finish. All right, and it’s done. I went ahead and I tried one of the commands it recommended and it errored out. So, let’s see what the error was. Let’s see. Network error after three at all networks. So, it looks like we have to configure our end, right? Or our environment file so that we can actually connect to the bookstack backend. But that’s good because that means it baked

10:50 in some error correcting and troubleshooting. What we’re going to do is just copy this request and we’re going to rest steer the model and we’re going to tell it the prompt we told it to rest steer is upon manual testing script has errors when connecting to a bookstack backend. The next scenario should report why connection failed unable to connect to a Bookstack server and guide users on how to configure connection. Right? So even though we gave it a plan, the plan did not include that option. So we’re just going to have it added. Right? Uh we’ll see and then

11:20 we’ll continue from there. All right. And we are back with that error message and some guidance about where the Bookstack wiki is actually hosted. It was able to get everything spun up. All right. On my LM Studio instance, we’re waiting for a response back. Let’s see here. Live query autonomous agent. Yep. Now, let’s run Bookstack agent again. And let’s see what it does with tree. And look at that. It’s created a knowledge tree of all our different pages within our Bookstack wiki. It

11:50 gives us a pretty neat layout. As you can see, I have some Portainer information, some Windows information about different virtual machines. But that’s it, boys. That is how you build your own agent harness. I hope you found this video useful.