Harness Arena (Fully Tested): This NEW Benchmark TESTED Every AGENT HARNESS (which is the best?)

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Harness Arena (Fully Tested): This NEW Benchmark TESTED Every AGENT HARNESS (which is the best?) · Channel: AICodeKing · 05:54 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, AI developer tools

Summary

The video walks through Harness Arena, a platform that compares AI agent harnesses by giving them the same task and model configuration in separate workspaces. It demonstrates how to browse recorded runs, anonymously judge outputs, and explore leaderboard rankings. It also shows how to configure a new comparison by choosing tasks, harnesses, and a model, then follow its progress and evaluate completed work. The presenter cautions that early rankings may be based on few votes and notes that the public battle log and leaderboard are accessible without an account.

Key points

A harness includes the software around a model, such as its instructions, tools, and execution environment.
Harness Arena runs multiple harnesses on the same task and model configuration, with separate workspaces.
The battle log lets users filter recorded runs and find rounds awaiting judgment.
Evaluators inspect anonymous outputs and score every required output from 1 to 10; harness names are revealed after submission.
Evaluation should check both the task requirements and whether the application still behaves as expected.
The leaderboard includes ratings, win rates, vote counts, wins and losses, and median completion times.
Rankings with few votes should be treated cautiously, and speed alone is not a measure of output quality.
Users can create comparisons with their own data set or existing tasks, but need at least two harnesses.
Completed tasks can be judged as they become available, without waiting for the entire data set to finish.
The back end is available on GitHub under the MIT license; self-hosting can still involve model and hosting costs.

Chapters

Start	Chapter	Summary
00:02	Introducing Harness Arena and the battle log	The presenter explains how the platform compares harnesses and demonstrates finding a recorded coding round to review.
01:37	Judging outputs and reading the leaderboard	The walkthrough covers anonymous scoring against the task rubric and explains how to interpret leaderboard metrics and limited votes.
03:48	Setting up a new comparison	The presenter shows how to select tasks, harnesses, and a model, and how to submit a benchmark and follow its progress.
05:23	Evaluating completed tasks and closing notes	The video explains where to access evaluation and setup, notes the back end’s MIT license and possible costs, and closes with a channel sign-off.

Quotes

A harness is the software around the model, including its instructions, tools, and execution environment. — 00:02

The harness names are revealed after you submit all the scores. — 02:09

A fast run that leaves you with broken code still needs work, so speed alone wouldn’t decide my choice. — 03:15

You need at least two harnesses to run a comparison. — 04:20

The back end is available on GitHub under the MIT license, and running it yourself can still involve model and hosting costs. — 05:23

Concepts and entities

Harness Arena, Harness, On Demand, Codex, Hermes, Claude Code, Qwen3.8, GitHub

Full transcript (original language)

00:02 [music] >> Hi. Welcome to another video. So, today we’re trying Harness Arena, a platform for comparing AI agent harnesses using the work they actually produce. You can inspect recorded runs, judge outputs without seeing which agent made them, and compare the results on a leaderboard. This is a sponsored look at Harness Arena. The platform is sponsored by On Demand, which also has a harness competing in the arena. A harness is the software around the model, including its instructions, tools, and execution environment. Harness Arena’s comparison method gives multiple harnesses the same

00:35 task and model configuration with separate workspaces for each one. For example, an agent needs to read the relevant code, make an edit, and check whether it worked. How the harness handles those steps can affect the result even when the model is the same. That’s what makes this comparison interesting. So, let’s open harness-arena.ai. I’m starting with battle log in the sidebar. This shows recorded runs, and you can filter them by status, category, or outcome. The status menu includes runs that are queued, in progress, or awaiting judgment. If you want something to evaluate, look for a round awaiting judgment. That gives you a starting

01:07 point without having to launch a new benchmark yourself. For this walkthrough, I’m opening category and selecting coding and software build. The task listed here is called 3D globe application, targeted overlay removal. It’s a web app task. And this recorded round lists Codex, On Demand, Hermes, Open Code, Open Claw, and Claude Code. Now, click the arrow on the right to show the comparison. This expands a table with one column for each harness. The model row shows the Qwen 3.8 configuration used for this round. You can also see rows for your score, community ratings, and deliverables. In

01:37 this particular round, the community rating says no votes yet, and the deliverable counts are hidden until judged. So, there’s no winner to call here yet. This is a useful place to check which tools ran together and what configuration they used. Next, click the task title to open the evaluation. This step requires an account. If you’re signed out, you’ll get a sign-in dialogue, so sign in before you start reviewing. The site’s methodology page explains the judging sequence. Read the task and its rubric, inspect the anonymous outputs, and score every required output from 1 to 10. The

02:09 harness names are revealed after you submit all the scores. For this globe task, I would first check the exact overlay removal requirement in the prompt, then compare the submitted apps against it. Does the requested overlay disappear and does the globe still work? Those are the kinds of checks to make before choosing a score. I would also check whether the agent changed anything unrelated. If rotation or zoom is part of the original app, I would try those controls after the edit. Removing the right overlay is only useful if the rest of the application still behaves as expected. The key is to inspect the files and behavior the task asks for.

02:41 Give each output the same checks. Then submit your evaluation when you’ve reviewed all of them. I’d keep a short checklist based on the rubric while doing this. It makes it easier to score consistently when you’re reviewing several outputs, especially if one looks more polished but misses a requirement. Now, let’s open leaderboard from the sidebar and click code. That switches the ranking to coding tasks. The table shows the rating, win rate, votes, wins and losses, and median completion time. You can switch to research operations or another group depending on what you use agents for. The rating summarizes the judgments, while completion time tells you how long

03:15 the runs took. I would look at both. A fast run that leaves you with broken code still needs work, so speed alone wouldn’t decide my choice. On the coding view I’m looking at, several harnesses have the same displayed rating and each has only one vote listed. So, take this early ranking with a grain of salt. Check the amount of judging behind a score before treating a small difference as meaningful. If you want to submit a task yourself, there’s a new benchmark option in the sidebar. That also requires signing in. So, once you’re signed in, this is where you can set up your own

03:48 comparison. You choose the tasks, the harnesses you want to test, and the model they’ll use. At the top, there’s a link to artificial analysis for creating a data set. You can also download the Excel template, put one task on each row, and upload your own data set. So, if you have specific coding tasks you regularly work on, you can bring those into the comparison. There are already tasks loaded here as well, which makes it easier to get started. Now, under choose harnesses, select the tools you want to compare. You could pick Claude Code, Codex CLI,

04:20 and OpenCode, for example. You need at least two harnesses to run a comparison. If you want to include on-demand, you’ll need to add your on-demand API key and set up first. The skills section also requires that key. Further down, you can filter the tasks by category. Let’s say we’re interested in coding. Click code, then use view task to read the instructions before selecting a task. For the globe example we looked at earlier, this opens the full prompt explaining what needs to be removed and what must stay unchanged. Check any reference files the task needs as well. Then, under model for this benchmark, choose the model for the comparison.

04:53 Right now, I can see Kimmy K3, GLM 5.3 Flash, and Quen 3.8. Those model options are currently marked free in the interface. But, before submitting, check the counter showing how many tasks and harnesses you’ve selected. Personally, I’d start with one task and two harnesses, so the outputs are easy to compare. Once everything is ready, click submit for evaluation. You can then use view progress to follow the run. Each finished task becomes available for judging through evaluate or the battle log. So, you can start reviewing

05:23 completed tasks without waiting for the entire data set to finish. Then, check the outputs and score them using the process we covered earlier. Setup is where the site puts model profiles and harness settings. The public leaderboard and battle log are accessible without an account. The back end is available on GitHub under the MIT license, and running it yourself can still involve model and hosting costs. Overall, it’s pretty cool. Anyway, [music] let me know your thoughts in the comments. If you like this video, consider donating through the Super Thanks option or becoming a member [music] by clicking the join button. Also, give this video a thumbs

05:54 up and subscribe to my channel. I’ll see you in the next one. Until then, bye. >> [music] [music and bell]