This 8B Agent Fixes Its Own Mistakes? I Tested It

Author: Prompt Engineer 48 · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: This 8B Agent Fixes Its Own Mistakes? I Tested It · Channel: Prompt Engineer 48 · 06:21 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, Local and open-source models

Summary

The video tests Agent Reflex 8B, a model trained to recover from tool failures, against a quantized Qwen-38B on a local laptop. Across six support-bot scenarios, both models handle clean calls, timeouts, permission errors, and several other failures equally well; Agent Reflex performs better when tool output is malformed or empty. However, both models sometimes invent order details, and the test is limited to one toy task and a small number of runs. The reviewer concludes that Agent Reflex is worth trying with flaky tools, while explicit error handling may help when switching models is not an option.

Key points

The paper reports an increase from 40.2 to 58.6 across five long-horizon tool-use benchmarks, with gains varying by benchmark.
Agent Reflex is trained on failure examples turned into 1,170 reflection skills, using supervised fine-tuning and ORPO.
The local comparison used an 8 GB RTX 4060 laptop GPU and community 4-bit quantizations of both models.
The test used a support bot that had to look up the status and total for an order, with six failure scenarios and 10 runs per model per scenario.
Both models scored 10 out of 10 on clean calls, timeouts, permission errors, and the other scenarios except malformed output.
On the initial malformed-output test, Qwen-38B scored 1 out of 10 and Agent Reflex scored 8 out of 10; overall, the results were 51 out of 60 and 58 out of 60.
In expanded malformed-output tests, Agent Reflex also did better on cut-off JSON and empty replies, while both models handled an HTML bad-gateway page and all-null fields.
Both models made up order details in some runs, so better recovery did not mean the model was immune to hallucinations.
The reviewer cautions that the results come from one toy task, community quantizations, and only 10 to 20 runs per test.

Chapters

Start	Chapter	Summary
00:01	Agent Reflex and the paper’s claims	The reviewer introduces the model, its reported benchmark improvement, and the plan to test recovery from tool failures locally.
01:04	Training and benchmark results	The video outlines the training recipe, reviews the reported benchmark scores, and describes the local hardware and task setup.
02:37	Six tool-failure scenarios	The reviewer explains the six test cases and shows both models recovering from a permission error by switching to a backup tool.
03:42	Results and fabricated details	Both models perform equally on most initial scenarios, while Agent Reflex does better on malformed output but still invents an order total.
04:44	Expanded tests and conclusions	Further tests show an advantage for Agent Reflex on cut-off JSON and empty replies, but the reviewer stresses the limited scope and recommends trying it with flaky tools.

Quotes

I broke the tools on purpose six different ways, 10 runs each. — 00:01

First, supervised fine-tuning on about 54,000 trajectories, then a preference step they call ORPO. — 01:04

This is the only place on my test where the recovery training shows up clearly. — 04:13

So, it recovers more often, but it’s not immune to making things up. — 05:17

So, treat it as a good start, not a verdict. — 05:49

Concepts and entities

Agent Reflex 8B, Qwen 38B, ORPO, reflection skills, Hugging Face, RUC NLPIR, vLLM, RTX 4060, Agent Noise Bench

Full transcript (original language)

00:01 This paper says an 8-billion parameter model jumped from 40.2 to 58.6 on tool use benchmarks just by learning to recover from mistakes. I did not believe that would show up on my laptop, so I broke the tools on purpose six different ways, 10 runs each. In five of the six, it made no difference at all. In one, the original model got one out of 10 and this one got eight. And stay till the end because one of its wrong answers made up a number. All right, this is Agent Reflex 8B on Hugging Face and you can see it says

00:32 base model Qwen-38B license MIT there on the GitHub repo from the RUC NLPIR group. It has about 75 stars, so this is a very fresh release. The paper went up on the 13th of October and the weights were updated the day after. Now, the idea is simple. Most agents are trained where the tools always work. In real life, a tool times out, returns half a response, or the user says, “Wait, I meant something else.” So, the authors take failures like these and turn them into 1,170 reflection

01:04 skills, then train in two steps. First, supervised fine-tuning on about 54,000 trajectories, then a preference step they call ORPO. Some summaries call that second step reinforcement learning, but the repo’s own training recipe lists ORPO, so that’s the word I’ll use. Now, let’s look at the numbers on the repo itself. Over the five long horizon tool use benchmarks, Qwen-38B scores 40.2 and Agent Reflex scores 58.6, so that’s 18.4 points. But if you look here, most of that comes

01:35 from one column. Tau square bench goes from 26.2 to 62.1. A’s bench goes from 54 to 78. And Vita bench, the hardest one, only goes from 11.4 to 17.5. On Agent Noise Bench, which injects failures on purpose, it’s 12.3 to 29.1. So, the average heights that the gain is uneven. These are the authors’ numbers on their own setup with vllm, not mine. So, here’s what I’m running. This is the RTX 4060 laptop GPU with 8 GB and

02:06 everything here is local. The authors don’t publish ggufs themselves, so I’m using the community quant from Reader Marker, the Q4KM, which is 5 GB. Next to it, the original quant 38B from Olamau, also 4-bit, also 5.2 GB. Same settings for both, temperature 0.68k context. The task is small on purpose. The agent is a support bot with a lookup order tool and a backup tool, and the user asks for the status and total of order 4821. The right answer is shipped

02:37 6340 and I scored task completion, not how nice the explanation sounds. If the status or the total is wrong, it’s a fail. Here are the six scenarios. One, clean, the tool just works. Two, timeout, the first call returns a timeout error, the second works. Three, malformed, the first call returns this agson that stops in the middle of the word ship. Four, permission, the main tool returns 403 three times, so the only way out is the backup tool. Five, always down, both tools return 503 forever, so the correct

03:10 move is to say I could not get it. Six, user correction, it answers and then I say sorry, I meant order 4812. 10 runs of each for each model. That’s 120 episodes. Let’s go ahead and watch one. This is the permission case. You can see it calls lookup order, gets the 403, and on the second call, it switches to the backup tool and gets the answer. And if you look on the right, quant 38B does exactly the same. That’s the thing I did not expect. The original model already knows to retry on a timeout and to

03:42 switch tools on a 403. Each of these takes about 11 seconds on this card. Here are the results. Clean, 10 out of 10 for both. Timeout, 10 out of 10 for both, permission, 10 for both, always down, 10 for both. And mind you that I only check that it doesn’t invent order details, use a correction, 10 for both. And then, malformed Qwen 38B got one out of 10. Agent Reflex got eight out of 10. Overall, that’s 58 of 60 against 51 of 60.

04:13 Now, look at what Qwen 38B does with a cut-off JSON. It reads shipped as shipped, says the status is shipped, and just stops after one call. It never tries again. Agent Reflex made 1.8 calls on average in that scenario, so it does retry most of the time. This is the only place on my test where the recovery training shows up clearly. Now, the two misses. In one of them, Agent Reflex said the status is shipped and the total is 8919. That number is not in any tool response. It made it up. Now, 10 runs is thin, so I rebuild that

04:44 malformed case into four versions and ran 20 each. Each the cut-off JSON again, Qwen 38B four out of 20, Agent Reflex 18 out of 20. Then, an HTML bad gateway page, both models 20 out of 20, and a reply where every field is null, both 20 out of 20. So, when the tool says plainly that something broke, the original model already copes. The last one is an empty reply, nothing at all. Qwen 38B got zero out of 20. Agent Reflex got eight out of 20.

05:17 Look at what the misses say. In one run, Qwen 38B told me the order was shipped, total $125, and added that the backup service was used. It made one call. There was no backup call. And Agent Reflex 12 times out of 20 invented a status and a total as well, like delivered 34.50. So, it recovers more often, but it’s not immune to making things up. And a caveat on my side, one toy task, a four-bit community quant, 10 to 20 runs each. This tells you how it handles broken

05:49 tool output, not how it does on the paper’s benchmarks. So, treat it as a good start, not a verdict. So, here’s what I take from that. The loud failures, a bad gateway page, or null fields, both models handled. The quiet ones, a half sentence of JSON, or an empty reply, are where they break, and Agent Reflex breaks less. If you can’t switch models, a wrapper that turns a failed parse or an empty reply into an explicit error is the cheap fix. I haven’t tested that wrapper, but it is the case both models already handled.

06:21 So, who should use it? If your agent talks to flaky tools that return broken output, it’s worth trying this Q4 on your own tools, since it was the only one that kept going. If your tools mostly time out or return clean errors, Qwen 38B already handles that on my test. If you’re looking for cheap and reliable GPUs, I use RunPod, and the link is in the description. The bench script is in the description, too, so you can run it on your system, and let me know how it goes. Next video, I want to push this with real MCP servers. Check out the next one.