Stop Giving Your Agent Tools. Give It a Sandbox - Mikyo King, Arize AI

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Stop Giving Your Agent Tools. Give It a Sandbox - Mikyo King, Arize AI · Channel: AI Engineer · 14:02 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks

Summary

Mikail King explains Code Mode as an alternative to giving agents a large catalog of tools: agents can discover tools and use a programming environment to compose and execute them. This approach can reduce context bloat, token use, cost, and the number of turns while enabling more complex, long-horizon tasks and emergent behavior. He describes how Arize Phoenix uses Bash, Python, SQL, and UI-bound programming, while also outlining sandbox risks such as security, reliability, and operational complexity. He closes by describing how Arize evaluates Code Mode with the Harbor framework and uses evaluation results to improve agent behavior.

Key points

Coding agents can dynamically build software, weakening the traditional divide between software for engineers and software for non-engineers.
Traditional tool calling can bloat the context window with irrelevant tools and outputs, increase turns, and add latency and cost.
Code Mode lets an agent search for specific tools and execute them in a programming environment, including sequentially, in parallel, or with conditionals.
Progressive tool disclosure, composability, token efficiency, and emergent behavior are key benefits of Code Mode.
Arize Phoenix uses Bash, a Python sandbox, REST CRUD endpoints, SQL, and a UI-bound programming paradigm.
Sandboxes introduce considerations including CPU and memory, network access, secrets, permissions, failures, debugging, and tool discovery.
Open challenges include resuming failed work, determinism, moving binary data out of a sandbox, and using smaller models.
Arize evaluates agents in sandbox environments with Harbor, repeated trials, and verifiers that score task completion.
In the reported evaluation, Code Mode reduced token use by 43% and cost by 31%, while latency remained pretty constant.
Capturing trajectories and evaluations helps identify problems in agent instructions and improve performance over time.

Chapters

Start	Chapter	Summary
00:01	Why coding agents change software access	King introduces the shrinking divide between tools built for engineers and software that non-engineers can use, as coding agents can build tools dynamically.
03:08	How Code Mode works	He contrasts tool-by-tool calling with a programming environment that can compose tool calls and return only the information needed in context.
06:14	Code Mode in Arize Phoenix	King describes Phoenix’s use of Bash, Python, REST operations, SQL, and a UI-bound programming paradigm.
08:17	Sandbox trade-offs and open problems	He discusses security, resource management, reliability, debugging, double sandboxing, and challenges such as determinism and handling binary data.
11:25	Evaluating Code Mode and results	King explains the Harbor-based evaluation process, reports lower token use and cost with similar latency, and describes using evaluation data to improve agents.

Quotes

you give it the ability to search for the specific tool — 03:08

it only has to put in exactly what’s necessary into the context window — 04:10

being able to build its own tools — 05:42

decreased the amount of tokens by 43% — 12:58

give your agent a sandbox — 14:02

Concepts and entities

Code Mode, sandbox, tool calling, Arize AI, Mikyo King, Arize Phoenix, Harbor, MCP, Bash, SQL

Full transcript (original language)

00:01 [music] >> Thanks for everyone for coming. So, my name is Mikail. I head up the open source group at Arise AI and we work on a fully open source observability and evaluation framework and we also work on our own agent. So, today we want I want to kind of walk you through what Code Mode is

00:32 and our journey embracing Code Mode. So, if you look back in the last 30 years of software development, you could kind of segment software into two different categories. You could think of there’s software built for software engineers and software for non-engineers. Take for example, you have you know, you’re asked to get all the invoices out of your email inbox. If you’re a programmer, you could write a script, [snorts] you can go in there, you can dynamically like see if there’s an attachment, download it, put it into

01:02 a folder. It might be a you know, 50 lines of code and you get this tool that you can use versus if you were a non-technical, you’d have to go into these inboxes, you have to manually download it. You’re restricted by the interface of the software and you had to kind of buy that software. And the thing that’s really interesting about things like Cloud Code is that that boundary is completely dissolving, right? Basically, anything you need, the coding agent can actually build that software for you dynamically. And the labs and the

01:33 foundation model providers know this, right? If you look at all the benchmarks like SWE-bench, that is what these models are being trained to do. And so, doesn’t it make sense basically to align the agents you’re building to be able to kind of build the software you need dynamically to dissolve that boundary. So, before I get to started in Code Mode, let’s look at kind of what an agent looks like. So, hopefully you’ve built this uh uh kind of agent before. You give an agent

02:05 kind of a huge catalog of tools, and then iteratively a user question comes in, it calls the tools that get pushed in the context window, it reasons about it, and then it calls another tool that get pushed in the context window. And hopefully most of you have built an agent like this. Uh certainly if you have a certain set of tools, this can work pretty well. But, we’re now in that age where we want a lot more. We want much more emergent behavior. We want to put a lot more context into the agent. We want to have longer horizons. And so,

02:37 this is where you kind of get this bloat inside of the context. You’ll see there’s a lot of tools that get added into the context window that might not be relevant to the task. Um maybe the tool call is not responding with a lot of junk, or maybe needs to paginate through data. And so, you deal with this kind of context rot. You also deal with multi-turn, a lot of turns, as a result, and you’re going to have to deal with compaction. So, luckily CodeMon solves a lot of these things. It’s kind of an alternative

03:08 architecture. Um instead of, let’s say, just giving it a huge index or a menu of tools, you give it the ability to search for the specific tool, and then you also give it the ability to execute the tool. But, you don’t give it just the ability to execute the tool, you actually give it a programming environment to execute those tools in different types of flows. So, they can do it sequentially, they can do them in parallel, um can put conditionals around it. Um and the nice thing about this is

03:38 you can call multiple tools in one turn, and also return only what’s necessary to push back into the context window of the model. And so, to kind of summarize it, put those these two things hypotheticals side by side, traditional tool calling, you know, you accumulate context window blow every time you call a tool. That’s obviously going to, you know, add to latency and cost. It’s also going to increase increase turns. So, the time your user takes to get to value increases, right?

04:10 So, has a lot of overhead. Um, on on the flip side, code mode potentially, when done well, can unlock a lot of abilities because it can call as many as it wants in a single turn and it only has to put in exactly what’s necessary into the context window to solve its problem and you can also get a reduction in turn. So, that’s sort of the the goal of of code mode. And so, let’s just kind of go over some of the reasons why code mode works. Code

04:41 mode works because one, you don’t have to give it the giant catalog of tools up front. It can just pick and choose the tools that it needs and understand those tools and then execute just those. So, there’s a progressive disclosure of tools. Uh, certainly there’s things like tool search now also built into the foundation model providers. There’s also the composability. There’s a huge amount of benefit to not being able to being able to call tools in different sequences, either in sequence or parallel, fan out, chaining them together. You can create much more

05:11 complicated and efficient things. If you’ve heard of obviously Claude’s dynamic workflows is built on this composability but composability of sub agents. I already mentioned sort of token efficiency. You don’t have to spit out all the tools outputs and worry about the tools outputs being shoved into the context window. The model can dynamically kind of trim and search for certain areas and only put what it needs back into its context window. And the fourth one, which is the kind of the one of the major reasons we’ve embraced code mode, is this idea of

05:42 emergent behavior, being able to build its own tools, right? So, for example, traditional with traditional tool calling, if I said for example like take all these records and remove personally identifiable information, it wouldn’t be able to do that because it doesn’t have a tool, or it might try to do it inside of its its context window, but then you’re leaking credit card numbers in the context window, and that’s not great. So, the emerging behavior of being able to create its own tools has a huge amount of benefit. So, here are some ways in which we’ve

06:14 embraced Code mod. Uh so, we build our own agent uh and harness inside of Phoenix, our open source uh our observability platform, and we started with Bash. You know, Bash is a great tool of being able to compose together things like uh you know, said and grep to be able to uh compose together, you know, built-in Linux primitives with also our own CLI and be able to offload those into files for later use. We also added um

06:44 you know, a Python’s sandbox, and we hoisted all our rest CRUD endpoints. Those are like perform getting traces or uh manipulating data sets, and we put that into this coding environment so that it can compose those create, read, update, delete operations, which unlocks like a whole category of things. And we also added one additional one, which is the ability to write SQL. And you know, text of SQL, you’ve obviously like heard like that probably many times before, but this also unlocks a lot of

07:15 capabilities. The reason why is not everything fits into the tool category of just creating, updating, and deleting. Maybe you need to be able to do aggregate queries like tell me all the LM spans that have P95 latency of X, Y, and Z. You don’t want to have to paginate through all your traces to do that. You want it the LM to be able to pinpoint find exactly what you’re looking for. This is kind of a a neat and creative way we’ve also used code mode. So, we’ve actually also created a sort of a

07:45 programming paradigm in our UI that’s bound to our UI. So, we’ve been able to kind of be able to give it the ability to change the time range or be able to read data from the data store inside of the front end. And what’s really cool about this is you kind of almost get this computer loose use like emergent behavior because we’re able to put that programming paradigm in a web worker that has a two-way data binding directly into our browser and it can kind of do complex UI interactions for you and it’s really pretty fun to to watch it uh

08:17 cook. So, code mode is not uh obviously for free. Um you’re dealing with LLM generated code. If you’ve been at to the the Anthropic talks, you certainly probably seen some of these. You’re dealing with resources. So, if you’re working with hosted sandboxes, you have to worry about CPU, memory, how long those sandboxes stay around. Networking is a huge one. Maybe you want it to be able to go out to the web, but you never know what’s out in the web. And so, you have to carefully think about network egress. Uh secrets is a

08:50 hot topic. I think every sandbox provider is trying to come up with a different vaulting strategy. But, let’s say you want to use Jev inside of your sandbox like Pi uh durable just shipped, you know, that’s a crazy cool idea uh to be able to make fast decisions inside of a sandbox, but you have to make sure that those secrets don’t come back into the context window. And then the other one is permissions. Uh there’s some very interesting research around this as well, but typically permissions are overbroad. And

09:20 so, maybe in the first five sets the permissions are correct, but in the sixth operation it is not. Or maybe in composition uh the permissions are not. And so, you have to kind of think of guard railing some of these things if that is necessary. There’s also operational complexity. Uh certainly sandboxes don’t boot uh all the time perfectly. And so you have to you know be able to deal with failures. They’re harder to debug because now you’re debugging code and you’re not debug debugging individual tools. And

09:52 tool parameter accuracy becomes a much bigger issue because the tool parameter now is code. Uh so what if something fails in the middle? Uh tool discovery is also still an issue. Previously if you did have a tool, the LM would up front know it, but now it might have to go through a handshake to be able to understand which tools are in its catalog of things to be able to do. And then this is sort of a big one, you know, you see code mode being injected into MCP, but then you see a lot more code mode being pushed also into

10:23 harnesses. And now you have a double sandboxing uh problem. So sandboxing, you know, is going to return stuff out as serialized JSON and then it needs to be parsed and then put into another code mode. And so there’s this like double duty that ends up happening that also can be on operational complexity. These are sort of open problems. Uh they’re not like blockers, but durability is a problem. So let’s say you build a giant piece of code that needs to execute and it breaks in the

10:53 middle. You can’t resume from that step. Um if determinism and mathematics are important to you, for example, it might actually force you to think about which programming language you’re using. So JavaScript is something LLMs are really good at doing, but maybe not great for math. Um For example, uh code mode can be great way to do multimodal and manipulating images or creating bytes, but getting that out of the sandbox is quite tricky, right? Because you don’t want to serialize the whole binary, the whole image. That’s

11:25 going to end up in the the context window. So you might have to do some sort of signed image handoff. And then the other one which is like the Frontier models are amazing at code mode, but if you’re trying to do cost saving and you want to swap out a smaller model, its capabilities might not just quite um get there. So, how do we at Arize evaluate code mode? Um we use our own We dogfood our own platform, which is called Arize Phoenix, um and we run our agents through various

11:56 permutations through something called the Harbor framework, which basically puts the agents into uh sandbox environments and runs them on clear instructions with network access turned off, and we run this multiple times in trials, so we eliminate a lot of the flake. And we build evals in the form of Harbor verifiers to make sure that the agent’s doing the correct thing. And then, Harbor’s sort of rooted in this RL framework. We give it a a one or a zero to indicate whether or not performs the

12:27 task, and we test it across many different permutations. So, this is kind of why you want to test like this, because as I mentioned, you can get a lot of flake. Um sometimes it might work where you’re dealing with an indeterminate system. It might take different trajectories uh depending how you do it, and we uh track pass at K of each individual task that we think it should pass. So, you know, how does this work? Uh this is kind of our results from uh running this

12:58 Harbor run. You can see pre uh code mode and post code mode, we, you know, decreased the amount of tokens by 43%. The cost went down by 31%, and you know, there was some overhead with the sandbox, but the latency stayed pretty constant. So, we knew that across these multiple surfaces, whether or not it was cloud code or MCP or the CLI, we had accomplished something that uh was definitely better than it was before. How do we use this data? It’s really uh a huge unlock if you have you both capture the trajectory and the evals. As

13:30 you’ve seen with a lot of the other talks, this gives you a very great environment to reinforce and look at the behavior of these agents and be able to tell it how to hill climb. So, maybe, you know, first time we introduce a new tool, sometimes actually the efficiency goes down. And so, we can tell a coding agent why in these specific scenarios, what parts of the instruction are are guiding it in the wrong direction and we’re able to hill climb these evals to be able to make our agents better. So, if you don’t take anything other than

14:02 this away, hopefully this will sum it up. We really like code mode at RI because of the fewer turns you get out of code mode. We want you to be able to bring any model for us. So, we want it to be cheaper and we’re really excited about the emergent behavior and the long horizon tasks that our agents can do. And so, I hope this talk convinces you to give your agent a sandbox and then use one of our tools to evaluate it. And that’s it. Thank you.