Hill-Climbing Skills: Improve Agents Without Changing the Model — Shubhankar Srivastava, Browserbase

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Hill-Climbing Skills: Improve Agents Without Changing the Model — Shubhankar Srivastava, Browserbase · Channel: AI Engineer · 51:22 · Watch on YouTube

Topics: AI agents, Task and workflow automation, AI evaluation and benchmarks, AI safety

Summary

Shubhankar Srivastava explains how to improve browser agents through a repeatable “auto-browse” loop, without changing model weights. A doer agent performs a task while a teacher reviews its traces and updates a persistent strategy file, producing reusable skills that can reduce cost and improve speed and reliability. He demonstrates the method on Google Flights and OpenTable, then discusses how to adapt it to other domains, evaluate outcomes, and keep skills current. The talk also covers practical requirements such as replayable environments, verifiable goals, and safeguards against prompt injection.

Key points

Browser agents use browser-control tools, such as a CLI or MCP, to interact with websites.
Reliability is presented as an engineering challenge involving tools, skills, and the lack of learning between agent runs.
The auto-browse loop pairs a doer agent, which performs the task, with a teacher agent, which reviews traces and records lessons.
A persistent strategy file carries learnings between iterations; the completed skill can then guide future runs.
In the demonstrations, agents learned to use direct query URLs and avoid website paths that caused access denials, improving efficiency.
Auto-research loops need replayable environments and verifiable or quantifiable outcomes to assess progress.
Loop objectives can prioritize success, latency, number of turns, reliability, or cost, depending on the use case.
Skills can be applied beyond browser agents, including coding, research, and internal operations workflows.
Skills may be updated after production failures or checked on a schedule, with human review incorporated into the repair process.
Prompt-injection defenses can be applied at the query, skill, and infrastructure layers, including URL and action restrictions.

Chapters

Start	Chapter	Summary
00:00	Introduction and browser-agent primer	The speaker introduces hill climbing for agents, his work at Browserbase, and how browser agents control websites.
02:39	Why browser agents struggle	Examples from Google Flights and OpenTable illustrate reliability problems and the limits of agents that start each run without learning.
06:01	The auto-browse approach	The speaker connects the idea of auto-research to browser workflows and begins a live flight-search demonstration.
08:10	Doer, teacher, and persistent strategy	He explains how a doer performs browser tasks while a teacher reviews traces and saves lessons in a strategy file.
09:46	Learning from iterations	The Google Flights demo shows the agent discovering a more direct URL-based approach and turning accumulated lessons into a reusable skill.
12:55	Examples of cost and reliability improvements	The speaker reviews prior runs, including an OpenTable example where the agent learns to avoid blocked paths and use a search URL.
16:06	Production use and browse.sh	He shares reported task improvements and introduces browse.sh as a collection of website skills for agents.
20:30	Applying the method beyond browsing	The approach is proposed for coding, research, and internal operations, with replayable environments and verifiable outcomes as key requirements.
22:18	Verifying progress and designing robust loops	The Q&A addresses how to judge convergence, handle failures between iterations, and invest in environments that can be replayed.
26:52	Loop costs and adapting auto-research	The speaker shares resources and discusses iteration budgets, amortized costs, and how customer workflows informed the auto-browse design.
30:35	Inspecting traces and harness design	He shows how the teacher appends learnings from each iteration and discusses choosing and evaluating skills and tools for a use case.
36:01	Skill storage and keeping skills up to date	The discussion covers storing skills locally or in cloud storage and triggering updates when production runs fail or scheduled checks detect problems.
39:23	Optimization criteria and WebMCP	The speaker explains that loops can optimize for different goals and comments on current support and adoption for WebMCP.
42:06	Ground truth, real websites, and browser challenges	The Q&A covers pass criteria, real-world browsing environments, CAPTCHA handling, and how the teacher evaluates traces.
46:44	Portable skills and prompt-injection defenses	He describes skills as portable Markdown artifacts and outlines safeguards at the query, skill, and infrastructure layers.
50:15	Scheduled checks and closing	The speaker describes using scheduled agent runs to detect failures and trigger a healing loop before closing the workshop.

Quotes

these browser agents, I call them geniuses that never learn. — 05:27

This doer agent its job is to actually go and do the browsing. — 08:10

the key thing that happens is there is a strategy.md file. — 09:16

reliability is like the most important thing for me — 16:06

I know browser is a crazy threat vector when you think about it — 47:30

Concepts and entities

hill climbing, browser agent, auto browse, skill, Browserbase, Shubhankar Srivastava, browse.sh, WebMCP, prompt injection

Full transcript (original language)

00:00 All right. Hello. Thank you everybody for being here. Uh I know it’s the last workshop of the day. I appreciate all of you turning up here. U I am show uh I’m here to talk about hill climbing skills. uh specifically how can you improve your agents without really going into the models without really going in the

00:33 weights. Uh I have a slido uh up here as well. I’ll just uh flash the QR code in case any of you want to uh put in any questions and we can address them towards the end as well. U a little bit about me. I’m Sh. I’m uh on the sales engineering team at browserbase. Uh I essentially work with our customers deploying browser agents in production. Uh I’m not sure maybe we’ll do a show of hands. Uh how many of you know what browser agents are?

01:06 Anybody? All right, a few. Amazing. Uh have any of you built browser agents before? Have any of you used these tools? Show hands. Okay, a few of them. Amazing. Nice. uh maybe I’ll I’ll talk a bit about what browser agents are before diving into the whole concept of hill climbing for skills uh so that it becomes clear what we’re doing towards the end of the talk we’ll also look into how you can do this in your domain in your uh field as well.

01:39 All right. Uh so quick primer on what browser agents are. If you have an agent that needs to drive a browser, you basically call that a browser agent, right? Browser becomes the tool to your agent. And how do you control it? You control it with some sort of CLI, some sort of MCP, some sort of method where you can control the browser and say, “Hey, go and click on this red button. Go and click on the refresh button. Go and uh take a

02:09 screenshot or take a snapshot of the accessibility tree of this website.” Right? So these are this low-level tools that your agent can have access to to control the browser. Uh and browserbase is essentially a platform for you to build and run these browser agents. We provide the infrastructure and the SDKs, the skills, the CLI that go with building these browser agents. Uh and and we’re going to be using all of these tools to see what hill climbing does

02:39 look like. All right. Uh I want to talk a bit about the biggest problem with agents today essentially reliability right uh people are frustrated with browser agents getting stuck or getting into an anti-border capture issue uh or maybe just forgetting or being lazy and not completing the task. Uh and we see reliability essentially is an engineering problem. I think the recent models the Opus and the Fable when we

03:11 when it came out for those three days it was really really smart right uh but we see that a lot of the work that has to be done for reliability is the hardness around it is the tools that you give it it is the skills that you give here u and and that’s what we’re going to be sort of looking at how do we improve the reliability of these browser agents I wanted to give you all a few examples of like where these agents suck today, right? If I zoom in a bit, you’ll see this browser agent is trying to like go on Google flights and click on the uh

03:47 it’s trying to drag the time filter and it’s unable to figure out how do I drag that. Dragging is a tricky operation for these agents to do and it’s getting super confused. It doesn’t know what the HTML element for that is, what the JavaScript for that is. It’s not really able to do this and it gets stuck and it’s not able to book my flight tickets. Uh, another example I wanted to show uh is a browser agent on open table. Let’s say I have my open claw or my Hermes agent and I need to book a date night and I tell hey go to open table and and

04:18 find me some and and look at the anti-bot that it runs into, right? Uh, and it’s just like access denied. I don’t know what happens here. Uh, but like can my browser agent get around these issues? Can they be more reliable? And can they book my date night restaurants is the biggest question I wanted to ask here. uh and this is where we sort of look at different ideas to improve the reliability of these agents and one of them uh oh actually let’s let’s dig into the root cause here right like why this is happening u we have

04:52 like these opus models these five five the super nice general purpose you might have used codeex computer use very nice but when you want to sort of do uh longunning tricky websites like the open tables or the Google flights it sort of becomes very difficult to run these in production and these sightspecific tricks or these javascript that you need to run on your browser the agent doesn’t know about them right and the most important thing is it’s every run starts from zero you’re not really having any sort of learning baked into your runs I

05:27 think Darkh talked about this a couple days ago it’s like why is progress on computer use still so slow today. Uh although computer use seems like a verifiable problem, why is it still not solved? Essentially, there’s no learning happening. Every agent run is a start from scratch run and it’s not learning from its production traces. U and these browser agents, I call them geniuses that never learn. Um and some people might say, oh, why don’t you fine-tune on something some like open table or restaurant websites. Uh but it it’s not

06:01 really scalable, right? uh if you’re thinking of building agents, you can’t really be fine-tuning for every new task that comes up. You want some you want a a modality where it can learn from its production traces uh and get better over time. This is where we sort of started thinking about and this was like six four to six months ago where Karpadi came up with this idea of auto research. This is essentially a loop, right? Uh and this loop sort of takes a verifiable problem statement and goes at it, tries

06:34 it again and again, looks at its own traces and tries to improve itself. And we wondered how can we apply this to browser agents? How can we apply this uh to these complicated workflows, these complicated websites which are not deterministic, they change every day, they have different captures every time. Uh how can we learn from these? Right? Uh this is where I want to sort of switch into a demo and sort of show this live. Uh and I’m going to start with uh

07:08 a prompt over here. So I want to maybe do something like uh go to Google flights and uh find me a business class flight from SF to New York for this Saturday. I want to have one carry-on bag. Uh and this flight should be in the morning. I hope my Okay, this is correct. Just paste this and run this. Right. So starting a browser based session here.

07:40 Uh so what you see in the center is a remote browser has just spun up and has navigated to Google flights. If I expand my agent here, uh it’s doing all of these things. This is the reasoning traces that you can see live and it’s trying to sort of apply the business class filter. So it it’ll do its job and and and we’ll come back to this. Uh but I wanted to talk about what’s really happening here. Uh u and the architecture behind this auto research

08:10 loop. So we we fired the Google flights task. Uh and and and the architecture here is this auto browse loop that we call this. We took auto research and we took browsing. We call it auto browse. And this loop essentially is two clawed agent SDK loop uh agents. Uh you can call one as the doer agent and one as the teacher agent. This doer agent its job is to actually go and do the browsing. It has access to the browse CLI that we talked about. It has access

08:44 to the browser uh and it goes and sort of clicks on buttons, looks at the screenshot, tries to get me information and the teacher teacher’s job is to look at the traces of the doer and learn what went wrong and what went right and they keep working together until they find a very good solution that converges. Right? Uh we don’t tell it how many loops you need to go for. We don’t tell it what’s the success factor success uh rate that we want. We don’t

09:16 tell it hey we need to do this within x number of tokens or x number of seconds. We just give it a broad prompt and we let the bitter lesson we embrace the bitter lesson and we’re like okay let the doer and teacher go at it. Uh and the key thing that happens is there is a strategy.md file. uh that is essentially the artifact across multiple runs. After every iteration, the teacher thinks about what happened, looks at the screenshot, looks at the logs, and then

09:46 thinks, “Oh, you did not click on this button. U you should actually click on this button instead, and it puts that in the strategy.mmd file.” And that artifact carries over to the next iteration, and it improves from there. Maybe we’ll go back to our demo and see what happened. All right. So, iteration number one, uh, there was a failure here. Uh, and the learning from this iteration was skip the entire form calendar widget because you can just do a query parameter and you don’t really need to click around, right? The agent

10:19 was smart enough in this instance to think, oh, I’m wasting so many tokens and I’m wasting so much time clicking around where I could just use Google Flight’s query parameter. Now this is obvious in hindsight but maybe as a human I wouldn’t have thought of this and I wouldn’t have written this complicated query parameter myself right uh so this is the learning from the first iteration now it’s going and doing things in iteration number two it’ll keep going on till it feels like it has maxed out all the tricks it has maxed

10:50 out all JavaScript hacks that it could do uh and and and then uh think about okay this is the skill that I can use from here. I think the key the key thing uh that I wanted to also talk about is at the end of this converged loop the output that I want is a skill.mmd file. This skill.md file would be what I give to my future agents. Uh and it would have a

11:22 collection of all the hacks or all the tricks or all the learnings from previous runs. We’ll we’ll look at what that scale might look like. Uh it could look like, hey, a pre-filled URL is mandatory. Don’t go around and waste your clicks and tokens clicking on these buttons. There is a reopen bug in the calendar. Don’t do this. Uh if you want to move the drag uh if you want to drag your time stamp thing, uh use JavaScript injection. Don’t try to do it with the

11:53 mouse. Right? These are the kind of tricks that are now persisted for every future run and gives me so many more savings uh from a performance, reliability and cost perspective. Now uh we’ll go back to our uh run and see if it has passed. Okay, so iteration number two has passed and if you can see uh the savings in in cost are like 4x. Uh this chart is being created on the

12:23 fly. Uh so it started from $12 and the second one was $3. And keep in mind my agents here are Opus 48 both for the doer and uh for the uh teacher. Now iteration three just came in. Uh the number of turns decreased. The latency is also better. Uh but there was a failure in iteration number three. there’s something that it sort of overcorrected on and iteration number four should probably be learning from that and we’ll see if it does that.

12:55 Uh while it’s doing its own thing, I wanted to show some previous runs that I did uh just in preparation for this uh demo. U so for example, I went to uh auto browse and I was like hey go to the AI engineers website and tell me which talks and workshops are talking about hill climbing, right? uh and it did that across multiple iterations and you can see the replay of each of these. These are replays that are coming from browserbase super helpful important for uh observability perspective and as you can see uh it sort of uh got better in

13:28 ter in terms of cost 7.4x 4x uh even though it was able to pass in all the iterations, I still got improvements in cost. Now I wanted to actually see what are the learnings from this skill, right? Uh how do you sort of uh persist all of these learnings? It says go straight to the shed schedule page. If there are open errors, don’t try retry blindly. Run this. Uh type the keyword into the session search input. Wait briefly. So like have these weights

14:00 because sometimes websites take a lot of time to load. So you you can have these kind of tricks uh and then read the results. Uh you know like there there’s a lot of learning that is happening in these skills. Maybe I wanted to show some interesting ones. I wanted to show this open table one because I started with this blocker of like access denied when we started this demo. Uh and if we go to the uh screenshot, this was the access denied thing that we got right in in iteration number one. Let’s see how it

14:31 learned from there. Uh it realized that the uh block is environmental because the website doesn’t want this agent to do something. uh in iteration number two, it figured that if I do a query parameter and if I put a latitude longitude metro ID, I automatically get that uh UI that works. Do not open /r that’s blocked and do not reload / s. These are the places where the blocks kick in. If you don’t do that and you follow this technique instead, your

15:03 agent will work, right? Uh and as you can see, uh it sort of went from $6.22 22 to 0.51. The time taken went from $189 seconds to 27 seconds. Uh and the scale it came up with was here. Uh build the search URL, do the date math like this. Uh this is the known metro pins for San Francisco. Uh and and follow these steps, right? Never call / r because that will return

15:34 access denied. Now these are the kind of learnings where a human could never have done but you just sort of embrace the bitter lesson and a let agents do the job uh and they will find these uh tricks for you. I want to uh also show uh a few examples of like the kind of improvements we see. Uh the reason why this problem is so close to my heart is because I from a sales perspective am deploying these browser agents into production. uh the

16:06 sale is only done when the browser agent is successful. So reliability is like the most important thing for me and this is what I this is the key tool that I take to my customers and I’m like just fire your agents let them do this auto research thing and your agents will be successful in production. Uh and these are the kind of u success metrics that people are looking for up to 16x cheaper 12x cheaper 3x cheaper uh on different tasks. Of course, every task is different. The bottlenecks on each task are different. So, uh you might see

16:37 variance there. But the idea is you can hill climb on these uh techniques. I wanted to go back to our live demo and see if there’s any uh improvement. All right. So, we have a graduated skill u and the skill applied end to end. No further learning. The results are showing correctly. one-way business class, San Francisco to New York, July the 4th, one carry-on bag included. Uh, and there is 8 a.m. $649. Right? So, it was able to get all of this and I can

17:11 sort of watch the entire replay to see if that is accurate. Uh, it was able to sort of figure out this entire skill and the fast path is encode the whole query in the URL. strongly preferred the again like I know I’m I’m showing these things but when I first started looking at the the results here this was like my moment of like this is the mythos moment for browser agents you know like just reverse engineering the hell out of things uh I I look at this with some of my customers the customers are mind-blown uh earlier they would be

17:45 writing scripts for each of their websites they would be like hey I need to spend so many engineering hours on like just reversing these websites. How can I scale this to 100 a thousand websites and now our customers have actually built these entire pipelines of uh these skills that get generated on the fly pull requests that get created automatically. If there is a failure in production, an agent kicks in and starts healing the skill as well. So there’s a

18:15 lot of uh engineering that we can do around this to make leverage to leverage this technique. Uh I wanted to also show what we’ve done with this right we actually went ahead and created this product called browse.sh uh we were like if we can do this for one website we can do this for five why not do this for like 500 websites and give it for free for anybody to do this. Uh and like if you remember in the 19 or wherever like in the ’9s before internet you had the yellow pages uh you had

18:49 details about accessing people, businesses etc. This is yellow pages for browser agents right it has details about how to do any task on any website on the web. U I wanted to actually show this to you all. uh you can actually install browse CLI and use all of these skills completely for free. We’ve eaten up the token costs ourselves and we’re making this available for you all. Uh so if I go to let’s say search products on Amazon uh I

19:23 can see the entire scale is over here. When should you use the scale? How should you look at like pricing and availability and whatever right? uh this is how you use this lot of details about Amazon the screenshots u and I just give this browse skills CLI to my openclaw and it just knows when to pull the alltrails.com skill when to pull the weather.gov skill and and this makes my life so much easier. uh we essentially are running the same auto research loop

19:56 behind the scenes uh and making this uh available for you all. In in fact if you would want to you can go here and you can be like hey go and make a skill for me to uh look up this uh events in San Francisco for this week. Uh you do this generate a skill and this will sort of do this for you. This will send you an email and you can then use your skill uh in whatever agent you want to, right? Uh so we’ve done this for browser agents. Uh and I wanted to talk about like how

20:30 can you potentially do this for any other domain, right? Uh I think coding agents is like one place where I have seen this frustration where the agent is relearning the repo conventions every time. it’s not applying the linting rules correctly and I have to tell it every time and maybe you can have your cloud MD and agents MD and improve that but the the fundamental premise is like the agent should not be relearning your tasks from zero right uh I’ve seen this

21:04 in research agents we run a bunch of like deep research agents for our internal workflows uh and every time it struggles and it tries to relearn oh I pulled this information from Wikipedia IA uh but I did not pull it from this other website. Uh I was like why not? Uh like we always do this other website you should learn from your previous traces. Uh and you should pull information. Uh internal ops you have all these dashboards all these databases all these internal tools but your agents don’t know which ones work and which ones are

21:37 preferred. Ideally you should be applying this all research technique. Uh a and the two conditions that are required is that a you should have a replayable environment of some sorts so that you can run these loops and b you should have some sort of verifiable outcome some sort of quant quantifiable metric for you to run these loops. Yes, I see a question there.

22:18 I realized that Yeah. >> Yeah. >> Yeah. Exactly. Uh so I think the question is like how do you know that the auto research loop is accurate

22:51 whether it is converging and when it says it’s converging it’s actually converging. When it says it’s good it’s actually good. When it says it’s bad it’s actually bad. like how do I have confidence uh on what the agent is saying and what the loop is doing. I think that’s where I want to sort of talk about the verifiable outcome here. Uh for Karpathi, it was quite easy because he was looking at the log log loss of the experiment when he was training neural networks. Right? So it’s very easy to see whether your log loss has gone down or not. Uh but in many real world scenarios that is not the

23:24 case for browser agents. As I explained, uh, whether you book a flight ticket or not, that might be straightforward, but did you find the best flight options for me might be slightly fuzzy uh and might not be the best uh

24:05 Yeah, if if I understand this correctly, you’re saying like one mistake in the loop can sort of throw off the entire auto research run. Uh and how do you prevent against that? I I I have actually seen where some uh iterations actually fail. uh for example if we uh look at the previous slide uh there’s like failures happening after there have been successes right uh so it’s not always that once you succeed all the future iterations will always be

24:38 successful but I think it’s the design of your loop where you can make it resilient to u failures uh where it doesn’t overindex on those uh and still sort of correct protects itself uh from any mistake that happened in previous runs. Uh I think we’re trying to sort of prevent like uh what do they call like exploding sort of loss sort of scenarios. Uh I think you could do that. Uh in fact Karpathi talks about this like this is one way of doing auto

25:10 research. You can actually auto research on different ways of doing auto research itself. Uh it’s like is the teacher doer loop the best? Should there be a multi-agent architecture to do this or is there any other approach that might actually work better for research tasks in neural networks versus browser agents? Right? You could have different designs of your loops to prevent situations like those. Uh and I think going back to my point, as long as you have a verifiable outcome or some quantifiable metric, I think you should

25:40 be able to design uh a good uh loop here. uh replayable environment I think sometimes is under discussed uh but extremely important. I think he was talking about Darkh he said having a verifiable outcome is important but also you should have uh extremely grindable environments or extremely grindable uh use cases as well. Right? If your agent cannot uh grind through failures, it will be difficult for it to uh go for multiple

26:15 iterations and fix itself. Uh so do you have like coding agents is a good example because you can start from a docker image or you can start from uh a particular EC2 and and you can replay from different parts of the uh stack. Uh but maybe uh knowledge work is probably not that easy to replay. uh maybe there’s like a lot of different human in the loop actions that make it very hard to replay from a particular point in time. So investing in your environment also can be super uh rewarding in these cases.

26:52 Amazing. Uh I just wanted to share the uh URLs of uh the auto browse loop that I uh demoed today. So if you scan the QR you’ll be able to pull the uh GitHub for that. Uh and also the second QR here uh is the skill itself. You can just do /auto browse on your claude code or codeex and be like hey /auto browse use opus48 on both of these uh teacher and doer and go create a skill for me to do this. Uh

27:25 I’m looking at one of these questions. Uh and also if any of you have ideas of something we want to sort of put the agent against feel free to type it in the slido and we can actually try it live. Uh but let’s look at this question. Is there a way to optimize for convergence? Uh I think this is the same question that we were just talking about. Uh high cost is interesting because you might think that u doing longer iterations might become costlier over time. Uh I have run a bunch of evals on

27:59 this where I can throw a sonnet at this problem but run it for longer iterations versus throwing opus for lesser iterations. I think you can get the same job done by throwing sonnet larger number of iterations but still cheaper overall. Uh also this cost is amotized right like one auto browse loop that we probably run is like maybe $30 $40 if it’s super complicated $100 at most. Uh so this is amotized as long as those costs make sense for you uh to deploy

28:31 this in production uh just go for it. Uh but there’s a lot of interesting things uh that you could do uh with evaling the kind of loops that you’re generating. Uh yes I will share the slides as well. Uh I can do this on the GitHub itself. Uh can you share how you thought about uh auto research into auto browse? Uh I mean the process was fairly straightforward. Uh took the GitHub repo that Karpathi put out uh and then

29:03 there’s all this context about my customers uh that I have uh where I was like hey these customers face these kinds of reliability problems uh the way I solve it for them is I go into their office I sit with them very forward deployed uh and I try to sort of just brute force and reverse engineer this. This is what I do. So I am the teacher agent. Uh can you automate me out of the process and create a loop borrowing from uh Karpathi and and and and build the

29:33 same thing? And of course there’s a lot of iterations on top of that. Uh there’s a lot of engineering as well uh to simplify this. Uh but yes, that was the process of converting this. Can you use this in a proxy? Can you be logged into a site? Yes. Uh again this goes back to the environment that you design. Uh browserbased the infrastructure that all of this is running on uh gives you the ability to log into websites uh save the cookies uh reuse those cookies in future runs so

30:05 you don’t have to log in every time. Uh makes it super easy for you to uh replicate real world workflows. Uh and yes you can use proxies as well. If you have uh particular IP addresses, residential IP addresses, you can bring those in as well. Uh a lot of success is definitely how good your infrastructure and environment is, especially in these adversarial scenarios, uh like captures and antibots. Uh so making the infrastructure as good as possible is

30:35 definitely of value. Amazing. any any asks for something that we might want to run live? Yes. Question. >> Great, great question. Actually, I wanted to sort of show the code as well. Uh so if you uh look at the GitHub, you’ll be able to find this evaluate.mjs file. Uh essentially, it lists out the entire

31:09 loop. Uh what’s happening here? Uh but to answer your question, the teacher and the doer are u working one after the other for every loop. So the teacher is finding learnings in every iteration. Uh maybe if I show the demo uh over here uh let’s go to the one we just did. Iteration one had these learnings and then iteration two had some other learnings. Right? you should include the

31:40 explicit ISO date in this particular format. Iteration three would have some other learnings. The date resolution still opens the bare page and clicks wasting three turns uh reading today’s date uh or having the executor compute today. So it’s figuring out something about the date time. So in every iteration the teacher is learning something new and putting this back into the strategy.mmd. Maybe I’ll make this real by looking at the traces for one of these runs. Uh so

32:12 over here these are all my runs. Uh so for example this was the open table reservation example u and uh this was the strategy.mmd that the teacher sort of appended to every time. So uh step zero you should do this so on and so forth. Uh this was the task this was the uh prompt that I gave it. Find me reservations for the big B bigs restaurant. uh and these were the kind of traces that uh it was sort of learning from. It would take screenshots would take these sort of traces at every step is doing browse open this is what

32:47 the output is. Then turn number two browse snapshot. Snapshot essentially takes the accessibility tree and this is the snapshot. So it’s reading this entire thing and at every step thinking of like what went right and what went wrong. U yeah I mean this is like machine code. uh I won’t show this entire thing. Uh but yeah, this is the summary that it also writes uh at every iteration. Uh so yeah, this there’s a lot lot of like engineering tweaks that you can do to suit this to your domain as well. U and uh like the easiest way

33:21 to uh read this is we put it out here on skills.sh as well uh in auto browse u and you can sort of do something like this.

33:54 Great question. Such a good question. I think that goes back to like how do you increase the reliability of your uh agents. Uh reliability is an engineering problem. Uh you can design your harness to sort of go for one skill for each task or you can be like hey one skill for every task that you will ever

34:25 see. Uh I think it entirely depends how you do the harness engineering entirely depends on the use case that you have u and you run evals with all these different strategies uh and see which one works for you. for us. This was Yeah.

34:55 Yes. >> Yeah. Yeah. And this is where again the harness comes into the picture. If the harness has the intelligence to bring in skills from Open Table and from Resi and Google res Google reservations, maybe it can bring all of those in to figure out what is the best way to book me a date night. That’s your solution, right? Like the harness should be able to uh figure out

35:26 which tools to use to get the job done. That’s like how I would go about it. uh figure out what are the most common use cases that your customers have for those use cases. What’s the best scale files, prompts, tools, maybe you want to do like specific tools for specific use cases, right? Maybe you want to do generic tools for all use cases. Like that’s the same argument here is like do you do one skill for each or one skill for all uh and eval across all of these strategies and and let the hardness figure it out.

36:01 Yes. >> Yes. Yeah. I mean it’s a very similar concept as this, right? Uh so DSPY. >> Yeah. I I think those guys were on to something really cool like they were around like 3 four years ago where they started building DSP and Japa all of that. I think uh we have been doing a lot of research uh along these lines as well. Browserbase works with uh Microsoft. We actually released a paper uh about how to build the right sort of verifiers uh for your agents. Uh I think running evals is straightforward until

36:36 you start questioning okay how should I eval my eval right like do I need to put all these humans to verify all these runs or is there a good verifier that I can build here and all of that u uses the uh learnings from uh these papers as well. I can share all of that in the GitHub as well. I’ll also see if there’s any new questions here. Where and how were these skills stored retrieved at scale. Uh right now for

37:10 this repo uh all of these get stored in my cloud over here. So each of these is a scale file. This was my recipe searcher. Uh I wanted to build some I wanted to cook some dinner yesterday. So I was like, “Hey, make me some recipes.” Uh, so all of these are currently in my local uh skills folder. Uh, but if you want to ship this uh in production, of course, you can uh bake this into your agent loop. Uh, or if you want to I think for browse.sh the public uh publicly available uh skills that we

37:45 have, I think we use versel blob storage to store all these skills uh and make them on demand. Hope that helps. Uh, websites change frequently. What’s your STR strategy for monitoring that and keeping the website skill updated? Yeah, this is such a great question. Uh, and this is what I was alluding to earlier is like how do you keep these skills up to date because the website’s going to change. The the current working theory that we have right now is you don’t want to waste

38:17 tokens running auto research every day. you want to probably trigger or if it’s failing at any particular point in production, right? So, we build these loops or like systems where as soon as we run into any production failure, we run an auto research loop to see what went wrong to heal it, create a pull request, that goes directly to a human engineer and they fix it automatically. The skill gets updated. Uh so yes there is that notion of like production

38:49 failure when this might happen. Uh but it’s a trade-off. You can run an auto research or you can run a sample run every 2 hours every 3 hours to see if the skill is working uh perfectly. So there’s different ways of having the uh cron jobs uh check for the accuracy of these skills. This is actually one thing that I’m actively working on with the customer right now to implement. And there’s all these questions, right? People don’t want 100% accuracy. People might want like 10x cheaper cost. So like uh that’s the uh question that

39:23 comes up. What’s your teachers criteria here in the example? Separate from reaching the goal seems to do some optimizations for turns. What other criteria do you use? Uh so again this is the design of the auto research loop. Uh there’s one way of telling your order research that hey I want to focus on just success. As soon as you get me success I’m fine just give that to me. Uh that’s typically not what everybody wants to do. Uh people want to

39:53 minimize on uh latency. Uh because maybe this is a real time run where you’re focused on how quickly can you get the job done. U so if that is the optimizing metric and you focus on getting it as fast as possible. What I’ve seen sometimes these auto research loops are so smart they realize that browsers are not needed at all and they just reverse engineer the API and they fire the API and that’s the fastest that you’ll ever get to. Right? Uh so sometimes uh I’ve seen situations where uh it finds a solution that works in 500 milliseconds.

40:28 U sometimes it is the number of turns, right? you want to minimize your context loop uh and you want to minimize the number of uh turns it takes to complete the task. Uh in my instance, I think what I have done is it’s your decision to pass or converge whenever you want to pause it. I’m not really concerned about the costs. Uh I care most about reliability. So keep doing it as long as you feel it’s relevant and then it’s the teacher agent’s job to uh call it day when it wants to.

41:03 But yeah, you can uh try all different strategies. Uh if a website supports WebMP, will this use it? WebMCP is so cool. U it’s essentially this concept. I don’t know if everybody knows this, but it’s this concept of like making your website programmable. It’s it’s come out in this new Google Chrome release recently where you tell your u website that if anybody wants to book a reservation they can do xyz on the website. It’s kind of like an API but

41:35 it’s on the browser. Uh right now I don’t think our oral research loop supports this. Uh but I don’t think it would be uh too hard to do that. Uh in fact we recently shipped support for WebMCP with our SDKs that we launched. Uh we also made some tools for website owners to ship web MCPS at scale. Uh but yeah this is an interesting one. We’ve not really seen a lot of adoption with Web MCP as yet.

42:06 Uh ground truth for pass. That’s interesting. Uh I think as long as the job was done as long as for example in Google flights uh if it found the cheapest flights in open table if it found a reservation uh so it it evaluates itself. It’s typically a binary uh and yeah again I I I fully embrace the bitter lesson. I try not to put many guardrails in uh these evaluations. Uh, have you considered a verifier as a

42:41 service for browsing tasks, including something that allows us to mock certain side effects like purchases? Uh, I might have to ask uh a followup on that. What do you mean by verifier as a service? Ravish, if you’re here. Yes.

43:29 Uh so uh do you mean that yeah if I understand this correctly like you have like different environments that you can use for RL training and you want to like mock what happens if I am replicating Amazon if I click on purchase what happens after Uh no because uh we try to rely more on real world websites. Uh browser base we

44:01 have the privilege of being able to access most of the web with our uh access to different websites uh and capture solvers etc. Uh so we try to mimic the real world as much as possible but we’ve been seeing a lot of these RL environments where people are mimicking websites to train uh and build browser agents that are uh really good at like browsing Salesforce, really good at like browsing SAP. Uh we try to do that on real world portals as much as possible.

44:33 [snorts] Are edge captures easy for the skill to move past or is it token incentive? uh what is the point of edge captures for website owners? Edge capture if I’m understanding this is like a particular kind of capture. It’s kind of like Google recapture etc. uh yeah I would probably talk about this separately. Uh I don’t think we can talk about like what it what captures we can solve or not. Uh but it’s not a token inensive job per se because it’s an inf like it’s thing that’s done at the

45:04 infrastructure layer, right? uh we don’t really need to use an AI to solve a capture. Uh it’s solved at the Chrome binary layer. Uh and we do that out of the box. How does the teacher evaluate the doer and when is the skill markers graduated? Yes. So uh it evaluates the doer by essentially reading these traces. Uh for every iteration there is this folder that gets created. uh there are these uh

45:35 JSON files. Uh it reads these uh it has a clawed agent SDK loop uh and it essentially figures out what the learnings from that loop are. Uh actually I think I have a slide that talks about this at the end. Yeah. Uh this is kind of like the prompt for this uh where it says that you’re a browser automation agent. You can use these tools uh and the strategy.md gets injected at every iteration and this is how the uh

46:10 uh doer the teacher evaluates the doer’s job. Uh so yeah it reads the traces it has the system prompt it has the previous strategy and it comes up with a better strategy. [snorts] Amazing. Uh I hope we’ve covered all the questions. Uh any other questions live that we might want to talk about? Yes.

46:44 >> Yeah, it can work with the stage hand SDK. It can work in your cloud code. Uh it’s essentially a markdown, right? Uh I took uh a skill and I told my clot code, can you find events in SF for this Friday, July the 3rd? and went on and did this. So it it need not use browser base. It need not use stage hand. It is essentially a skill. It is a portable artifact you can use in any agent or any SDK.

47:30 Yeah. Yeah. Great question. I think if I understand this correctly, how do you prevent against prompt injection? Uh because websites could trick my auto research loop in learning something bad that can do some malicious behavior in in a production setting. Uh there’s a few ways we do that. One is at the infrastructure layer itself. Uh we have a bunch of uh protections around our browsers itself. Uh we have this browserbased safe mode

48:03 where you can sort of control which URLs it can go to. You can control what kind of actions it can take on different websites. So at the infrastructure layer, at the network firewall, we control things. Now you might say that it is possible that Amazon.com gets hacked someday. How do you prevent against like malicious actions on those portals as well? Uh I think we also try to uh run our skills that are generated against some security uh reviewer agent of sorts uh before we actually publish this on browser.sh. Uh so that’s like

48:37 one way of doing it. uh we encourage our customers to have uh a good level of uh prompt injection guardrail uh surface area uh if they’re sort of giving this out to c consumers uh for example some customers of ours uh might want to automate something on Amazon and what if somebody puts uh a listing on Amazon which has some hidden instructions and how do you prevent against those kind of things uh and that’s where uh we work with our customers to build the right

49:09 kind of uh guardrails before the uh instruction even reaches the hardness. Um so yeah I think I would put like three layers here. One is as the as the query is even entering our surface area we control that at the skill layer we make sure that our skill is not uh tricked and at the infrastructure layer we have network restrictions and action restrictions. You can even do something like what codeex or plot code does u like always

49:39 allow or allow once or allow any action. Uh you can this is what like browserbased safe mode offers out of the box. I know browser is a crazy threat vector when you think about it like it has access to anything on the wild web. Uh and take that super seriously from a prompt injection POV. Amazing. I’ll wait for 10 more seconds if there’s any other questions. This was fun. I really enjoyed this. I love the kind of questions as well. Uh

50:15 and yeah, I will put the QR code back on screen in case any of you want to grab this. Yes. Yeah. >> Yeah. Yeah. Exactly. Yeah. I think we talked about it a while back. What you

50:49 could do is like I used uh entropic managed agents quite a bit. Uh so I would run a chron job on managed agents where I would say use this skill every day at 9:00 am to do this job and if this is a success all good if this is a failure send me a notification here that will trigger a healing loop uh so that’s how I sort of can bake in uh checks and balances around websites changing antibbots changing etc.

51:22 I’m going to hang around here. Uh also uh I’ll be at the browserbased booths all days of the conference as well. So uh in case you’re curious and want to chat more uh love to do that. Thank you. >> [music]