How to Build Financial Agents That Work Across the Web - Jay Sahnan & Arik Bird, Browserbase

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: How to Build Financial Agents That Work Across the Web - Jay Sahnan & Arik Bird, Browserbase · Channel: AI Engineer · 19:08 · Watch on YouTube

Topics: AI agents, Task and workflow automation, Data, databases and embeddings, Markets and economy

Summary

The speakers explain how browser agents let financial companies access and act on websites that lack APIs or are difficult to reach, including login-protected and anti-bot-protected portals. They describe Browserbase’s verified Chromium approach for running browsers reliably at scale, and Stagehand, an SDK that combines deterministic browser controls with natural-language actions and structured data extraction. Examples involving Ramp, Bill, and Stripe illustrate receipt collection, vendor payments, and KYC verification. Live demos show the same browser workflows running across multiple company websites and sources.

Key points

Browser agents give software agents a way to read information from websites and take actions where APIs or MCP integrations are unavailable.
Ramp uses browser agents to retrieve receipts from portals and match them to card transactions; the speakers say this has saved over 30,000 manual hours and increased scale fivefold.
Running ordinary headless Chromium in a data center can expose automation through changing fingerprints, fresh cookies, and data-center network reputation.
Browserbase describes a modified, verified Chromium that removes the automation flag, aligns its identity with Chrome, and keeps the browser process alive between tasks.
Stagehand is an open-source SDK for browser control, combining Playwright-style deterministic commands with natural-language actions.
Stagehand uses the DOM and accessibility tree to identify targets, and the speakers demonstrate reducing an example input from 122,000 raw HTML tokens to about 4,000 tokens.
Running Stagehand next to the hosted browser reduces round trips and instruction latency, according to the speakers.
The Bill example focuses on using one adaptive agent across many vendor portals with different interfaces and checkout flows.
The Stripe example describes automating KYC research that had previously required people to visit government and business-information portals.
Stagehand’s highlighted primitives are observe, act, and extract; extract can return web data in a Zod schema.

Chapters

Start	Chapter	Summary
00:01	Browser agents and financial websites	The speakers define browser agents and explain why finance companies need them to access difficult-to-reach web portals.
02:07	Ramp: scaling receipt collection	The talk describes Ramp’s shift from human receipt retrieval to browser agents that log into portals and match receipts to transactions.
03:40	Reliable browsers at scale	The speakers explain browser fingerprints and data-center challenges, then describe Browserbase’s modified Chromium and demonstrate hosted browsers.
09:22	Bill and Stagehand browser control	After summarizing Ramp’s reported results, the speakers introduce Bill’s portal-automation challenge and Stagehand’s approach to browser control.
11:25	Natural-language actions and efficient page context	The speakers compare brittle Playwright scripts with Stagehand’s natural-language actions, smaller accessibility-tree context, and browser-local execution.
14:28	Stripe KYC and Stagehand primitives	The Stripe example introduces browser-based KYC verification, followed by Stagehand’s observe, act, and extract primitives.
16:32	Research demos and getting started	Live demos show Stagehand gathering information from Chipotle and multiple company sources at scale, before the speakers close with ways to get started.

Quotes

anything that you need to do on the web, your agent should have the tool called the browser. — 01:04

Ramp spins up a browser agent. That browser agent goes to Delta. It logs in. It captures that receipt, and it matches it to the transaction. — 03:08

One agent, thousands of different portals. — 10:55

Stage Hand will then be able to take the DOM and accessibility tree and everything it sees on the website, pass it to an LM, pass it to an agent, and understand where to click and go and click it. — 12:55

Those are three key primitives that Stage Hand we give we expose to you as developer. — 16:02

Concepts and entities

Browserbase, Stage Hand, Playwright, Chromium, Ramp, Bill, Stripe, KYC, verified browser, Árvore de acessibilidade, Zod, Jay Sahnan, Arik Bird

Full transcript (original language)

00:01 [music] >> Hello. Hello everyone. We are here to talk about how to build financial agents that work across the entire web. I hope you’ve been having a great day so far in New York. I am Jay. >> I’m Amarik and I think the place to start is what even is a browser agent,

00:32 right? So if we think about it and first if we want to go back to the previous slide, quick show of hands, who is building agents today? Show of hands. Okay, we’re in the right place. >> And who works on like finance operations? I heard about some conference going on today. Anyone in finance? Okay, couple more engineers. If you also want to ask us any questions throughout, you can scan this QR code and you’ll be able to ask away. >> Cool. Back to the show. So what is a browser agent? An a browser agent if you

01:04 think about it, if an agent needs to access the web, interact with the web, read information from the web, take action on the web, it needs a browser. Whether it is booking a plane ticket, booking a flight, planning a date night for you and your wife, getting some pricing information on a competitor, it typically needs a browser to be able to access those hardest to reach sites that don’t have an API or don’t have an MCP. And so in a nutshell, anything that you need to do on the web, your agent should have the tool called the browser.

01:37 Show of hands, who knows these three companies? Ramp, Bill, Stripe. They familiar with the crowd? Cool. When you think about what these three companies have in common, well first they’re all financial institutions, they’re dealing with highly sensitive information, but when you think about it for like Ramp, Bill, and Stripe, a lot of this information lives on the hardest to reach parts of the web, whether that is behind a portal where you need to log in, whether it’s on a website that is heavily protected by antibot. It is extremely challenging to get access to

02:07 this information, and the answer is using browser agents. So, if we want to start off, we’ll go with Ramp first. So, Ramp, I’m sure everyone in this room is familiar with Ramp. We actually are big users and fans of Ramp. Uh they make it incredibly easy to for a company to manage spend. When they were growing and scaling, they started having to interact with hardest to reach portals, and more portals than than ever before. And so, the problem or the gap was, how can we

02:38 fetch these receipts efficiently at scale? They didn’t know about the browser agent. So, instead, they’re like, let’s just put together a team of humans, and they will log onto these portals, and they’ll go fetch these receipts. Well, that can that can be good for a series A, series B company, but when you start growing and scaling to the to the masses that uh Ramp is at, you need a more efficient way. You can’t continue to hire humans. You’re just going to be bottlenecked by the limitation of scale. So, in comes browser agents. And now,

03:08 again, we are big users of Ramp. I booked my flight to this conference using my Ramp card. Immediately after booking my flight on that Ramp card, Ramp spins up a browser agent. That browser agent goes to Delta. It logs in. It captures that receipt, and it matches it to the transaction. Um so, once they do that, that’s that’s all good and dandy, but when you’re doing it at scale, you’re going to run into things like antibot. So, how do you reduce that friction? How do you eliminate getting blocked?

03:40 >> And to understand this, I really want to go back to how does the browser work on your laptop? Think about Chromium or Safari or dear. Chromium allows you to access the web. It’s an executable binary. And right now from this slide we can see that we are booking a flight to New York. And in the background, your browser has an entire fingerprint. It knows your browser version, uh

04:12 there’s OS, there’s fonts, there’s screen size, and even time zone. The website also sees a familiar visitor. Maybe you visit the site before, maybe you’ve got some cookies. You’ve logged in or you’ve booked too many trips on the same account. There’s a familiar pattern as well. Each request will go through a net You’ve got a network reputation. And all the navigation, you are clearly a human. And now the problem comes when you try and put that same version of Chromium

04:42 into a data center. And that’s when we scale beyond the laptop. We need to replicate it. For companies like Ramp, they are not going to have thousands of laptops open running a different human version of Chromium. They’re not going to have one person with a cron job running this either. They need scaled infra that lets them get to sites reliably. The problems start to appear though when you put that same Chromium in the data center. Um let’s say we do this.

05:13 Let’s say we’ve put that in. The problem now is when we launch a called out browser, that headless browser in an an an EC2 instance somewhere. Every single time we launch it, it’s going to have a different fingerprint. It’s going to have fresh cookies. You may need to log in again. And proxies. It’s going to clear indicator that you are coming from a data center instead of a residential IP. And sure, you could trick the website by

05:44 using a proxy service and changing your proxy. And that means is you might not be presented as many captures or you can fake your location from that data center, but that still doesn’t change the underlying problems with Chromium. You’re using that same Chromium on a laptop. And if we just step one step down and go a little bit deeper and think about what makes Chromium so good is if you look at the browser process, right? Inside you have a page engine.

06:14 This is a very simplified version. And where it renders the content and it has all the HTML, JavaScript. Um then you have your network and if you ever like inspected before you go on a network tab and you can see all the requests coming in and out. And you have your device and graphics. The engine is so great that they can make a change every 2 weeks, which makes it terrible for us as a company trying to automate the web. So we forked it. And I could go on forever on how many changes we made to the Chromium binary.

06:46 Um but we forked it and came up with a verified browser that we run in the data center. And what we did with three things I’m going to share now is we first removed the automation flag. An obvious one, right? When you’re driving a headless browser um through an automation, you will pass this webdriver equals to true and we just swap that to false everywhere. Then we aligned on the identity. Instead of now being called headless Chrome, we’re calling it Chrome. Now these may seem like simple fixes, but because how

07:17 complicated Chromium is, this goes deep into the code and it’s in so many different places that we need to find. And every 2 weeks is a new change. That’s why we have our own version. We also [clears throat] keep the process alive. What I mean by that is a lot of scraping jobs or automation jobs or workflows that people are doing as developers locally or even in the hosted infra will start and stop. They kill the browser and they kill a Chromium instantly as soon as the task is done. Instead, we leave chromium running and close the windows out.

07:49 So, why don’t we put it to the test? We’ve got a snippet here and I’m going to ask Aric, can we launch a browser? Okay. And bear in mind the Wi-Fi has been terrible today. So, it’s launch is going to take a second to render and we are now looking at an iframe of a live view of a hosted browser on BrowserStack. That’s great. We can go to Rome. Okay, nothing special. Let’s stop that. And let’s go to the next slide.

08:20 Why don’t we launch 100 browsers? Do the exact same thing but we’ll launch them concurrently. Okay. And to the next slide. Um now what’s going to happen in the background is I’m calling the same thing with a bunch of different websites and you will be able to Aric scroll through the um some of the sites and we’re not automating at the moment. We’re just going and visiting the sites as 100 browsers. Now I’m going to try something for the next minute. Um I set this up very quickly and the

08:51 Wi-Fi is a little sucks a little bit. And it’s all through an Engrok portal on my laptop. So, if it doesn’t work, uh we’ll move on. Um I want everyone to scan this QR code and you will be given a random website out of one of these 100. Um someone tell me if they can see a browser. I’m trying to prove to you that these browsers in our cloud and you have access. Someone shout out when they’re when it’s loaded. >> Hopefully the demo gods are with us.

09:22 >> Yeah, you got a browser? Awesome. Yeah? Got another one? Great. Cool. Sweet. Nice. You can actually interact with this as well. Um through the iframe you can click, control, move and do things. And that takes me on to the next thing. I’m going to uh stop that now. Thanks, Eric. Um Um let’s take me on to the next thing is control. How do we control these browsers? >> And just to sum this up, Ramp now can deploy thousands of browser agents. They

09:53 have saved north of 30,000 manual hours in data entry that is done that used to be done by humans that is now done by agents, and their scale has gone up 5x. So, just by switching from humans to agents, they were able to accelerate their business and save a bunch of those hours, which now they can dedicate to other initiatives across Ramp. When we talk about Bill, so Ramp scale and access. Now, it’s about control. Everyone’s familiar with Bill. Um Bill, you know, processes a ton of payments across thousands of different

10:23 vendors. When you think about using agents to do that, the biggest challenge is well, every portal Next slide, please. Every portal is different. The checkout flow is different. The process to fill out the payment is different. The UI changes. So, in reality, the way Bill thinks about this is it’s impossible to automate with agents. That’s why we need a team of humans to go click through portals and do this on behalf of our customers. Um and again, the answer to this is an agent that can use a browser.

10:55 Um and so, for Bill, the notion was how can we build a single agent that can interact and act upon thousands of different portals. One agent, thousands of different portals. >> Um that brings us to Stage Hand. Stage Hand is our open-source SDK that lets you control a browser. Think about Playwright, but built way on top of it. >> [snorts] >> Your agent will sit and use Stage Hand to control the browser. I’m going to show you a little bit how. Go to the

11:25 next slide. I want to start with understanding again what is wrong with Playwright. Playwright’s great. I first learned how to code by using Selenium and beautiful soup and writing Playwright scripts by hand. This was before AI. Now, I can’t remember last time I coded. And so, with Playwright, what the problem is is as soon as that website changes, the button moves, the text box changes like from first

11:55 name to just name. In the in the back, like when you’re looking for that XPath in Playwright or you’re trying to click on an element or a selector, your script’s also going to break. And now, that’s fine if you’re doing one-off tasks and you’re running everything locally, but when you try to scale like a company like Ramp or Bill, you need reliability every single time. The reason we like built Stage Hand was because we wanted to give the ability to agents to navigate the web and control them. Next slide.

12:25 So, how does it actually know? What’s different? Well, if you play play Sorry. What’s different? So, Stage Hand actually works a little bit like Playwright in some sense. You can use deterministic rules and go page.click or everything you’d see in a Playwright script. But, when it gets a little bit more complex, you can actually send natural language prompts to go and do actions on the site. Pay page.click or page.act, go fill out

12:55 this form, or enter my name, or enter password, find the red button. Stage Hand will then be able to take the DOM and accessibility tree and everything it sees on the website, pass it to an LM, pass it to an agent, and understand where to click and go and click it. Now, you may be thinking it might get expensive, but that’s where we have our excellent engineers behind Stage Hand. And to demonstrate this quickly, Eric, if you could hit that compare button.

13:27 If you were to use an agent with playwright, you’d be sending everything on the left. Uh, that’s 122k tokens. And that’s a raw HTML. And you ask your agent, “Where do I click? Where do I go to?” What Stage Hand does before it hits that agent is it shrinks the DOM to a really cool accessibility tree, where we’re only sending like 4,000 tokens for this same example. And it’ll give direct selectors. So, Stage Hand will go, “Click that.” Save a lot of money on um the amount of tokens that you’re sending, but also reduce the

13:58 context and get quicker results. And if we go next, another reason why we use um Stage Hand and with Browser Base is because there’s fewer net There’s fewer round trips. Let’s say we are currently controlling a browser in a data center. We’ve got to talk back. We send the HTML back and forth every single time. What we do at Browser Base is we put Stage Hand next to the browser. And that What that means is Stage Hand is a Chrome extension on a hosted browser. So, those actions become a whole lot

14:28 quicker, and there’s no round trip latency from those actions and those like instructions from the agent of where to click and where to go next. And so, when you bring these two things together, you need scale from the browsers, Stage Hand gives you the control, and we’re going to show you an example that use both. >> This is our last example. So, with Ramp, we saw that browser agents solve the issue of scale and access to heavily

14:59 protected sites. With Bill, it was adaptive automation. How can browser agents enable you to interact with thousands of of different portals with ha- without having to manage thousands of different playwright scripts. Now, with Stripe, everyone’s familiar with Stripe in this room. Stripe onboards thousands, tens of thousands of merchants on a monthly, quarterly, quarterly, yearly basis. And again, they knew that the only way to access the information to verify that these businesses were real and they were the right business to work with is through a

15:31 browser. And this was before the age of browser agents. So, they had a team of humans going on and going to Secretary of State portals and using the browser to get that information. So, a quick switch is being able to use the browser agent to go and automate this KYC verification. >> Um let’s dig a bit more into Stage Hand here. Um so, we have instead of using a direct agent like Bill, you could use act, extract, and observe. Those

16:02 [snorts] are three key primitives that Stage Hand we give we expose to you as developer. Um one is observe, find the learn more button. Um this is taking action. Next slide. Act, click the login button. Next slide. And then the extract. Now, this is my favorite one because you can extract anything into a Zod schema, which is great for engineers because we’re taking really messy unstructured data on the web and converting it to a really easy-to-use data and readable and send

16:32 it to an agent. Next slide. Now, I’m going to show a couple more demos. We’re going to start with one company website and we’re going to go to Chipotle and find out what it sells using Stage Hand. Um and next slide then. Okay. So, this browser will now spin up on a hosted browser and browse page and it’s going to use Stage Hand um extract to extract it. And there we go. We’ve got Chipotle Mexican Grill. Um it’s selling some lime chips. Cool.

17:03 Uh if we press stop. Now, this is where it’s going to get fun. Let’s go for the next one. And now we’re going to research four different sources for Chipotle. We’re going to go to their website, investor page, SEC filing, and New York Stock Exchange listing. Okay? So, we’re running and all right, you can also scroll once they load. Okay, great. So, we’ve got the SEC filing down at the bottom, and we’ve got New York Stock Exchange, and up top I think we’ve got the website

17:34 that will appear in a second. But, there we go, they got investor relations, as well. So, these are actually happening quicker than they are on the screen. This is already being happened in in the back. It’s just the live view that’s streaming back to us. And obviously, when you’re running so many browsers at scale, you’re not always going to be checking the live view. This is more for development. Okay, cool. Let’s stop that. And let’s go to the next one. Now, we’re going to do 20 companies for the same three sources. And we’re going to hit run. And bear in mind, this is the same code that’s running on every single website.

18:06 Then, we go to the next slide. And we can see Chipotle will come up at the top. I’m getting hungry now. Um JP Morgan. Um there we go. All right, keep on scrolling. Let’s see what else is there. Golden Sachs. Going and doing the same tasks across every single site without interruption, unblocked, all using Stage Hand as well. And it’s really cool cuz it’s a verified browser that I mentioned at the very start, which means that you don’t get hit with as many CAPTCHAs. And if you do, we

18:36 solve them. And the Stage Hand allows us to navigate without anyone looking at this. Okay? And we can stop there. And I think we can see some results on the right-hand side. Um next slide. Okay, cool. We’ve got some of the results here. Um SEC filing data. Keep on scrolling. I honestly have no idea what the SEC stuff means. I’m not a finance guy. But, hopefully it’s useful. Cool. Um let’s go to the next slide. And so, that SEC template I was

19:08 something I built a while ago. If you want access to that particular, just the SEC one, you can do. Um, but we also have if you want to get started with anything browser-based or anything I showed, we have if you go to our website, we have a copy prompt which you just send it to your agent and that’ll do a lot of the work as well. Uh, next slide. Okay. Thank you so much. That’s our talk. >> [applause]