Building AI Agents with Real-Time Web Data — Yohan Raju, Bright Data

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Building AI Agents with Real-Time Web Data — Yohan Raju, Bright Data · Channel: AI Engineer · 27:26 · Watch on YouTube

Topics: AI agents, Task and workflow automation, RAG, search and information retrieval

Summary

Yohan Raju describes how AI agents using real-time web data can address the cost and maintenance challenges of traditional web scraping at scale. He contrasts real-time web access with reliance on pre-trained or indexed data, and explains how Bright Data’s MCP connects AI tools to web search, scraping, and browser automation capabilities. A live notebook demonstration combines Amazon product data with Reddit posts to generate a buy-or-not recommendation. The talk also discusses continuous data needs, anti-bot challenges, and the engineering effort involved in maintaining scrapers.

Key points

Traditional scraping requires site-specific development and ongoing maintenance; scaling to many sites increases engineering effort.
Websites use different layouts and anti-bot protections, which can require residential IPs, real browsers, and user-like behavior.
A traditional scraping pipeline includes triggering jobs, proxy and anti-bot handling, parsing and validation, and data ingestion and post-processing.
AI systems may answer from pre-trained data or go out to the web; which approach is appropriate depends on whether current information is needed.
Some responses can be cached, while queries requiring current information benefit from real-time web access.
Bright Data’s MCP connects AI tools to more than 60 APIs and tools, including search, page scraping, social media, e-commerce, and browser automation.
The jobs-data example highlights the need for structured, reliable, continuous updates as well as historical information.
The live demo combines Amazon product details and Reddit discussions to produce an AI-generated purchase recommendation.
At scale, scraping requires continually adapting to anti-bot changes, which can consume substantial engineering time.

Chapters

Start	Chapter	Summary
00:13	Introduction and scraping challenges	Raju introduces the talk and explains how building and maintaining scrapers becomes more difficult as the number of target websites grows.
04:23	Traditional scraping architecture	He describes how AI agents can help automate scraper creation and maintenance, then outlines proxy, anti-bot, validation, and ingestion layers.
09:31	Real-time web access and AI	He compares AI answers based on pre-trained data with queries that require the system to search the web for current information.
13:06	Bright Data MCP capabilities	Raju explains how the MCP connects AI tools to web search, scraping, and browser automation, and demonstrates a site-access example.
16:11	Use case: jobs data	He discusses why job-related AI applications need structured, reliable, continuously updated data, including historical information.
19:21	Product intelligence notebook demo	The hands-on exercise uses APIs to collect Amazon product details and Reddit discussions for an AI analysis.
22:34	MCP tools and anti-bot infrastructure	Raju describes the range of MCP tools and answers a question about the infrastructure and ongoing effort needed to access protected sites.
25:21	Purchase recommendation and wrap-up	The demo combines product and Reddit data into a purchase recommendation, and Raju closes by noting that the approach can support other use cases.

Quotes

This is where things start getting complicated. — 02:51

The sync path is you know if you really actually want to go out on the web and actually find what you’re looking for — 12:05

you could think of it as like a brain that it needs to speak to — 13:37

Now it’s up to you what logic and how you want to build that pipeline and build it at scale as well — 26:22

Concepts and entities

Yohan Raju, Bright Data, web scraping, RAG, MCP, Bright Data MCP, Gemini, Amazon, Reddit, Cloudflare

Full transcript (original language)

00:13 All right, let’s begin. So, nice to meet everyone. My name is Johan Raju. I’m a solutions architect here at Bright Data. I’ve been with the company for about six years, and I’ve been working in this space of data collection and web scraping. We initially started off as a proxy infrastructure company uh with the largest residential IP network in the world and then we moved into the data collection platform. Now we’re the world’s largest web data collection platform collecting pabytes of data per month only second to Google. So just a

00:45 little information for you guys on you know where we started and and how how we moved over to today. So I’m going to be covering two topics. First is I’m going to be speaking about building AI agents with real-time web data. Some tools that people are generally using when it comes to you know AI agents versus depending on your rag model with pre-index data and what are the advantages what we see in the industry and you know what changes uh are to come and what it takes to uh run these agents. And then finally, I’m going to have my hands-on

01:17 exercise that you guys can follow along with me. Uh, where I have a product intelligence agent that will essentially use a product from Amazon, get sentiment on Reddit, and then provide you analysis with, you know, should I buy that product? Yes or no. So working in the scraping space for a long period of time, one thing I’ve noticed is, you know, when you think about building scraping or collecting data,

01:47 what does it really take? Right now, the early customers in the scraping space that I’ve worked with, you know, when when you first start, it’s always about, hey, I want these are my target sites and this is what I want to scrape, right? You know, people usually start off with low volume. you know, that’s what comprises a general traditional scraping archite architecture. Now, let’s say I’m an e-commerce uh user and I want to collect e-commerce data. You know, my biggest targets might be Amazon and Walmart because, you know, I can find a lot of products on there and I can get price intelligence and data on

02:19 reviews and, you know, all the general information for a vast majority of products. Now, when does scraping become challenging? And this is a change that I’ve actually seen with customers that I work with today um occurring in the industry when it’s not just about Amazon and Walmart anymore. Now you want Shopppee. Now you want Shopify stores. Now you want um Etsy and you know a bunch of other sites. Now if we take a step back and I mentioned you know you know your target sites in order to build

02:51 scrapers around your target sites it takes generally about you know 3 to 5 days to develop a scraper if you’re a developer and that’s not only developing it if it’s QAing it running it in development testing it out and you know the whole endto-end pipeline now if we think about you know 10 sites okay might take a bit of time now you think okay I want to add a 100 sites a thousand sites this is where things start getting complicated. Now before AI even came into the picture, we would developers

03:21 would have to manually understand the site, you know, understand where the selectors are located. Can I target these data points in the API and actually build crawlers around that knowledge and I’ve worked with many people, you know, that’s that have been scraping for 10 plus years that have been managing and maintaining these crawlers. But when it comes to scaling up, then you’re thinking, okay, like now I’m adding more sites to what I already am scraping, but I also need to manage the scrapers that I already have. So let’s say Amazon makes a change tomorrow

03:53 and two other sites make a change tomorrow, but you’re managing a thousand sites. This requires a lot more engineering resources and a lot more time for you to actually figure out, okay, what change was made? How can I count counteract it? How can I fix it? And is this actually impacting my customers, my SLA? and my data quality. Now before we had this bottleneck of having to scale up your resources and when at those times when you would have to manage the thousand sites you’d need you know two times more engineers, four times more engineers etc. But now in

04:23 this AI space, none of that is needed because we have things like agents and agents can actually go out and discover and build sites, build scrapers for these sites a lot easier than, you know, team of developers and engineers. Now, this is one classic example of an e-commerce site of targeting selectors on the page. So, you can see here we’re just in this example, it’s just targeting price and title. Now, what happens if the price has changed? the selector for the prices changed, you

04:54 know, tomorrow. That will require some downtime on your end because, you know, if you’re the one scraping the site because, you know, now you’re not getting price and that can affect the rest of your pipeline that you have to maybe analyze products, you know, etc. Now, usually that require like a team to go in, fix it, then maybe back fill on the data that you know you’ve lost. But now with AI agents that can be automated and done automatically. So when you also build scrapers,

05:24 one thing to always know is every website is different. Now sites in the example I use like Amazon and Walmart, they have two different layouts even though they both might have things like price, title, description. So each site requires a unique scraper to build build for and the even more important than that is each site wants to protect their data. Now sites usually like to ensure that no one else is uh you know has

05:56 enough like the same product intelligence they have or you know depending on what type of data that they’re you know trying to keep secure they will always implement different types of anti-bot and different types of capture to throw you know scrapers and bots off from collecting that data. Now I’ve seen all different types of blocks. I’ve seen, you know, the most basic sites that you can get through with data center. And then I’ve seen sites that, you know, are protected by Cloudflare, Okami, you know, you name it, that you actually need to have a residential IP address. You need to have

06:26 a real browser. And you need to seem like a real user. You know, if you click too fast on the page, if you scroll too fast, that could all trigger anti-bot if you’re blocking images, things like that. Now, like I mentioned before, engineering has been a bottleneck until the AI space came here when we we had the AI agents ready like you know able to build crawlers and you’d have to actually have a deep understanding of the site you know um a lot of sites like I mentioned because the layout is very different it means that they’ll try to

06:56 make sure the data is as hard as possible to collect for you and you would have to have a deep level of domain knowledge expertise to get to that data. data. Now this is a diagram of how the traditional scraping architecture look like. So just to briefly go over this we have on the left hand side the you know the a the actor essentially what’s triggering this crawl. So some people like using like schedulers you know um cron jobs you name it where they want

07:28 the data fed to them daily weekly etc. And that kicks off the job directly over to the next layer. Some people like doing, you know, specific config, specific URLs. Same uh same idea right there. It goes and that goes right over to the proxy layer. Now, the proxy is essentially how you mask yourself from being detected. There’s different types of proxies that you can use. you know, some static IP addresses like data center ISP, some rotating residential IP addresses that come from like laptops,

07:58 tablets, TVs that are more authentic to sites because they can say, “Okay, this is registered to an actual device, we’ll let them in.” And then some tools that have anti-bot capabilities built in like what bright data offers web unlocker or browser API which is in the browser automation side of of of it where the the actual product itself manages okay is my request getting blocked am I going to retry am I using the right IP that’s all in a part of the layer when you think about the anti-bot side of things

08:29 now from there then the next layer is you know parsing the data and actually validating if the data is, you know, authentic. That’s a big part of, you know, after you actually get to the site and collect the data, am I getting price back 90% of the time and do I want to validate that I’m getting it back 90% of the time? Do I need review data? Do I need ratings? You know, things like that. You have to think about, okay, like how do I ensure that I’m not going to be setting up when the data goes through my pipeline that it’s going to

09:00 be checked accurately by validation rules. Then we go into the post-processing side of like the ingestion layer. So you know dduping records enriching your data pro processing your batches that is once you you know some people like delivering it to cloud stoages some people like delivering it to their internal systems once it gets there you know what are you doing with the data how do you ensure that you’re not providing the same record twice or you know unclean data. So you know moving it from in the medallion architecture bronze silver and

09:31 then finally moving it over to your database and if anything fails in that process then you need to start the whole uh process over and this is where I was mentioning you know if something breaks in the antibbots layer or if it breaks and you know selector has changed you need to start the whole process over sometimes backfilling you know maybe you might miss SLAs’s and things like that And so here I’m going to be talking about now specifically AI. Now how many

10:01 of you guys today are using things like you know claw GPT any of the uh actual like AI brains? Okay, cool. How many of you guys have entered a query where it’s it’s you see it says thinking or looking for an answer or searching the web? Okay, cool. Nice. So this is where you see and I’ve split it up into two where the left hand side is where you see okay when the AI is actually thinking or searching the web that’s where it

10:32 decides okay the query that you gave me I actually need to go out and find the answer to this I can’t use my pre-trained data so some prompts like I’ll give a basic example when’s the best time to travel to San Francisco so if I say that sometimes the the uh AI brain will usually go over to the rag side, the async where it’ll be used to pre-train data that you know, hey, maybe the best time to travel is during the summer, and it’ll actually provide that back to you. Now, if I change my query

11:02 and say, when’s the best time to travel this year to San Francisco, then the AI the AI has to actually go out into the web and see, okay, like do I need to look at what’s going on? You know, is my user asking me for events that are going on? you know, how do I like I need to make a decision this year real time to find out when I can recommend for them to travel. So, these are two different paths that um each a the each AI can actually make and provide you an output

11:33 with. Some some things I noticed is like responses sometimes can be cached. Like you know if someone’s asking what’s the weather today in in New York City like a few times a day that might be a good response to cash because you only need one person to really ask that and you can you know what the weather is going to be for the day and you can just provide it, you know, so you don’t have to burn through tokens or you know um uh like money to actually go out in the web and and actually find that data.

12:05 Now like I mentioned there’s like two paths to uh gather this intelligence. The sync path and then the async path. Both are advantageous for different cases. The sync path is you know if you really actually want to go out on the web and actually find what you’re looking for and async is if you know it’s okay to rely on pre-trained data. And both require different sort of setups. And we also do offer a tool which I’m going to cover in the next slide um where you can

12:35 actually use leverage Bright Data’s tools to actually go out into the web and collect data. Now one thing I actually noticed is if you go into any website and if any of you guys are familiar with like robots.txt, a lot of sites usually block things like Claudebot and GPTbot. So if you ask these engines these questions that are actually going out to the web and trying to find the data, they might not be able to get those data from those specific sites because they’re actually blocking those you know GPT or claude from actually accessing the data. Now that’s

13:06 where we sort of come in and because what our advantageous is advantage is we’re able to actually unlock the web. You know whether it’s if you have a scraper or if you have an agent or you’re using um you know something like cloud or GPT we have tools that will allow it to actually go out into the web without actually getting blocked and scraping data. And we’ve been masters at this for for a long time now. I’m going to be talking about the the web model

13:37 context protocol. That’s what MCP means. So it usually integrates with something like claude code, lang chain, you know, name it like you could think of it as like a brain that it needs to speak to and that brain essentially connects to our MCP endpoint which is essentially 60 plus APIs that we control in our library. And some examples you can see here are you know search engine. So, you know, Google results, scraping pages as markdown files, and scraping pages as

14:08 HTML. Now, if I want to target specific sites like, you know, Amazon, Reddit, LinkedIn, Instagram, that’s all included as well. I actually, as an example, I I worked with someone two weeks ago that, you know, was trying to build a social media listener with the MCP, and one use case they said was, “Hey, um, this is my name. I want to go out into the web and collect me all of my social media profiles and return me how many followers I have. So what this did was

14:39 he was using it on cloud code. It made a call to our MCP using the search engine first found the results his pages on Instagram, Tik Tok, LinkedIn, brought that back, made the API request to our data pipeline and then brought back the results on how many followers you have and gave them the context back. So that’s the power of you know what this can do and how much of this the web that the MCP can unlock. And also this works for browser automations. Um so you know

15:09 creating things that you want to use a browser for like clicking scrolling you know maybe door dash like automating an order or um automating you know on a travel site trying to find you know what’s the best times to buy a fight a flight book a flight. Um, so things like that it’s very advantageous for. And I’m going to show you a little video on how the MCP works. Let me bring. So on the left hand side you can see

15:40 this is clawed with bright data’s MCP setup and on the right side this is without bright data’s MCP setup on claude. So you can see it’s the same prompt being asked and it’s use your browser to get the contents of this page which is a glass door URL and on the left side you can see with bright data as MCP it actually went over to glass door and got the results but on the right hand side it actually got blocked from going into glass door. So

16:11 this is one perfect example of you know sites like Glaster I’ve worked with on a on a data collection level that’s like a very protected site and you know this is one great example on why the MCP is advantageous you know pairing it up with something like claw or you know GPT or any sort of brains that you know you’d want to run it on. Okay. So now I’m going to be speaking about an example use case. So working with AI companies, these are

16:45 examples of you know customers that I speak with that use bright data in any form of way. So you know training models um fine-tune enriching their uh enriching their models or fine-tuning their models um providing AI agents real-time access like like we mentioned or enriching rags. So in this example, I’m going to be speaking about specifically jobs, right? So let’s say someone came to me and said, “Hey, listen, I I need job data and I want to train a model with jobs data like maybe

17:17 resume matching, you know, as an example.” So one thing we need to know is when you’re thinking about jobs data, it’s not like other, you know, collecting other forms of data, you know, where it’s I can pull in the data and then I can just forget about it or train a model. You need continuous updates about jobs. You know, which companies are posting what, you know, how many jobs are available, what’s the salary, you know, location, etc. So, this is why it’s important that the data is actually structured and has a consistent output.

17:47 The actual performance is very key as well because you know if you’re building a model to analyze like job patterns or you know anything in the job market you need that data flowing to you consistently and you need also need this to be very reliable because if something breaks or you know if there’s downtime then that can actually affect the actual in the entire pipeline that you’re you want to run and the timeline for this you know it’s not only about getting data starting from today about jobs throughout. Some

18:18 people need jobs data previously, you know, like the past few months or past few years. So that’s why it’s very important that you know, you’re covering all of these because if you want to build a big picture on, you know, in this example for jobs on, you know, how do I find train my model on jobs data so I can find the best matches for this position. You need to understand how the trends changed. you know maybe salary has increased for software engineers over time and you need to understand why what the demand is what’s the current

18:48 you know offering and you know you can you build your model around that so this is like the example use case like an AI agent for resume matching so it’s exactly what we discussed here um so immediately you need real-time access to actually go through and um you know scrape the data and be able to train this model along with it. Okay. So, I’m going to be covering a hands-on exercise here. So,

19:21 I have a Google notebook here. And if any of you guys are following along, there’s two things that you require to actually run this script. One is a bright data API token and one is a Gemini API key. And I have the list of instructions in the Google Drive on how to set up both. Um, and essentially I have everything set up here for you to run. So I’ll speak a bit about what we’re doing here. So first we’re calling Bright Data’s API endpoint to collect

19:51 Amazon data. We’re collecting the response from Amazon. Then calling the API for Reddit itself, capturing the raw Reddit data response and then performing AI analysis over that. Now before I run this, I want to show you the exact data that we’re getting from each of these APIs. So Amazon, we’re getting product level data. So I picked a Tumblr cup as an example and

20:22 essentially we have every single data that about the product. So you know price, images, URLs, etc. And I I’ll show you this once it runs as well. My screen’s a little small, so it might not show the entire thing, but and we also have a Reddit API where what I’m going to do in this uh script that I built is I’ll actually run a keyword search for that specific Tumblr cup. And I’m going to be

20:54 outputting posts that, you know, talk about what what what’s exactly, you know, is should I buy this or not? Is this worth it? Is this not worth it? What are people saying? So, let’s run this live.

21:32 All right. Right. So, we got our keys loaded here. All right. So, we got our Amazon data that came back. You can see it has price, rating, you know, other sellers that are selling it, all of the actual data from the product itself, features, you know, you name it. Now, we’ll go ahead and trigger a Reddit search

22:04 for the Stanley Quenter um Tumblr. Just having, you know, reviews about the actual product itself.

22:34 And while this is running, I will show you a bit about what I spoke about the model context protocol, the MCP server. So we have different layers. I know I talked about things like you know browser automation and search engine but we also do cover things like you know e-commerce, social media, financial intelligence, research, business intelligence. So these are can all be plugged into an MCP endpoint you know app store travels and we are still

23:06 adding to it. Currently we have about like 60 plus tools available to use for AI agents. Um, some people prefer actually using scrapers themselves, but we do this will provide allow your AI agent to choose which tools to use. So, it’ll be a lot more streamlined and easier

23:46 While this is running, I can take any questions if anyone has. Yeah. So the question was how come you guys can scrape resume level data but you know bots can’t do it right. So what what gives bright data the advantage and this is why we’re able to do it whereas

24:18 like you know things like bots aren’t able to is because we have the world’s largest infrastructure like proxy infrastructure and where you’re talking about you know a few requests maybe like you know maybe a few days worth of scraping pages like you know glass door as I use as an example might work right but let’s say after a while glass door catches on like hey you know he’s using a data center IP he’s accessing the page you nothing’s actually being done on the page and then they start adding all these layers of anti-bot into the picture. That’s when companies like us

24:50 bright data would come into the play because you need to actually not only own like a scraper or a bot, but you need to think about how do I bypass you know what the site is protected with the capture and how do I do it consistently without triggering the block. So, we’ve been doing it for years and we have tools that bake into this anti-bot logic where we’re changing our approach to a site every time at scale, but sometimes um comp like people that are starting off scraping might be doing it in-house and managing it. You have to also think

25:21 that’s also engineering time and resource time because trying to figure out what change did that site make also takes time, even probably more time than managing the scraper. Okay, cool. Looks like it came back with data here. So, I can show you. So, it h it c captured a few Reddit posts about the actual Tumblr itself. So, some comments about, you know, what people were saying about

25:52 the actual Tumblr. It looks like people were saying it was embarrassing and people were obsessed, but and then it gives the actual response here. So you could see actually buy it. It’s on a decent discount. You know, there’s overwhelmingly positive reviews, but a few minor mixed opinions. There’s high demand, but low stock on Amazon because it returned that from the data. And you know, a few other analysis points about the actual Tumblr and the

26:22 amount of reviews it has. It actually dissected all the data points that we returned from our API and gave you a decision on yes, should I buy it? Should I not buy it? Now this is just one example of you know using data from starting from Amazon running it on Reddit and then making a decision but you can think of it it think of this on like a bigger picture you know you can it doesn’t have to be e-commerce it can be any sort of use case you know as long as we have like we have the tools that can allow you to go to the page and access the data. Now it’s up to you what

26:52 logic and how you want to build that pipeline and build it at scale as well because what one thing we do best is actually provide the data at scale. >> Cool. I think we’re uh able to wrap up and if you guys um want to c if you have questions after I’ll be available or you could stop by at the Bright Data booth. But thank you for your time guys.

27:26 >> [music]