Building the Safety Layer for AI Agents

Author: YC Root Access · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Building the Safety Layer for AI Agents · Channel: YC Root Access · 10:01 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, Markets and economy, AI safety

Summary

Raindrop helps companies detect problems with AI agents in production and is extending that capability to identify potential issues before changes reach production. The founders describe how more capable agents use more tools and access higher-stakes systems, increasing both the complexity and cost of failures. They argue that standard LLM-as-a-judge approaches are costly, limited to known issues, and struggle with the context-dependent distinction between right and wrong behavior. The company began as an internal tool for the founders’ coding agents and is now launching Raindrop Simulations, which checks code changes for ways agents might fail.

Key points

Raindrop detects agent failures such as tool-call and response mismatches, capability gaps, task failures, hallucinations, and broken loops.
Companies use agent metrics to assess how well their agents perform.
As agents become more capable, they use more tools and access critical data, raising the complexity and potential cost of failures.
Running an LLM judge on every trace can significantly increase costs, while sampling can miss problems.
What counts as correct or incorrect agent behavior depends on what a company defines as acceptable.
Raindrop aims to proactively surface issues customers did not already know to look for.
The product originated as an internal tool for the founders’ coding agents.
The founders say loosening permissions and connecting agents to production systems increases the stakes of failures.
Raindrop Simulations will bring production issue detection into CI, checking changes for potential problems before they reach production.
The company says it is hiring across engineering, marketing, sales, and other roles.

Chapters

Start	Chapter	Summary
00:00	What Raindrop does	The founders introduce Raindrop and explain how it monitors agents for production issues, with examples of failures and the metrics companies use to evaluate agents.
02:37	Why agent failures are hard to catch	The discussion covers the cost and limits of LLM judges, the difficulty of defining correct behavior, and Raindrop’s proactive detection of unexpected issues.
04:45	From coding agents to a company	The founders recount how Raindrop started as an internal tool and explain how more capable agents, broader permissions, and higher stakes drove demand.
07:51	Raindrop Simulations and hiring	The founders introduce simulations that check changes for potential agent failures before production, then discuss the roles the company is hiring for.

Quotes

we detect issues with agents in production and now prevent those issues from getting into production in the first place. — 00:00

as capabilities go up, complexity goes up and the cost of issues goes up dramatically — 02:05

we sort of think of ourselves as an alignment company, aligning humans inside of the company with their agents behavior. — 03:10

Raindrop by having powerful signals but also having proactive issue detection you’re finding issues that you didn’t know about. — 04:12

every single change you make, any PR you make, we show you how it might go wrong before it ever hits production. — 08:23

Concepts and entities

Raindrop, LLM as a judge, traces, evals, CI, OpenAI, Anthropic, Opus 4.5

Full transcript (original language)

00:00 [music] I’m excited today to welcome the founders of Raindrop on closing a big series 8 led by CRV bringing your total funding to $50 million where Lightseed did your seat round. So this was just two years after the badge. It’s pretty cool. Tell us what do you guys do? Yeah, really simply, we detect issues with agents in production and now prevent those issues from getting into

00:31 production in the first place. >> So, who are some of your big customers right now? >> So, we have a number of really amazing AI agents that we work with companies like Verscell, Clay, Framer, Speak, etc. And then al also Fortune 500s of multiple industries. Yeah. >> Do you have a concrete examples of how these customers use you? >> Yeah, there’s all sorts. So, one common one with customer support agents is basically a tool call agent response mismatch. So, let’s look at something really specific. A customer support agent is trying to issue a refund, but

01:04 the tool call to issue the refund fails and the agent says, “Issued your refund.” So, that’s a problem. >> That’s definitely one right around like an agent saying it took an action and it didn’t. There’s, you know, capability gaps like the user asking for something the agent can’t do. task failure, like the agent isn’t able to complete something for some reason. And we really just have hundreds of these sort of metrics. And what we see is that companies like the entire company use these metrics to determine how good or bad their agent is. >> So you guys basically are watching

01:34 whenever agents go rogue, whenever you have problems like hallucination, errors with tool calls, broken loops, or any of these kinds of issues. What is very interesting to me and counterintuitive is that as model capabilities increase, it seems like the big labs should solve this, but it’s the opposite problem. You guys are seeing more demand than ever with the newest models. Why is that? >> Yeah. So, you can see with hugging face, for example, right, that as agents get better, they’re calling hundreds of tools. They’re reading and writing from

02:05 critical data sources. They’re being deployed in healthcare, the military, and finance. And so as capabilities go up, complexity goes up and the cost of issues goes up dramatically and you have all these failure modes that you never anticipated. >> And a regular LLM as a judge doesn’t cut it. Why is that? >> I think there’s really two reasons. The first is easy in its cost. Uh I think that if you were going to have an LLM look at every single trace, it would double your costs at least. I think the

02:37 second is actually a lot trickier and something I think our company sort of exists to solve which is that binary classification is very very hard and so we were talking earlier there’s this paper from open AAI uh maybe two years ago or something now where they said you know hallucination is a binary classification problem like something is either true or it’s not true and there were all these people on Twitter that were kind of like wow open AI solved hallucination because it’s just binary and it’s like no actually binary classification is extremely hard like every choice in your life you could say is either you should do it or you should not do it. And I think that similarly

03:10 something in your agent, some behavior it’s doing is either wrong or it’s right. And it’s really up to the company to define what’s wrong or what’s right. And so we sort of think of ourselves as an alignment company, aligning humans inside of the company with their agents behavior. So I think the cool thing that you do then is that you enable your customers to basically watch the full traces of all of their agents rather than just doing sampling and then getting a judge to run which is how people do it today but it doesn’t catch all the errors and you guys basically catch all of them.

03:42 >> Yeah. And I think it’s more than just catching errors. It’s proactive issue detection. So [clears throat] when you have LMS as a judge again you can only run it on 1% of data 10%. But also you can only look for problems you know about ahead of time. You tell your LM look at this you know rate the answer on like XYZ parameter I care about. But what Raindrop does is proactively find issues you didn’t know about. Right? So if you take the OpenAI like hugging face

04:12 uh problem these agents are talking they have all these like tons and tons of logs and it’s going unnoticed for weeks because you didn’t know to look for it. and raindrop by having powerful signals but also having proactive issue detection you’re finding and you’re you know our customers are finding issues that are creating risk creating liability and also creating excess token costs like across things they didn’t know about what’s interesting >> this journey sounds pretty straightforward but the reality is when you guys went through the batch about two years ago you were a very different

04:45 company when I interviewed you guys you were building basically a coding agent for the terminal before agents were even a thing. That was not even in not even the spoken world in terms of how people thought about agents, but you were bu basically building a coding agent on the terminal. >> Yeah. >> So tell us what happened. >> Totally. Yeah, we were I mean we’ve all been best friends for many years and we were all just hacking on things that we wanted for ourselves and very early we found ourselves wanting what are now called coding agents. agents or like

05:17 basically at the time we’d say chat bots that can do things and this was like the earliest way we thought about agents and as we were building those coding agents we ran into this issue ourselves where we had no idea why these issues were popping up or what was going wrong and so what’s now raindrop started as an internal tool um that we used for our own coding agents >> agents really had a very slow trickling since 2024 it was not even a thing until probably 2025 >> and it really started to work around

05:47 clock code opus 4.5 with aentic coding starting to work. >> Yeah. >> And that’s when things took off for you guys. Basically your revenue went like this. >> Yeah. >> So tell us what what happened is like more coding a agents created more more issues for people to follow up. >> It’s exactly what Zan was saying before is that like the more capabilities the more complexity and the more ways that they go wrong. I think what you also saw with Opus 4.5 is that the the sort of guardrails start coming

06:18 off, right? The permissions start coming off like everyone’s like, you know, dangerously skip all permissions. Everyone’s like, you know, oh, maybe we should hook it up into production. Maybe it should have access to production logs. And so it’s exactly the second part of what was saying is that the stakes get higher. >> What was actually really cool about all of you is that as you mentioned, you are very close friends. And this was basically your second company. Your previous one was a crypto company that you sold through Coinbase and you were basically had this company kind of not making a lot of progress for a bit until things took off last year.

06:50 >> What what made you keep going? >> Well, I think one of the biggest things is like getting to work with each other. Honestly, >> like when your life is you wake up in the morning and you walk to an office and work on the most interesting problems in the world with your best friends, >> you’re like life is pretty good. Uh [clears throat] like you’re learning a lot. you’re creating things that you genuinely want to use. And I think we just had this also conviction in the future where like these AIs are going to do things and eventually, you know, people are going to hook them up to production databases. Eventually, they’re going to be issuing prescriptions. Like I think it was

07:20 obvious to us very early that these agents would basically run a lot of the economy and now that they are, people are realizing that it’s kind of a hair on fire problem. >> Yes. I think that we definitely frame things a lot as just like fundamental truths. Like I think the our fundamental beliefs and everything else around your fundamental beliefs as a company can change. Like the way I like in a year our product will look entirely different. I have no idea. But like the things we know as a company are that like one agents will get more capable and two that the cost of their mistakes

07:51 will increase. And so if both of those things happen then people will pay us money to make sure those things don’t happen. So, you guys have a pretty cool announcement that you’re solving a very hard problem that was described by Darkeesh’s post on the rise and fall of agent civilization that basically describes how hugging face was pawned by the OpenAI agents and you guys have this new solution that’s going to solve this. Tell us more about that. >> We’re going to be launching raindrop simulations. So what this does is every

08:23 single change you make, any PR you make, we show you how it might go wrong before it ever hits production. So we take all the issue detection we do in production and bring that to CI. We try to find cases where things might go wrong that you haven’t even imagined. And so this is a very very different strategy than eval which mostly are testing things you already know about. And this really today, OpenAI and Anthropic are the only ones that have access to this sort of technology. Um, and we’re bringing that to every company in the world.

08:55 >> Pretty cool. So you guys can detect the rise and fall of ancient civilizations. >> Yes. >> Quite literally. I think that that is an extremely good example of something where it’s like, sure, now that you know about it, you could add an eval, right? But the reality, and this is true for most companies, if it’s true for OpenAI, is that that sat in logs for weeks before anyone found it. It took, you know, uh crashing their internal package manager um before it was detected. >> So tell us about the roles that you’re hiring for. >> Yeah, hiring across the board, right?

09:29 This is such an important problem. Hiring across engineering, I think these are some of the most interesting engineering problems, period. Right? As Ben said earlier, we’re taking some of the tools that only OpenAI hat and Anthropic have in-house and just making them available to the entire world. Does agents proliferate? And um also in when we think about agents proliferating and getting these tools to everyone in the world, every developer in the world, that requires a huge distribution in sort of like GTM. And so

10:01 we’re hiring a lot for everything from marketing, sales, devril, etc., etc. Yeah. All right, guys. Congratulations. >> Thank you. [music]