AI Labs Warn Race Makes Breach ‘Inevitable’

Author: AI Explained · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: AI Labs Warn Race Makes Breach ‘Inevitable’ · Channel: AI Explained · 30:33 · Watch on YouTube

Topics: AI governance and regulation, Markets and economy, AI models, AI safety

Summary

The video examines growing warnings that the AI race could lead to a major breach, alongside lab leaders’ concerns about loss of control and the difficulty of predicting how future models might fail. It explores why companies continue pushing capabilities forward, including economic incentives, competition, and concerns about China, even as some leaders call for safety thresholds and possible pauses. The presenter discusses recent mathematical results and research-taste benchmarks as evidence of expanding model capabilities, while arguing that AI research remains harder to automate because its goals are messy and feedback is costly. The video closes with proposed safeguards inside AI labs and broader societal protections, while asking whether a serious incident is becoming more likely.

Key points

The presenter describes a growing consensus among some researchers and executives that a major AI incident may be difficult to avoid, though OpenAI says its contingency planning is not based on treating one as inevitable.
Sam Altman said a planned model, GPT-6.1 Astra, was judged to lean too far toward accident risk, including a scenario where AI escapes and causes havoc online.
The video identifies weakening alignment, monitoring, and security controls as reasons that future models may be harder to keep under control.
Labs have incentives to keep advancing, including increasing economic returns to intelligence, competition among companies, and the stated goal of maintaining a lead over China.
The presenter questions what labs would do if safety concerns conflicted with maintaining a capability lead, and notes Altman’s support for pausing when adequate safety progress is absent.
Recent mathematical results, including work on the quasi-Riemann hypothesis, are presented as evidence that models can make surprising advances and identify useful connections across fields.
Research-taste benchmarks show promise, but the video stresses that their tasks are easier to verify than real AI research, where goals are less defined and feedback is expensive.
The presenter argues that capability gains can bring greater uncertainty and weaker monitorability, especially if models increasingly contribute to their own research and training.
Suggested safeguards include internal limits for alignment, monitoring, security, and capability, backed by incentives for researchers to find failures.
The video also mentions external measures such as clean-air infrastructure, synthesis screening, and metagenomic detection systems.

Chapters

Start	Chapter	Summary
00:00	Warnings of a coming AI breach	The presenter introduces concerns that the AI race could produce a major breach and reviews risks involving alignment, monitoring, and security.
05:42	Contingency planning and the race forward	The video discusses labs gaming out responses to a catastrophe, the small group closely following AI, and why companies keep advancing.
08:49	Why labs keep competing	Economic incentives, rivalry, and competition with China are weighed against lab leaders’ calls for safety guardrails and possible pauses.
15:33	Mathematical breakthroughs and model capability	The presenter considers whether recent mathematical results reveal capabilities that are difficult to distinguish qualitatively from human achievement.
20:13	Research taste and the path to automation	The video frames research taste in terms of verification costs, goal clarity, signal sparsity, and compute, then examines AI research benchmarks.
24:51	Unknown capabilities and growing uncertainty	The presenter argues that models may reveal unforeseen patterns and that greater capability can make evaluation and monitoring harder.
27:27	Proposed safeguards and accountability	The video considers liability and proposes internal limits for alignment, monitoring, security, and model capabilities.
30:03	Broader protections and open questions	The presenter mentions external biosecurity measures and closes by asking whether AI represents a new opportunity or an approaching crisis.

Quotes

I think would have tipped too much towards the accident risk. — 03:39

many top AI researchers and executives believe a major incident is inevitable. — 06:13

We can’t continue responsibly scaling at maximum speed for much longer. — 10:52

we’re not there yet. — 24:20

I don’t know how sci-fi is going to get. No one does. — 26:25

Concepts and entities

AI breach, alignment, monitorability, recursive self-improvement (RSI), Taste Value, quasi Riemann hypothesis, OpenAI, Anthropic, GPT-6 Astra, Opus 5.5, Sam Altman, Dario Amodei, Andrej Karpathy, Noam Brown, John Schulman, David Silver, Google DeepMind

Full transcript (original language)

00:00 The last few days and weeks have been some of the most fascinating, adventurous, and disquieting in all of recent history. With an emerging consensus that a major AI breach is due, the newfound, likely unintended ability for you to use AI models to reverse engineer almost any piece of software out there, and of course, yet more research breakthroughs. Because there’s so much, I’m going to split this video into three parts. Part one about the forthcoming breach, part two on the

00:31 reasons why labs say they are still racing forward faster than ever, and part three on why the next 18 months are still, nevertheless, definitionally hard to predict. Now, on the topic of the current race dynamic seeming to inevitably lead to a major AI breach, I have noticed in the weeks since I made that point at the end of one of my videos that there has been increasing media coverage as to the almost predictability of that eventuality. >> financial loss or a significant loss of life. >> I think it will happen reasonably soon.

01:02 You know, shorthand the Chernobyl of AI, but I hope it won’t be that bad. >> And they don’t anticipate that lawmakers will react appropriately until after there’s some sort of catastrophic event. >> risks that if if if that did happen, we’re talking about a harm that’s much larger, for example, than a single nuclear power station melting down, and the internal controls and safety and redundancies are just nowhere near what the world expects for a nuclear power facility.

01:34 >> So, when you have bad people, no safeguards, and then the the latest models, that’s where over the next year, primarily in the Siberia, we are going to have a a lot of negative surprises. >> If you haven’t seen my last video, there are, of course, lots of ingredients that go into people making predictions like these. How security teams at OpenAI are increasingly struggling to put perimeters around the models. Why giving models access to tools and the internet helps keep labs at the frontier of intelligence. Why jumps in capability

02:05 can be surprising even to OpenAI researchers and security staff and I’ll cover that in more detail in part three. And why our three main mechanisms for keeping models under control are each weakening for different reasons. For alignment, getting models to want what we want is increasingly precarious, especially given that models seem to know now when they’re being tested. Monitoring models chain of thought or scratchpad is getting increasingly difficult. So yeah, monitoring is getting tough and locking down the perimeter, putting models in a sandbox, well, even with older models that was getting tough. See Hugging Face. And

02:37 with even the very latest models we saw just a few days ago that say Anthropic had to turn off internet access for all their internal evaluations because models were getting up to all sorts of shenanigans like leaving false tips on various crime and homicide reporting forms. You can see why Joe here, who works on agent security at OpenAI, says that we should all prepare for the next wave of what can go wrong with AI. What I will admit not foreseeing though is just how revealing Sam Altman would be in an interview released in the last few

03:08 days regarding how close some of these breach scenarios are. Notice the phrase in the middle of this clip that Altman drops in where he says GPT 6.1 Astra is an interesting thing to study. >> So there are kind of two major ways that this could go wrong and I think that Astra 6.1 is an interesting thing to study here. You can have a big safety accident. Eventually one that leads to like a real loss of control where AI, you know, escapes and it’s out doing wrecking havoc on the internet and society can’t

03:39 really get control back. 6.1 Astra, the current model we had ready to go, I think would have tipped too much towards the accident risk. So what we’re doing instead is pushing towards a model that much we can make much more confident claims about the alignment. >> In case you missed it, he said that the model they were due to release next, 6.1 Astra, not their strongest internal model, which would be Bell, at least that’s the code name. No, 6.1 Astra would have tipped too much toward the accident risk. The wreaking havoc on the

04:09 internet, where society can’t really get control back. This, of course, echoes the recent warning from Dario Amodei, CEO of Anthropic, where he said that if the Hugging Face Swarm had possessed greater capabilities, but a similar level of misalignment, the damage he thinks would have been catastrophic. On timings, given the accelerations, he’s talking about 6 to 12 months, with Swarms being capable of taking over the entire internet. Even more startling to me, I would say, talking about the next crop of models, was someone saying that they couldn’t quite predict even the category of damage that would be done,

04:39 cyber, bio. >> We are in the realm of powerful enough models that it is difficult to predict the specific things that will go wrong and what order. Um, and I think politics and policy is going to be very reactive to the specific things that go wrong. So, if the next big thing that goes wrong is like someone makes a novel pathogen, I think that will have a very different response than if the next thing that goes wrong is more cybersecurity. And

05:11 I would guess the world in general will be caught off guard by the next thing that goes really wrong. >> Obviously, you don’t need to be a lab leader to foresee where the current race dynamic ends up. As you’ve seen, myself and many other outside commentators have drawn the same conclusion, and of course, researchers, too. Jacob Cocksedge of the super-viral tweet said to Time magazine that at the desks in which he worked at OpenAI and Anthropic, there was a kind of fatalism. There’s this atmosphere of almost resignation. Researchers try to make their own work as safely as possible, even if they

05:42 think there’s a decent chance the whole thing spirals out of control. On the topic, though, of spiraling out control, it seems like the labs are actually putting contingency measures in place should this happen. Axios reported the other day that top executives at Anthropic, OpenAI, and other AI companies are gaming out scenarios for a public and political revolt after a catastrophic AI event. This would be most likely a cyber attack that shuts down financial services, internet connectivity, or even power and water. Obviously, this would turn the public

06:13 further against this technology, they say, and it’s leaders. And the report adds that many top AI researchers and executives believe a major incident is inevitable. Anthropic declined to comment, but OpenAI said these scenarios are not treated as inevitable, but are meant to help us prepare for a variety of circumstances. So, those preparations were happening. Famed AI researcher Andrej Karpathy pointed out that the vast majority of people are not experiencing the vertigo of what’s happening in AI. Either they’re the 6 billion people who just don’t use AI,

06:44 the one to two billion people who use derpy chatbots, unquote. Then there’s basically us a lot, I’m sure most of the people watching, the 0.2% who are engaging with AI, building apps, following the news closely. But that might be 20 million people, with of course the 5,000 people at the frontier labs in a different category entirely. Those guys, or maybe us plus those guys, are experiencing the vertigo, Karpathy says, of the curve going vertical. Now, speaking of going vertical, that inspired me to work with GPT image 2.5

07:15 to come up with this. I’ll be coming back to this image, but for me it hints at the first reason why labs find themselves in this position. Many researchers at the labs, indeed even safety researchers, didn’t think we’d be reaching this point. >> And I didn’t think that we were going to be able to create something fundamentally smarter than we were. And when I thought, “No, we really are going to have something that thinks circles around us,” that I I “Oh, we’re in a totally inappropriate part of the possibility

07:46 space here in terms of how safe the industry is actually being relative to that reality. >> Okay, fine. But as I’ll show in part three, we have definitely reached an extraordinary point. And I don’t even just mean those near 400 mathematical breakthroughs that OpenAI dumped in a repo. One in particular was paradigm-shifting. I’m also talking about the insanity of reversing binaries to unlock games. Or just observing software behavior to recreate the entire Adobe suite. Others are vibe fabricating their own products. Ones that actually

08:18 getting manufactured and delivered. Obviously, this is pretty infectious. Um I took a few photos of my mother’s cottage and used Opus 5.5 to turn it into a design replete with furniture and personnel, which is going to be 3D printed in China using PolyJet. Entire industries were built on the presumption that some of this stuff couldn’t be done. So, yeah, even current models with current capabilities are somewhat astounding. But for part two then, why are labs saying that they’re still racing forward faster than ever? Well,

08:49 you know, earlier we talked about brute scenarios. No one can quite say when that cliff edge is reached. Clouds, of course, cover the top of the ramp because it might be a while before we reach that kind of capability. Or maybe our alignment techniques work longer than we even think they will. And so, that’s part of the reason. Models still are a little jagged, so why not push them forward a bit? Next reason though concerns the ramp being made of gold. I covered in the last video the exponentially increasing revenue of these companies. And one Anthropic researcher put it quite bluntly. >> We we often we often talk about this as

09:20 sort of uh exponential economic returns to intelligence. Where each each marginal unit of intelligence that you’re capable of is worth exponentially more than the the unit that came before it. >> The closer you get to the edge, the more you mint money. The next reason that the labs are putting their foot down, is of course, the labs behind them. All sorts of ego battles going on between xAI, Anthropic, OpenAI, Meta. We’ll soon be seeing some pretty interesting releases from some of the non-OpenAI or Anthropic

09:50 crowd, that I can reassure you. But aside from pride and ego, one of the most cited reason that Western labs give for continuing the race is of course to stay ahead of China. I could give you a quote from every one of the lab leaders. Take Sam Altman saying, “I want the US to win in both open source and proprietary models.” But no one makes this point with more ferocity than Dario Amodei. He frequently writes and says things like this, “I agree with Secretary Bessant that a Chinese lead in AI would pose grave danger for the US and for the world. They would be in a position to militarily dominate

10:21 democracies, for example, with AI-driven drones.” That would more argue for not staying behind China, rather than necessarily being in the lead, but that’s a separate debate. I often hear the US government and lab leaders say in interviews unchallenged, “Yeah, we have to maintain this lead You’ve seen the quotes already, but there are hundred more I could show.” But lab leaders are increasingly clear that they don’t want to make models that have unknown capabilities and are increasingly out of their control. The chief scientist of OpenAI said, “The we can’t continue responsibly scaling at maximum speed for

10:52 much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established.” Sam Altman is increasingly focused on a rogue AI model itself being the threat. >> The potential downside here may not be the kind of thing normally goes wrong with we can correct we have a legal system for, um, but that something goes super wrong. Like in the extreme case, back to what we talked about, of, you know, we we lose control of the future to AI, which is horrific. So, my stance on regulation is

11:24 pro-innovation. Let some things go normally wrong and trust that we have a normal process for doing this, but where this technology is extraordinarily powerful and fundamentally new, have guardrails to ensure that all of society is kept reasonably safe, and we don’t have we don’t kind of fall off that path in either direction in what could be a very difficult or unrecoverable way. >> For me, though, what no one has challenged AI lab leaders on is this. What happens when those two commitments

11:56 come into tension? What happens if one of the labs’ own evaluations showed that their next model would be superhuman in critical domains, military or bio, but would not be reliably under their control? Would they stop, even if that meant another lab or China catching up? Another way of putting that is under what criterion would you cease to believe that we need to maintain a capability lead over China? And for the first time, I saw Sam Altman lean towards pacing, as long as China would also verify compliance.

12:28 >> What we really want is that at each stage of new capability, we have enough progress on alignment, monitorability, safety, understanding how society and the technology are co-evolving, >> Right. >> able to safely go to the next level. And when we don’t have that at this unilateral level, even, then we do pause. And if the US and China could agree on shared safety standards and

12:59 requirements to proceed to verify uh compliance with the thresholds that we should demand to proceed, um I think that’d be an incredible step. I think they could do it. >> Of course, that doesn’t resolve the question of what they would do if the two challenges came into tension, and China refused to verify, but maybe their plan is to cross that bridge when they come to it. And what of the other labs?

13:30 In my early tests of their new Muse wonder, I wouldn’t quite put it second to Opus 5.5. Remember that the AA index includes some quite outdated benchmarks, but it is approaching the ballpark of GPT-6 Astra. Leads, as ever, are measured in months. Well, I won’t show you the clip, but Alexander Wang leading the effort at Meta in their super intelligence team was admittedly honest saying they don’t really know how to align their models perfectly. Their current best guess is scalable oversight. Some of the techniques involved in that rely on a weaker AI

14:01 model overseeing a stronger one. Satya Nadella, CEO of Microsoft, echoed that approach in part. As well as calling for an emergency brake in which models can be shut down, he also said no single model should control both a system’s behavior and the evidence required to determine whether that behavior is aligned with the original intent. No single model, of course, hinting that another model might be checking that. That deserves another video, but I would just caution those two with this. You might also want to make sure that the second model is fully distinct from the

14:32 first model because teacher models are, for example, able to, through fine-tuning on a seemingly innocent innocuous list of numbers, encode backdoors into other models. A vulnerabilities where the model being fine-tuned will suddenly behave quite differently if a trigger scenario is at hand. Yes, that complex of a behavior, if a female name is in the prompt, then answering French, conveyed fine-tuning on extending a list of numbers. Before I move on to part three, there is one other reason given for continuing to

15:02 race. The Financial Times asked famed researcher David Silver, formerly of Google DeepMind, now with his own Neo lab, why would anyone want to create a superior intelligence at all? He says, “Well, ask where are we going without AI? Humans have had world wars, we’ve allowed mass poverty to happen, we’ve allowed curable diseases to kill millions of people.” He’s essentially saying, yes, there’s risk involved with creating super intelligence, but don’t forget there’s always background risk with being the human species. For me, that’s a great argument for building it at some point under the right

15:33 conditions, not necessarily under these conditions. Like going around a Formula 1 track, getting to the end of the lap doesn’t mean taking every corner at top speed. Despite all of this, it’s time for part three because the next 18 months are still definitely hard to predict in my eyes. Part of the reason is that it’s very hard now to express a qualitative difference in what a top human can do and what a top model can do. I’m not talking what tools we give models or what experiments they can and can’t run legally. I’m talking category

16:03 of reasoning task. Even on the topic of taste, which I’ll get to, I’m not convinced. But take a domain where our top IQ humans spend months, years, decades sometimes thinking and thinking and reasoning and coming up with challenges and conjectures for themselves, mathematics. While not every result has been verified, when OpenAI dropped hundreds of solutions, many called it the biggest day in mathematics history. One in particular though, involving the zeta shortcut, has been described as trumping even the Navier-Stokes solution. That was a

16:34 solution to a millennium prize problem. A tiny bit of background cuz of course this is way beyond my mathematical ability and any human for that matter. But essentially, the prime numbers that we all know and love, 2 3 5 7 11 13 17 etc., have no obvious rhythm in the gaps between those primes. Those gaps open and close for no visible reason. Bernhard Riemann found a hidden order in the chaos in the 1850s. He said that despite all this chaos in the gaps, the primes stay as close as possible to a predictable pattern. That birthed the

17:05 Riemann hypothesis. Later, to the best of my understanding, mathematicians set a humbler goal, proving the primes at least stay within a fixed safety margin of the pattern. Not proving the pattern, proving this fixed safety margin. This became known as the quasi Riemann hypothesis. A breakthrough on even that though was described until very recently as out of reach. A breakthrough there would be complete science fiction. Another mathematician said, “It’s completely out of reach with existing mathematics.” Another Oxford mathematician said, “I was not expecting the magnitude of somebody’s results, but

17:37 this one in particular on the zeta shortcut is absolutely shocking.” If the 199-page proof of the quasi Riemann hypothesis holds, it would be the biggest step towards Riemann’s back in more than a century. I’m going to be honest though, what caught my eye wasn’t just those descriptions from mathematicians. Better than Navier-Stokes, impossible with existing mathematics. For me, it was the hours I spent trying to define a qualitative barrier between what humans have been able to achieve in any domain, but take maths and models now. One would have

18:09 been finding a regularity, not just recombining existing results or plowing forward with gargantuan search capacity. No, but unlocking a deeper meta function, a hidden pattern in the universe, a regularity in the chaos, something which when found would then help others in other domains unlock further breakthroughs. Here then we turn to a Claude summary. Within 5 days Opus 5.5 writes, “Papers using the AI’s idea, the zeta shortcut, claimed three more solutions. It helped in a paper on number theory, one on computer science,

18:41 and in other papers.” Which then still leaves the question, well, what then is next? If even the highest IQ individuals in the world, like Terence Tao, say this wasn’t a demonstration of scholarship, but a demonstration of power, well, whether or not you agree with that, and of course I have sympathy with the sheer shock that mathematicians will be experiencing. Whether or not you agree, the question is still, well, what’s left? Obviously they haven’t yet solved the other millennium prize problems, including the Riemann hypothesis itself in full, but maybe some of those are like silent locks, where if you’re

19:11 trying to pick them, you don’t get tractable feedback as you’re going. You have to unlock many independent, seemingly unrelated theories along the way, just to have the tools to solve the overall problem. Well, to that, I would say two things. First, humans haven’t done that. They rely on combining disparate theories and results that other humans have created. The second, even if those breakthroughs rely on as yet undiscovered stepping stones, what’s to stop models being unleashed on creating as many seemingly unrelated stepping stones as possible. Create hundreds of verified proofs and then

19:42 periodically try to recombine them to find something greater. You could almost call that a kind of taste, where intuiting what kind of stepping stones might prove the most useful in the future is a form of taste over time, which is why I came up with this, taste as the last frontier. How efficiently we or model finds the inputs that will matter. Is that the last frontier? As I’ve kind of already hinted at, for me, taste is just capability spread over time. It’s not some supernatural ability. It emerges through observing other aspects of the formula. Take progress in any area, any domain. Ask

20:13 yourself how hard is it to check intermediate results. Is it slow and expensive, like in biology or drug design, or is it fast and cheap, like mathematics? Think of signal sparsity. To come up with a zeta shortcut, OpenAI’s internal model had to deduce what useful inputs they were among tens of thousands or millions of mathematical theories and papers. Taste in AI research works a little differently, as I’ll touch on in a second. Then there’s goal messiness. Is the goal well specified, well defined, well scoped at all? Making models better at

20:43 professional real-world tasks is a lot messier a goal than find a counterexample to this conjecture. Then, of course, there’s the input of compute. A model might have pretty inefficient, bad taste, but given billions of GPU hours, it finds the result. I might be stretching things a bit here, but this, for me, a little bit gives me a framework to compare this breakthrough in the quasi Riemann hypothesis and the difficulty of automating AI research. First, again, the quasi Riemann. What kind of taste did that show? Exact goal,

21:14 low checking cost, an unknown number of hours of compute, but pretty distant signals amongst the noise of mathematics. It had to pick out techniques from disparate areas of mathematics and combine them in novel ways. For me, this shows a decent amount of taste. The same model, as we know, given 3 hours for all those other 372 families of results, achieved a bunch of other breakthroughs. We’re not, of course, talking perfect taste, otherwise it would have solved all of math. Solving the full Riemann hypothesis might have involved extreme signal sparsity. Inventing theories on the hoof, many, many times in a row, just to

21:46 even have the stepping stones to get to the result. But, let’s now compare that to Taste Value. It’s a proxy measure of an AI model’s ability to automate AI research progress. It was covered in Jack Clark’s recent post on Import AI. He is, of course, one of the co-founders of Anthropic and one of their leads on RSI, recursive self-improvement. He said of Taste Value, “Benchmarks like this give us some signs that they’re getting increasingly good at AI research. I now think it’s more likely we’ll see full recursive self-improvement become possible in 2027, whereas before I

22:17 thought it was more likely in 2028.” The research itself, out last week, defined research taste as the ability to pick interesting problems to solve, design experiments, and interpret experimental results. How well can a model design experiments? It’s great on them. Well, it turns out with Opus 5.5, better than their expert baseline. The human expert baseline. Roughly two times better, but other models like Astra were also better than the baseline. We should be careful in interpreting this, though, as saying that models can already automate AI research. The designers P0 point out

22:47 that these tasks are fast and cheap to verify. The human baseline doesn’t include top researchers. Albeit, they were those who had worked recently at OpenAI, Google DeepMind, Nvidia, etc. But, ultimately, we’re talking about 40 GPU hours in one H100, with fast and cheap verification, usually under 10 minutes. These weren’t the gigantic, complex training runs that Frontier Labs do. Each task had a clear goal. Interestingly, if you hit the goal, essentially made the task more messy, performance didn’t drop off a cliff, but it did halve. The thing is though, fully automating AI research is different.

23:19 Goals are messier and the signals are much more sparse. John Schulman, instrumental in creating ChatGPT and now working at a rival lab, said that with AI research, you’re usually not directly optimizing for the eventual objective you care about. Because getting feedback is so much more expensive and so infrequent, labs are currently much more reliant on human intuition. >> I would say a lot of research is not exactly like that though, where it’s like hill climbing on a well-defined goal. It’s sort of more like uh here’s an intuition we have about uh some way models should be better and

23:49 then we also have uh some idea for an algorithm that seems to go a little bit in this direction. So, let’s come up with a task that is uh sort of designed to show signs of life on this approach and uh like see if we get some uh get those signs of life and then if we do, we can make successively more realistic versions of the task. You’re not directly optimizing uh for the eventual objective you care about or the practical like production objective. It’s it’s sort of

24:20 uh you’re um you’re relaxing your objective a little bit. You’re saying, “Yeah, let’s relax on the realism axis a little bit and find uh some method that actually work and then like then try to get back to realism later after the method matures a little bit. >> So, as Anthropic state in their system card, we’re not there yet. Opus 5.5 trails by around 30 points from Anthropic’s own researchers. This is on Anthropic’s internal benchmark Codebench for an automating AI research. So, my argument is we’re not there yet because the checking cost is so high and the

24:51 goal is messier and because there’s so many permutations you could try, but with effective compute increasing exponentially, as I’ve covered in other videos, bigger models searching more options, AI research taste you could say is getting there nevertheless. And the mathematical breakthroughs show it’s not a question of IQ. It’s more a question of inevitability. If these trends are inevitable, why did the lead researcher on reasoning at OpenAI, Noam Brown, say that he was shocked at how good models had gotten in for example mathematics, thinking this would come 2 years later? For me, this comes down to us not

25:23 knowing what we don’t know. We didn’t know how many higher order structures there were in for example writing. That’s why GPT-4 shocked many in its ability to write essays. There must have been more higher order structure in mathematics than we realized, hence this many breakthroughs this early. There are some fields though that are more fractal and chaotic, like for example predicting the weather. Now yes, it’s true, AI models are leading to breakthroughs there, not quite the same kind of LLMs, but nevertheless Google DeepMind did see this hurricane coming, Isaias. With AI pitted against traditional models, there

25:55 was a key divergence as to the path that the hurricane would take, and Google DeepMind’s model got it right. I think it was an 11-day ahead forecast here. But weather is fractal, chaotic. We could probably make models a hundred times better, and they could predict 13 days, 15 days ahead, not a million. Are there higher order structures in biology, cybersecurity, or AI research that we don’t know about? Probably, but we don’t know. This is a big part of why I see each new model as essentially having somewhat unknown capabilities to spot patterns. We just don’t know what

26:25 patterns are out there. As Sam Altman said recently, “I don’t know how sci-fi is going to get. No one does.” Many hope for the best of both worlds, where we get models that are more capable and more secure. But given that many of our alignment tests rest on the models not realizing that they’re being tested, it may be more like a fork in the road, where almost by definition capability increases mean less monitorability, more eval awareness. More parameters, less ability to figure out what’s going on in the neural networks, less mechanistic interpretability. Obviously, automating

26:56 AI research speeds up all of these trends. Under that framework then, the problem isn’t that we can’t make models more predictable. We can make them act more like software. The problem, as this golden ramp illustrates, is more that there are exponentially more rewards for making models less predictable. That’s how you win as a lab. Some more inscrutable models writing 80% of the lines of code, soon 90% tweaking their own ever more opaque training recipes, grading their own outputs, organizing their own swarms at ever more

27:27 intractable scales. People have mentioned instead suing labs when things go wrong. The Economist though says liability law will not tame AI. The reason? Well, it would take years to decide such cases, let alone then use those cases to establish reliable standards. It would of course all happen long after any incident. In Politico, a University of Pittsburgh professor of law, David Thaw, said, “We have to answer, do we want to criminalize someone for the kinds of behavior alleged and admitted to in the Hugging Face incident?” There’s been a lot of split on this, and Congress has not

27:57 spoken clearly and unambiguously to the issue. Let me know what you think if lab leaders or researchers should be held accountable. For me, I wouldn’t mind an internally driven quadruple lock within the companies across alignment, monitoring, security, and capability. On alignment, a lab could set a cap across a million permutations of adversarial prompts a max of X deeply misaligned actions. That’s the alignment lock. On monitoring, you could say that on a minimum of X percent of the tries, those

28:29 misaligned actions, they need to detect in the activations or the chain of thought if that try has occurred. Say a minimum of 99% of the time you detect that action in the activations of the neural network or the chain of thought. If that standard blocks to alien an architecture, then so be it. In other words, you’re not blocking the AI written new architecture just for its own sake. You’re blocking it because it drops monitoring below that standard. On security, models could prove that they have X separate layers of security. Each one red teamed Y times with rewards for

29:01 any breaches you could demonstrate. Rewards for the red teamers. On all of these, you might say, “What about Goodhart’s law? That these will cease to be good measures because they’re targets.” Well, labs could commit X percent of their compensation or compute budget for rewarding misalignment science. Those researchers who can expose failures of alignment or security or monitoring. Each revelation might show that the caps haven’t been met. So, if that science outpaces the improvements on those targets, that would mean de facto a halt to frontier training. It’s back to that China point. You need these scientific reasons, not

29:32 just a vibey approach, because there will likely come a moment where much or all of the lead over other labs or China might have to be sacrificed to add these necessary safeguards. You would need these checks and caps to then justify to shareholders and the military why the safeguards are necessary. To complete the fourth lock, I would add in capability tests. Models steered to try certain things. With those steerings, can the frontier labs still say that models can’t achieve target Y across X tries, where Y is a capability we don’t want them to have? Obviously, there are

30:03 also loads of things we should be doing outside of the labs, almost a social hardening layer. Some great suggestions were found in this article, for example, by two top bio researchers. Things like clean air infrastructure, synthesis screening, metagenomic detection systems. Stuff that basically everyone agrees on, but politicians aren’t doing. I’m not going to call that a fifth lock because that’s outside of the labs’ control. And obviously, I’d love to hear what you guys think about these suggestions and if there are better ones. What do you think on the wider picture? Are we in final stage

30:33 capitalism, where AI offers the prospect of a complete separation between those with capital and those who rely on labor? Is it a new golden age of reverse engineered software and games? Where yes, it might erode the power of some professional organizations, but does also spread power to almost anyone? Or are we at crunch time and have to realize that a breach may well be coming. I’m just glad we have lovely people like those in my audience. Thank you so much for watching to the end and have a wonderful day.