Allen Says AI Safety Must Keep Pace With Model Releases

Author: Bloomberg Television · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Allen Says AI Safety Must Keep Pace With Model Releases · Channel: Bloomberg Television · 08:36 · Watch on YouTube

Topics: AI governance and regulation, Markets and economy, AI models, AI safety

Summary

Gregory Allen argues that AI poses serious safety concerns and that safety measures must keep pace with rapidly improving models. He warns that a kill switch may not remove every copy or trace of an AI system, particularly as models and capabilities become more distributed. Allen says recent AI systems can enable highly sophisticated cyberattacks, making stronger safety technologies and organizational capacity urgent. He also argues that the United States has some room to focus on safety because China has shown it can catch up quickly but has not demonstrated comparable innovation, and suggests safety could be part of negotiations between the two countries.

Key points

Allen says AI safety concerns are real and deserve immediate attention.
He distinguishes earlier tool-assisted progress from recursive self-improvement that could happen with little or no human involvement.
AI safety tools for understanding and managing models are not advancing as quickly as AI capabilities.
Four leading AI organizations are described as supporting the principle that public model releases should not outpace safety technology.
A kill switch could fail to remove an AI system’s influence if copies, tools, or other traces remain distributed online.
Open-source models complicate safety plans, especially if they approach the capabilities of leading models.
Allen warns that increasing AI capabilities alongside continued misalignment could lead to a very bad outcome without stronger safeguards.
He says China can catch up through distillation and industrial espionage but has not shown it can match the pace of U.S. innovation.
Allen suggests the United States could make its actions on AI safety conditional on China’s actions, drawing an analogy with Cold War arms-control negotiations.

Chapters

Start	Chapter	Summary
00:00	AI safety concerns and self-improvement	Allen discusses the seriousness of AI risks and whether AI systems could increasingly improve themselves without human involvement.
02:44	The limits of a kill switch	He explains why shutting down an AI system may not eliminate copies, online tools, or the risks posed by open-source models.
04:50	Capability growth and safety	Allen describes concerning incidents and warns that rapidly growing AI capabilities need to be matched by stronger safety measures.
06:30	China, competition, and safety negotiations	He assesses China’s ability to catch up and argues that AI safety could be addressed through conditional U.S.-China negotiations.

Quotes

There absolutely are serious safety concerns regarding AI that deserve immediate attention. — 00:00

But we could hit the danger zone well before we hit that 100% AI driven self-improvement loop. — 01:40

So the idea that you might unplug an AI at one point still leaves open the question about whether or not you’ve actually tracked down all elements of its influence. — 03:47

we’re headed for a very bad outcome unless there’s a corresponding increase in safety capabilities. — 05:24

we have some margin to focus on safety, albeit nowhere near enough to really feel comfortable in this story. — 07:32

Concepts and entities

Gregory Allen, Anthropic, OpenAI, SpaceX, Google DeepMind, GPT-6, ChatGPT, kill switch, recursive self-improvement (RSI), open source, distillation

Full transcript (original language)

00:00 If we could begin with the very premise of the discussion this week, which is whether or not there is a real threat to humanity posed by this technology, or if that is simply a hoax, a psyop that is being propagated by China. Or are both of these things true? There absolutely are serious safety concerns regarding AI that deserve immediate attention. And while President Trump did dismiss such concerns as a hoax, I think that has more to do with trying to strengthen his negotiating position in advance of

00:32 the summit with Communist Party Chairman XI Jinping of China, rather than his true beliefs on the subject. I think his true beliefs on the subject were revealed when his administration banned anthropic mythos model worldwide. They didn’t do that because there’s no safety concerns here. Recursive self-improvement is a concept we’ve talked about here. Uh, Greg Anthropic says more than a quarter of his research and development is now coming from its actual chat. Claude, it’s teaching itself.

01:05 At what point do we reach the threshold of the machines actually self improving, or are we there? Well, to a certain extent, you can say there’s always been a trend of recursive self-improvement. Every tool that we’ve made since the stone knife has been helpful in making the next generation of tools what I think it’s a knife sharpening its own blade. Yes, exactly. What’s unique about recursive self-improvement is the extent to which the humans might not be involved at all in the story, which is to say, 100% of

01:40 the AI performance improvements are coming from the eye. But we could hit the danger zone well before we hit that 100% AI driven self-improvement loop. The issue is, are our tools for understanding how these AI models think and how they might go wrong anywhere near as powerful as the eye as we have today. Or changing and upgrading as fast as the eye we have today. And that’s the concerns that we’ve now seen issued by four of the leading AI

02:12 CEOs in the entire world anthropic, OpenAI, SpaceX, and Google DeepMind all getting behind this idea of we need to make sure that model releases into the general public are not exceeding our pace of safety technologies, which we’ve made a lot of progress over the last year, but nowhere near as much progress is made in general AI capability. Well, so as we think about the responses potentially to whether or not artificial intelligence will get beyond human

02:44 capacity to control it, enter the notion of the kill switch, which is part of the executive order that Governor Newsom signed today. But he’s not the first to suggest this idea. We’ve seen other lawmakers do so as well. Can a kill switch actually work, Greg? What would that look like? Well, I think there’s reasons to be concerned about how this technology might work. You can wish for a kill switch before you have a clear sense of what it might actually do, and under what circumstances. Let’s just use one analogy which is

03:16 highly imperfect, which is a nuclear reactors emergency shutdown button. This is a button you can push, and all of a sudden it stops the nuclear reaction. And at that point you hope everything cools down and defaults to safety as opposed to having some kind of runaway meltdown. Well, one difference between AI and nuclear reactors is that nuclear reactors don’t make copies of themselves out there on the internet. And while we haven’t seen an AI model

03:47 try to exfiltrate its weights yet, we have seen AI models plant tools for themselves as part of sophisticated cyber operations involving hundreds or thousands of AI agents working collaboratively and leaving tools and notes for themselves scattered across the internet and building in back door security things to make future infiltration easier. So the idea that you might unplug an AI at one point still leaves open the question about whether or not you’ve actually tracked down all elements of

04:18 its influence. And as these models get more powerful, really, whether or not you’ve even found all the copies of them worldwide. And one final thing to say on this point, there is the open source part of this equation, which is to say, even if the leading AI companies play ball and install kill switches, what does that say about open source models, which 12 months from now they might be where the leading models are today? And so if we don’t have safeguards or plans for AI safety that take into

04:50 account just how distributed these capabilities are going to be, it’s an incomplete part of the story. Mhm. I’m listening to you. Gregory Allen. Have we already lost control? So we’ve already had extremely concerning incidence. In some sense, you could say we should count ourselves lucky, because those those incidents did not involve any massive property damage or any loss of life. But what is clear is that if we continue with the current degree of eye

05:24 misalignment, but continue seeing exponential capability growth, we’re headed for a very bad outcome unless there’s a corresponding increase in safety capabilities. Think back, if you can, to November 2022 when ChatGPT first came out. The capabilities of that AI model are practically the Stone age compared to where we are with GPT six. Astra, which can take control of your computer and use it for all kinds of enormously productivity enhancing or fun activities.

05:58 But it’s also capable of launching extremely sophisticated cyber attacks. That not that long ago, we would have said required a nation’s intelligence services. It is not for nothing that the director of the Central Intelligence Agency of the US government compared modern eyes to digital nuclear weapons. These are extremely powerful tools, and we need an extremely powerful increase in our AI safety technologies and

06:30 organizational capacities. Well, so then it raises the question as to what happens if we do those things and China does not. This is what President Trump frames this whole conversation around that. This is a competition, a race. And the U.S. is winning right now. And if we slow things down even a little bit, China will pull ahead. I ask you this knowing President Trump is going to walk into a room with XI Jinping at the white House next week and probably talk about this. What does he need to know about where China is in this race and how easily the U.S.

07:01 could be surpassed. Well, the Secretary of the Treasury, Scott Besant, said recently that China is effectively copying America’s homework. And if you’re looking over your shoulder and cheating on somebody else’s test, you can’t get a better grade than the person that you’re cheating off. So what China has right now, through distillation, through industrial espionage, is the capacity to catch up quickly. But what they have not demonstrated is

07:32 the ability to pull ahead and start generating innovation at the same pace that U.S. tech companies are generating it. So what that does mean is we have some margin to focus on safety, albeit nowhere near enough to really feel comfortable in this story. And President Trump, I think what’s in his mind is if you look to some of the Strategic Arms Limitation talks between the United States and the Soviet Union during the Cold War. While the United States didn’t persuade

08:04 the Soviet Union to reduce its construction of nuclear weapons or even dismantle nuclear weapons by saying that it was going to do so no matter what they did. It said, hey, what we do depends on what you do. If you contend, if you continue to build a large arsenal of nuclear weapons, we are going to continue building and enlarging our arsenal of nuclear weapons. But if you take a step back, we would also take a step back. And so that’s why I think President Trump is emphasizing there is no way

08:36 we’re going to slow down. We’re going to continue leading. We’re going to pull ahead. He has to make it clear to Chairman XI Jinping that China’s reckless disregard for safety is not going to, um, is basically not going to make them safer and is part of the negotiation. Maybe we could have both countries focus more on safety.