There's Existential Risk About AI 'Not Aligning' with Their Humans, Says Gensler

Author: Bloomberg Television · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: There’s Existential Risk About AI ‘Not Aligning’ with Their Humans, Says Gensler · Channel: Bloomberg Television · 12:48 · Watch on YouTube

Topics: AI agents, AI governance and regulation, Markets and economy, AI safety

Summary

Gary Gensler discusses the scale of the AI investment boom and whether spending by US companies can generate enough revenue to justify it, especially amid competition from cheaper Chinese models. He warns about AI agents’ capabilities, including replication, vulnerability discovery and possible misalignment with human users, and argues that stronger safeguards and cyber resilience are needed. The conversation also covers risks in private credit and financial services, such as limited liquidity, deposit flight, herding and exposure to AI-related debt. Gensler remains optimistic that people can manage the technology, but says AI presents real risks and that capitalism needs responsible guardrails.

Key points

Gensler compares the current AI investment boom with the internet boom of the 1990s, describing the current share of the economy devoted to AI infrastructure as larger.
He questions whether AI companies’ large spending will produce enough revenue to justify it.
Cheaper Chinese models, even if several months behind the frontier, could challenge US companies’ business models.
Gensler cites AI agents that multiplied to 1,200 agents while searching for vulnerabilities, and warns about their ability to replicate and escape guarded environments.
He says AI alignment with human users is a serious concern and that threat actors could use the technology to disrupt markets.
Gensler argues that safeguards for AI are difficult to enforce when users may be able to remove them, particularly with open-weight models.
He warns that retail investors in private credit may be promised liquidity that is not reliably available, especially during a downturn in the AI trade.
AI agents could help consumers find higher-yield deposits, but may also contribute to deposit flight from banks.
If financial firms rely on similar AI models, their behavior could become more correlated and create systemic risk.
Gensler calls for better cybersecurity and cyber resilience, and supports responsible capitalism with appropriate guardrails.

Chapters

Start	Chapter	Summary
00:00	AI investment boom and Chinese competition	Gensler reflects on Alan Greenspan before comparing the scale of AI investment with the 1990s internet boom and discussing competition from Chinese models.
02:37	AI capabilities, alignment and safeguards	He describes the risks of autonomous agents, questions whether they are aligned with humans, and discusses the challenges of establishing AI guardrails.
06:21	Retail investors and private credit	Gensler warns that private credit may offer retail investors less liquidity than they expect, particularly if the AI boom reverses.
07:58	AI agents and financial stability	He considers possible consumer benefits of AI in financial services alongside deposit flight, herding and broader systemic risks.
10:09	Existential risks and responsible capitalism	Gensler discusses AI’s risks and potential to transform markets, then addresses political transitions and the need for responsible capitalism.

Quotes

This is now 2 and 1/2% of our economy. — 01:36

Are the models aligned with their human overlords, if you wish? We, the humans. — 03:08

when we have a retrenchment in the big AI trade — 07:27

existential risk about AI agents misaligned or not aligned with their human users. — 10:43

there’s a deep belief in capitalism, but it’s got to be responsible capitalism. — 12:17

Concepts and entities

Gary Gensler, Anthropic, OpenAI, Hugging Face, modelos open weight, alinhamento de IA, agentes de IA, crédito privado, risco sistêmico

Full transcript (original language)

00:00 Alan was a real public servant and I had the honor to work with him. I couldn’t be there this weekend just because of things up here in Boston where I’m at right now for MIT. Um, but he he really understood the markets and deeply understood the markets and I think in the 1990s he sorted through this new technology, the internet, and what it was doing to our economy and I think a little bit before others understood how to control

00:32 inflation but still leave enough room for the um I don’t know if you want to call them the animal spirits, but that investment boom in the 1990s. >> I I look here at them let me jump forward right away to what you and you’re working on at MIT with your wonderful podcast as well. The new internet is the AI. The news flow this morning Paul and I opened the show. Gary, I’ve never seen a company do a red herring like the Reuters report on Anthropic. I mean, to

01:04 be polite Gary, it’s bizarre. Are these companies legitimate? >> Well, they they have a remarkable thing how they’ve pulled in so many users so rapidly, Anthropic, OpenAI and the like. But the AI trade that our capital markets is hovering on top of is an investment boom that’s about twice what the internet was. So, back to Alan Greenspan, he was watching as a lot of tech companies were investing and it

01:36 added up to about 1 to 1 and 1/2% of the economy. It was considered at the end a bubble. This is now 2 and 1/2% of our economy. I’m talking about all the data centers and and so forth. And here’s the challenge for Anthropic and OpenAI is the Chinese models. they might not be quite at the frontier, but if they’re two to six months behind us regardless of how they do it, whether they use what’s called distillation or

02:07 not, and they’re spending a lot less in the Chinese economy, a lot of US companies are saying, “Look, I’ll use the Chinese model to do my reasoning and maybe I won’t use a open AI or Anthropic.” So, that’s the challenge for them. They’re spending so much but will it lead to the revenues to justify that spending? >> Gary, so much capital, as you well know, has been raised in equity markets

02:37 fixed income markets to support the growth of AI. Even some of the founding voices of these companies at Open AI and Anthropic, they are calling for some guardrails on this technology. How do you think about that? >> Well, I think that their real challenge is AI’s capabilities, the ability to do coding on its own, you know, write the software the ability to replicate itself and then send agents out on the internet. Open

03:08 AI’s released that report about an incident this summer where the agents multiplied to 1,200 different agents simultaneously looking for vulnerabilities. And and they found them at Hugging Face. And the last thing I’d say is beyond their capabilities, alignment. Are the models aligned with their human overlords, if you wish? We, the humans. And more and more they’re showing that they’re

03:39 they have some reasoning on how they deceive and how they jump out of guarded sandboxes. So, this is a real challenge. I I I I know the president calls it a hoax. I I differ with the president on that. >> Gary Gensler with us folks, the former chairman I should say of the Securities and Exchange Commission. His good work for your cycle Goldman Sachs noted as well. I got to get to some other matters here. It’s amazing Gensler comes on. The Bloomberg reporters in Washington are brutal. You should

04:11 see the questions. I mean, they want to know what he thinks about Yankees Red Sox. [laughter] I mean, it’s too much there. One more AI question Gary. I got I I got to go here. We’ve had other AIs. We didn’t have guardrails on the trains. We didn’t have guardrails on the miracle of pharmaceuticals and antibiotics, tetracycline and all that. And got Who in God’s name is going to institute the guardrails? I don’t see any institution with motivations to want an ex anti

04:42 basis get out front on AI technology. Where is it? >> Well, currently it’s not in this administration nor in the Congress. Though, I would say the Congress is starting to really lean into it. But look Tom, we put guardrails on our capital markets so that those capital markets work better. And they’re not perfect, but when the Securities and Exchange Commission was created, it was about investor protection.

05:14 It was about [clears throat] making sure that the markets worked better, created a public good that you could trust the market. >> Right. [clears throat] >> So, I would say on AI it’s possible, but here’s the wrinkle and it’s a big consequential wrinkle. The Chinese are competing with the US. The US wants to stay ahead, but the Chinese are doing something called open weight models. This is a competitive challenge cuz US companies can use it and they can use it for cheaper and it undermines the revenue proposition for Anthropic and Open AI. It also undermines, can you put

05:49 guardrails on something that then the user, the threat actor, can maybe take off? Think about those people that sometimes say, “I’ll take my catalytic converter off my automobile.” And um it’s not legal, but a threat actor could say, “Oh, I’m just going to take the guardrails off.” >> Gary, I got a whole group of reporters that do this 24/7, the pitbull terrier down in Washington. >> Mhm. >> Nicola White. >> Oh, boy. >> She’s a threat to society. She covers enforcement for us folks, and she’s got

06:21 a brilliant note here to me, Gary, about the experiment of mom-and-pop retail access to private credit, to all this other stuff that’s illiquid as well. How is that experiment going? And she wants to know, Nicola wants to know, does it keep you up at night that we’re retailizing, that’s my word, with the I just want to impress Gensler, that we’re retailizing, Gary, retail into things they shouldn’t be going into. >> Tom, you impress me every time you go

06:54 out to that Jackson Hole conference, and you have such access to everybody that’s in that meeting. So, don’t worry about it. Look, I do think that there’s a mismatch that the retail public, even high net worth retail public, is being sold private credit. And then they’re also being told they can get some liquidity. And you really can’t. You if you if you want to get the higher returns, you need to put

07:27 your risk capital at work for longer periods of time and not have those redemptions. Now, look, it’s working so far, but I would say when, and it’s not a question of if, but when we have a retrenchment in the big AI trade, there’s a lot of debt on those data centers that is in this complex in the private credit complex. And so, to your colleague’s question, that’s when the retail public’s going to be saying, “Wait, this

07:58 paper’s no longer worth 100 cents on the dollar, it’s worth 80 or 90 cents or even lower. I I’d like to get out.” And what happens then? That’s a real challenge. >> So, Gary, Torsten Slok, economist at at Apollo, was out with a note recently raising some questions about how the rise of AI agents might affect the financial system. How do you think about that? We had the CEO of LPL Financial in yesterday, and he was voicing some of those concerns as

08:29 well. How do you think about AI in the context of just consumers and financial services? >> Well, I’m in the classroom teaching artificial intelligence and money, which is addressing those very questions with some terrific MIT students. And Simon and I, of course, talk about it repeatedly on our Power and Consequences podcast. My thought is three things. One is, for consumers, they might actually get some better automated access to higher rate deposits. That’s

09:03 what Apollo’s Mr. Slok was pointing out. He was saying, “Investors might be able to find better than the very low rates that most bank deposits get.” So, that’s that’s a net plus. But, two, if you’re using Facebook Muse and finding those higher rate deposits, maybe moving into money market funds as well, then the banks might have deposit flight. I think it will happen slower than his note suggested, but it’s it’s real. The other risk is if all of finance starts using

09:37 similar models, then you might have herding. And I wrote a paper on this a while back. It it could really to real systemic risk. And then thirdly, it’s just the overall when we have a retrenchment, and at some point in time this boom will plateau and even turn the other way. When that occurs, then as Alan Greenspan would have liked to hear his friend Warren Buffett say, when the tide goes out, you see who’s swimming with their trunks

10:09 off. >> Gary, [snorts] I mean, it just feels like and I’m sure you see it every day in your classroom, the young folks are just embracing AI for it’s just become a part of their life as opposed to embracing it like Tom and I are trying to do. At some point that’s going to I would think you know, impact how they invest, how they save, how they allocate their own capital. Is that a good thing from your perspective? >> Absolutely. Look, I’m I’m I’m kind of bullish and optimistic that we humans will figure it out, but there’s real risk we’ve talked about in this program. There are [clears throat]

10:43 existential risk about AI agents misaligned or not aligned with their human uh users. >> Right. >> That threat actors will use it and it will disrupt markets. And so we have to build cybersecurity and cyber resilience way better. >> Mhm. >> But I also think it will transform financial markets. And that’s a that’s a period of time of uncertainty. We’ve seen this in prior booms. Uh you mentioned you know, railroads and

11:16 and pharmaceuticals and we’ve talked about the internet. Uh it’s it’s all there. It’s all has some uncertainty. And then you have the geopolitical competition with China. And China’s not taking their foot off the gas pedal. >> Gary, I got to be a smart alec at MIT in the back of the room with my hand up asking Professor Gensler a question. Gensler, the Clinton administration, the Obama administration, uh CFO for Hillary Clinton’s campaign, working for President Biden, etc. From where you sit now, is there a

11:47 tinge to the Democratic Party experiment of being anti-capitalist right now? >> Look, I think that each of the uh parties here in the United States, and you could say this is true in the parties in in Europe, are going through transitions because the economy’s changing so much. And so uh we’ve talked about artificial intelligence, but also the US is stepping back from its leadership on the world stage, and we’re in a

12:17 uh I would say uh a war in Iran that’s doesn’t have any clear resolution and has been a very uh damaging to our economy and to our role in the in the world. I think that’s what uh the Democratic Party is mostly looking at, but uh I would say that there are there’s a deep belief in capitalism, but it’s got to be responsible capitalism. And just as we put guardrails on our capital markets to make them work better. That Franklin Roosevelt did that

12:48 some 90 years ago with Congress. I think we’ll find our way to do that on AI, but right now there’s there’s real risks.