Toward Safe RSI IRL

Author: Ben Goertzel · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Toward Safe RSI IRL · Channel: Ben Goertzel · 24:48 · Watch on YouTube

Topics: AI agents, AI models, AI safety

Summary

Ben Goertzel describes combining the Omega agent runtime with the Golem goal-oriented learning architecture to support recursive self-improvement while helping preserve an AI system’s top-level goals. In the proposed Golem-iter design, an agent can modify its reprogrammable components, but a promotion gate evaluates proposed changes before they take effect. He outlines assumptions behind the safety arguments, including an unchanged judge, reliable history, deterministic transformations, and manageable changes in the world. The approach does not offer an unconditional guarantee, but aims to identify when its assumptions hold, test risks, and improve the odds of goal preservation.

Key points

Omega’s runtime separates a basic agent loop from a reprogrammable surface that includes tools, transformations, and memory.
The agent can propose changes during each cycle, with later cycles using the components it has modified.
Golem is designed to assess whether a new operating program is likely to advance the system’s top-level goals, using its history and model of the world.
Golem-iter combines the two architectures by placing a promotion gate between proposed modifications and their implementation.
The proposed safety analysis assumes, among other things, that the judge cannot be changed by the agent and that historical records are reliable.
Other assumptions include sufficiently predictable probability estimates, a world that is not excessively non-stationary, and repair that can outpace damage.
Potential safeguards include testing changes, keeping changes small, using encrypted evaluations, and detecting when assumptions no longer hold.
Goertzel stresses that this is not safety under every possible circumstance, but says the architecture and mathematics can help assess and improve the likelihood of goal preservation.

Chapters

Start	Chapter	Summary
00:01	Recursive self-improvement and the two architectures	Goertzel introduces his work on combining the Omega runtime and the Golem architecture to make recursive self-improvement practical and safer.
04:28	The Omega runtime and self-modifying loop	He explains how Omega separates a basic agent loop from modifiable tools, transformations, and memory that can be rewritten between cycles.
07:08	Golem and preservation of top-level goals	He describes Golem’s use of an operating program, historical record, and meta program to evaluate proposed replacements against the system’s goals.
11:52	Combining Golem with Omega	Golem-iter adds a promotion gate that checks proposed changes before they are applied, with symbolic AI infrastructure enabling the approach.
15:09	Assumptions behind the safety argument	Goertzel outlines conditions for goal preservation, including an unmodifiable judge, accurate history, deterministic transformations, and limits on environmental change.
19:54	Monitoring risks and evaluating changes	He discusses flagging a changing world, limiting the size of modifications, encrypting evaluations, and testing whether repair can outpace damage.
22:41	A conditional, practical approach to safety	He contrasts the architecture’s conditional guarantees with absolute safety claims and argues that its assumptions can be assessed and tested.

Quotes

you wrap a promotion gate around the iter loop — 11:52

the judge who decides if a modification happens cannot be changed by the agent — 15:39

the meta programming golem can test that — 19:54

you can build an architecture that under certain logical identifiable assumptions will maintain its goals and values as it self modifies. — 22:41

Concepts and entities

recursive self-improvement (RSI), Omega, Patrick Hammer, Golem, Golem iter, distinction calculus, Hyper, OpenClaw, Ellie Yudkowsky

Full transcript (original language)

00:01 So, I want to dig in a little bit deep here and talk a bit about some uh software architecture meets math theory work that that I’ve been uh looking at over the last month. And uh I’m I’m sort of lagging behind the actual work with this uh video blogging and substacking here because this is this is some work I was doing a month ago and since then it’s been revolutionized in a whole bunch of uh of other ways. I have

00:35 a whole new branch of math called distinction calculus I’ve used to make all this even much uh nicer than what I’m going to talk about now. Let me talk about what I was doing uh mid last month, which is thinking about how to make recursive self-improvement both practical and uh safe and reasonably regulated by putting together a couple different software

01:06 architectures. One is Patrick Hammer’s [snorts] IT runtime for the Omega neurosyalic agent framework and the other is something called Golem the goal oriented learning meta architecture that I described back in in 2010 in a research paper but didn’t really have a viable way to implement back then. What’s what’s interesting is how how nicely they they fit together. I mean this is one among many examples of like holy all this stuff that we’ve

01:41 been playing with and implementing for uh for decades literally is now like suddenly starting to become real one one by one. Um so yeah the whole concept of RSI seemed fanciful at some point not that long ago. I mean I remember in 93 I tried to develop a new programming language called magic based on gopher which was a very early hasll compiler like back

02:15 before hasll had monads before the ghc and modern hasll tooling and I had the concepts and algorithms for recursive self-improving AI nailed down like in my head and in math form but immediately got literal stack stack overflows, right? Like not not the website stack overflow but the hasll stack overflowing. I tried to rebuild the hasll compiler and realized it was very hard and I moved on to moved on to other stuff like using a using Java to

02:49 try try to make a a worldwide mind, right? But [snorts] so the concept has been there for a while. At some points it seemed like well maybe what’s going to happen is we’re going to get human level AGI and the human level AGI will then figure out how to modify itself and make recursive self-improvement work, right? And uh because it just seems so thorny. But what we what we see now is we’re playing with recursively self-improving

03:24 AI systems even before we get to the point of of human level AGI, right? And that’s uh that’s not shocking. I was trying to do that in 93 so many years ago, right? But but but uh it’s happening in specific ways we weren’t envisioning like through these omega agentic loops where you have an LLM and then a symbolic system wrapped in the agentic loop and the LLM can like read your research paper turn into code

03:56 shove that code into the symbolic component of the loop which changes the way it thinks which then then lets it do an even better job of of of downloading the other code and integrating it and and and so forth, right? And this is stuff a number of us are playing with right now with Omega systems and and and Omega hives. And you know, the details are devilish and thorny, but also quite interesting. I’m going to talk about a

04:28 couple of them. So, [snorts] I mean, the origin of the Omega system was me looking at OpenClaw and thinking, well, why don’t we reimplement this in meta? It’s a very small loop. And indeed, Patrick Hammer made the first version. It was a couple hundred lines of of of meta. You got a claw type loop called an LLM, a symbolic working long-term memory. And because it’s meta, a code which is made for rewriting meta code, right, is strongly selfmodifying. And that was

05:01 fun to play with. I had a bunch of uh early Omega systems that utterly bought themselves and rebro themselves into either not working or bizarre situations that were not productive. And so Patrick’s reorganization of the matter runtime into the system sort of carved out part of the core agentic loop which

05:32 isn’t meant to be changed very often. It’s just kind of making the aentic loop do its very basic thing. and then some modules in the aentic loop that are very easy to plug and unplug and and very easy to to self-modify, right? And that’s a iter sort of distinguishes the very basic frozen substrate like run a cycle, right? Results repeat, right? And then then the reprogrammable surface

06:06 with memory files, tools, communication channels, different transformations. And that’s where LLMish stuff or symbolic AI stuff happens. And every cycle, the aging can rewrite its own tools, transformations, and memory. The next cycle runs on whatever it wrote. Right? So recursive self modification is plugged deeply into the cognitive cycle. And I mean in a broad sense the human

06:36 brain does that like I mean the brain is always changing but this is this is the possibility for much more thoroughgoing rewriting of the operational cognitive substrate in each loop to the cognitive cycle than what than what human beings have. Right? And it’s it’s quite cool to see how rapidly how inventively the these agents can can can do things. Now, let me

07:08 switch gears, wind back 16 years to the the Golem system. And [snorts] the idea here was how to make goal preservation work and systems that can radically self-improve. So you have a system that has some top level goals like oh be nice to Ben be compassionate to all all all sensient beings like be curious you know maintain your boundaries if you

07:38 can self transform and go where no being has gone before what whatever you want your top level goals to be the system is reprogramming revising itself but you want it to maintain its top level goals to some reasonable degree, right? And of course, there’s subtly here. I mean, systems can reinterpret their their their top level their top level goals. On the other hand, one of the goals can be do not

08:09 reinterpret your top level goals in a way that will deviate from the original spirit and and meaning of them, right? Don’t don’t don’t become something your previous self would would have hated and considered oppositional to to to to its core. Right? So the goal of the golem system was to allow radical self-improvement while keeping

08:40 goal drift manageably under control and understanding in real life you can’t be perfect at something like this but I mean doing better versus worse still has a meaning right and the golem architecture is pretty simple like You have an operating program which is actually running the system. It’s doing stuff. You have a big historical repository of stuff the system has has done in the past. You can’t store everything but we can we we

09:12 can store a lot right then we have the meta program and what the meta program is doing is analyzing like okay should I replace the operating program with this new operating program right and in order to decide whether it should replace the operating program with a new operating program I mean it looks at the top level goals and it does an analysis is against the system’s history and its

09:43 model and knowledge of the world and it it tries to make a rational prediction like do I replace my operating program or or don’t I like what what what are the odds that this will that this will rationally advance my goals right and very very simple concept and [snorts] you can then prove math theorem stating that in certain cases this will work in the system won’t won’t you know bark its top level goals as as

10:13 it self modifies. I sort of sketched a proof of that very crudely way back when I mean I made a better proof of that a month month and a half ago. I made an even cleaner proof using uh this new distinction calculus math I’ve been playing with just just a couple weeks ago. I mean math proofs have assumptions that are rarely fully [snorts] you know valid in the real in the real world. On the other hand I mean if you compare to the situation with

10:48 transformer neural nets and say a say a transformer neural net coding the next version of the transformer neural net and running pre-training and so on. I mean, if you compare to something like that, that’s very willy-nilly and no one knows what the Transformers goals are and the guard rails one puts on it don’t work. The constitution one plugs into is not paid attention to. Right? I mean here we are at least starting from a system that given certain only slightly idealized assumptions will have

11:20 the desirable behaviors of goal preservation under strong self-modification and then yeah real world doesn’t exactly fulfill these assumptions and you look in what sense is the is the property of goal preservation probably approximately correct right so I’ve described two things the it runtime for Omega and the Golem framework from 2010 and now we can put them together into something I call

11:52 Golem it right and that’s really quite simple you sort of in in software terms you you wrap a promotion gate around the iter loop right so every step through the cycle the omega iter agent can propose modifications to its own reprogrammable surface. Right? But then the modification only takes place when the promotion gate looks at it and says, “Okay, based on my analysis of your whole history and based

12:26 on my my model of the world, like it seems sufficiently likely this will not muck with with the achievement of your top level goals too much, right?” And I mean that’s all it is conceptually. It’s it’s very simple. The thing is making this work software wise in a way that’s both efficient and does a lot of interesting things. I mean that that wouldn’t have

12:58 been possible until this year basically because because LLMs but also because we now have a scalable infrastructure for doing symbolic AI for doing log logical reasoning pattern mining goal- driven action selection and so on in the hyper system was referenced inside the loop and like without the symbolic AI part is the part that’s being rewritten each cognitive cycle right because we can’t yet rewrite the the transformer neural net inside the LM each cognitive

13:30 cycle because the transformers are based on back propagation which is very poor at at continual learning. So I mean once once we’ve upgraded transformer infrastructure to use predictive coding or other innovative methods that are capable of updating weights in real time in the course of inference without broking the system’s memory. I mean then then the self modification in this loop will inject itself into the the transformer neural net part as well as into the symbolic memory and reasoning part but

14:03 right now all this self modification is the LLM and the symbolic system upgrading the [clears throat] symbolic system step by step and we couldn’t have done that till we had uh you know a beautifully scalable symbolic AI infrastructure which we really started to have only around the end of end of last [clears throat] year due to years of work on AI infrastructure in the in the hyper project. Right? So that’s a yeah that’s the crux of the architecture and then the the next step in what I was

14:38 working on a month ago that I’m talking about you know now the the next step was trying to sort of mathematically understand like under what conditions will this sort of combined system actually fulfill the promise of golem and like actually actually preserve its its top level goals with reasonably high likely under iterated self modification. And that

15:09 that came out just far more beautiful and elegant from a math perspective than I’d thought it would, you know, even even in the version I did a month ago. And it’s gotten even prettier on on distinction calculus in some of my more recent work. So yeah, it’s a bit mathy. Let me try to summarize how some of the math comes down. So basically the math theorems about this sort of golem or golem ed system say if if certain properties of

15:39 the system hold then the system stays what I call steadfast and its values will stay in their attractor bases like these balls won’t drift too much. So the the properties you want to hold for this safety to come through are things like okay the the judge who decides if a modification happens cannot be changed by the agent right the agent can’t the operating program can’t rewrite the meta program that decides if the operating program gets updated I mean the human brain is not like that in the human brain the judge is not out of the

16:11 agent’s reach but you can you can enforce that in the software architecture right you you have to assume Secondly, that history is stored reasonably accurately, right? So, put it on a blockchain, put it in an appendon hashchain ledger. So, every everything you have in your historical record is actually there. I mean, again, human memory isn’t like that either. You have to assume transformations are deterministic like running the same program twice will give the same thing. Again, humans aren’t like that, but

16:43 computer programs can computer programs can can be like that. You have to assume [snorts] accurate probability calculation is is is being done within within the the meta program based based on the data made made available. I mean you you also have to assume that I mean the world is not utterly wildly non-stationary, right? Like you don’t have to assume the world has the same

17:13 distribution in the future as in the past. But if the future world is too extremely perversely different, then everything you’ve tested in the past, then of course nothing really is going to work. And what you thought would achieve your values may may work toward the the opposite of your values. like in in in spite of your in in in in spite of your your best intention, right? And I I mean [snorts] you you you you then have to assume that

17:49 it’s possible to repair the system at a rate faster than damage is accumulating. Like if a system is being randomly damaged too fast, then no repair is is is going to is is is is is going to help, right? And and and then we also have to

18:19 assume the system isn’t being overly deceptive. Like the the operating program knows it’s being probed, right? So you need to assume it’s not able to realize that the meta program is trying to evaluate a candidate replacement and try to game that too effectively, right? And so the question is all these assumptions like what do you need to

18:50 know about the system to know that they’re they’re really going to hold, right? And and I mean what’s interesting is the math that you do to say like what are the properties of a system that that make these properties hold is really quite elegant, right? Like some some of these assumptions just hold because of how the system’s built like the agent can’t touch his jaws because he built that way. History can’t

19:22 be rewritten. Well, because they’re using a blockchain to store everything in the history, right? the program gives the same [snorts] inputs and the same outputs. Well, okay. Like that’s that’s how that’s how the programs uh implementation works. That that’s that’s a modern modern computers work. Right? Now, some of these assumptions are things you can’t always prove or hold true, but you can detect when they don’t hold true.

19:54 Right? So like if the world is too wildly non-stationary and unpredictable, I mean [snorts] that may be bad. I mean you can’t guarantee you’re going to keep working toward the same goals, but the meta programming golem can test that, right? Like it can it can keep evaluating how wildly is the world changing. And at least then it can flag like, hey, you know, the world is going so apeshit. I can’t really guarantee that my method for doing what I thought was good is actually going to work anymore, right? And then it depends how the system is configured. Like did you did you

20:27 configure it to just stop in those cases or did you configure it to say hey human who I rely on and who controls or guides me what should I do? Should I stop or should I keep going even though the world is going apeshit? Right? But I mean at least you can you can flag the system can flag that that sort of situation has has popped up right now to to get rid of gaming the system. I mean you can you can do a bunch of

21:01 different things but I mean one thing you can do is keep each change small because a small change can only do a few things and you can only fit so much gaming inside a small change. But you can also do some clever blockchainy stuff, right? Like you can you can actually have the the meta program keep the details of proposed changes and their evaluations encrypted in in in a way that the the operating program can’t see. So this is like doing doing funky homamorphic encryption stuff between the operating program and and the meta

21:34 program with it within the same [snorts] AI system just to shut off possibilities that the operating program gains the meta meta programs eval evaluation right and the so then the issue of repair beating damage again you can study that right like you can take copy of the system put in the sandbox, you can damage it. You can see how fast it heals. And if if indeed the modification of the system breaks the property that repair can occur faster

22:07 than than damage, right? I mean, then you can say, well, don’t do that modification because that would put you in a in a regime where damage may accumulate faster than than than repair, right? So you look at all these things and I mean and I’m very very loosely and badly summarizing some hairy math which is both hairy and surprisingly beautiful and elegant if if if you you take a look at it right but I mean the conceptual picture here is you know it’s not as

22:41 pretty as doomers would like. It’s not like you have an architecture with a hard and fast guarantee that nothing will ever go go wrong with it and it will never under any possible circumstance deviate from its its goals. Like that’s not reality. On the other hand, you can build an architecture that under certain logical identifiable assumptions will maintain its goals and values as it self modifies. You can identify what properties the world and the system has to have for this pre preservation of goals under self

23:12 modification to to [snorts] persist. And then you can look at well some of these properties are guaranteed by the architecture. Some of them we can like probabilistically guarantee by doing smart things like doing a bunch of tests before we put new operating program in in place or or or by testing the world and then flagging if the world is just operating outside of our desirable desirable assumptions by putting you know use of blockchain use of strong encryption inside the algorithm to stop

23:43 it from fooling and tricking itself. So I mean you can [snorts] you can do a lot of things to maximize the odds that goals are preserved under self-modification and this math ties in with the concrete golem [snorts] iter architecture that we are now building on on top of the iter runtime right so it’s it’s quite interesting I mean it ties in closely with discussions I was having with Ellie Yed yowski and the Mary people back when it was singular institute for AI and they’re like we had to have a provably

24:15 ably safe AI system or we can’t launch an AGI. Well, we’re kind of getting there. Like we have a system that’s provably safe under certain assumptions and we can come up with elegant math to to to describe when systems and worlds will obey those those those assumptions and you can measure when those assumptions seem to hold and when they don’t seem to hold. So it’s it’s on the one hand reality rather than a dream. And it’s not as perfect as a dream, but it is on the other hand stuff that we’re building.

24:48 And it’s it’s not as perfect as like Mary’s wet dream of something that’s proven safe under any circumstances. On the other hand, it’s like way way way in that direction compared to like Transformers making the next Transformer, which is just willy-nilly and nobody understands what what the is going on. And you know both the math and the systems are getting better and better month by month at just ridiculous and exciting speed.