This Tiny AI Never Stops Learning (mini-AGI)

Author: Better Stack · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: This Tiny AI Never Stops Learning (mini-AGI) · Channel: Better Stack · 13:32 · Watch on YouTube

Topics: AI content generation, AI models, Local and open-source models

Summary

The video demonstrates Mini AGI, a small language model trained from scratch on a MacBook and designed for continuous learning. The host first trains it on children’s stories, then on Quentin Tarantino screenplays, testing whether it can learn a new style without losing its earlier English skills. Mini AGI’s mixture-of-experts design aims to limit catastrophic forgetting, and the experiment shows it can switch back to story writing and adapt to more screenplay data. After roughly nine hours, it produces recognizable screenplay fragments and Tarantino-like elements, but it still cannot write coherent dialogue and is not suitable for production.

Key points

Mini AGI starts as a blank model that reads text byte by byte and can be trained on a user-selected dataset.
The experiment runs on an M2 Max MacBook Pro with 32 GB of RAM, although the repository had only been tested on an Nvidia card.
The model learns first from children’s stories, then from Quentin Tarantino screenplays; Reservoir Dogs is held back as a test script.
Mini AGI is designed to train continuously, processing one chunk of text at a time without a fixed finish line.
Its mixture-of-experts setup stores specialist networks as files and loads only the experts needed for each chunk.
The model’s shared core changes more slowly than its experts, aiming to preserve existing knowledge when learning new material.
The video reports that Mini AGI’s author found it retained 99.84% of what it knew after additional chess training in a separate test.
The model initially blended screenplay fragments with children’s stories, then returned to English quickly when story training resumed.
Adding more screenplay data helped it improve on the hidden Reservoir Dogs test, but its generated dialogue remained largely incoherent.
The host concludes that Mini AGI is a toy-scale research model, not AGI and not ready for production.

Chapters

Start	Chapter	Summary
00:00	Introducing Mini AGI and the experiment	The host introduces a tiny model trained from scratch and outlines the plan to teach it English and then a Tarantino-like screenplay style.
02:13	How Mini AGI is designed to learn	The video explains the training setup, the challenges of training locally, and Mini AGI’s continuous-learning mixture-of-experts design.
05:23	Testing English learning and catastrophic forgetting	The host checks the model’s early story-writing and describes how its experts and slower-changing shared core are intended to preserve knowledge.
07:06	Training on Tarantino screenplays	The model learns screenplay formatting and character names, but starts blending the new material with children’s stories and overfitting to its training scripts.
08:12	Switching datasets and trying the screenplay chatbot	After returning to story training and adding more scripts, the host tests the model in a role-play app and finds its replies remain incoherent.
11:56	Results and limitations	The host reflects on what several hours of training achieved, explains the model’s limitations, and concludes that it is an interesting toy-scale project rather than AGI.

Quotes

It’s a tiny language model that you train from absolute zero on your own computer, and it’s designed to never ever stop learning. — 00:00

Once upon a time, there was very down. — 04:53

So the new stuff mostly lands in the specialists, while the foundation everything else relies on stays pretty much intact. — 06:34

It’s quite impressive that you can train a model from literally zero, and in 9 hours it will start spitting out some Tarantinisms. — 11:56

So, is it AGI? Hell no. — 13:00

Concepts and entities

Mini AGI, Mixture of Experts, catastrophic forgetting, Tiny Stories dataset, Quentin Tarantino, Reservoir Dogs, M2 Max MacBook Pro, Better Stack

Full transcript (original language)

00:00 Guys, AGI is finally here. Well, okay, not the real AGI. I’m talking about mini AGI. It’s a tiny language model that you train from absolute zero on your own computer, and it’s designed to never ever stop learning. I’ve been having so much fun lately with this model. So, I trained one on my MacBook, and 7 minutes into the training, it gave me sentences like thousand worded their lone Sheraton. But, just a few hours later, we got something like this. “It’s okay,”

00:33 Lily said. It was capable of learning English in less than 2 hours. And then, I fed this model a bunch of screenplays from Quentin Tarantino to see if I can actually produce a model that would be capable of writing like Quentin Tarantino. So, in this video, we’ll take a look at how mini AGI works. I’ll show you what happened when I trained one from scratch, and I’ll show you some of the weirdest and funniest things it said along the way. It’s going to be a lot of fun, so let’s dive into it.

01:08 So, first, before I get into the weeds of how this model works, let’s just start training one from scratch. So, step one, we start with a completely blank model. This model doesn’t know any words or grammar or nothing. It’s just a bunch of random numbers, and it reads text byte by byte, letter by letter. So, first things first, we got to teach this model plain basic English. And the cool part about mini AGI is that you don’t have to teach it a language, you can basically start teaching it whatever you want. But, for this model, we’ll start

01:41 with the tiny stories dataset, which contains a lot of examples of short children’s stories, and try to give it a voice. And once step one is done, and we will have confirmed that the model can indeed write short English sentences, then we will switch to step two, and I will try to feed this model a bunch of Quentin Tarantino screenplays. Because by the end of this training, I want this model to speak in a voice of Quentin Tarantino and spit me out some script-like sentences from his famous movies. And since this is a continual

02:13 training experiment, we will keep one script, Reservoir Dogs, hidden from the model the whole time, so we can test it on a Tarantino script it has never seen. So, by the end of this, I’m hoping we can get two things. One, a model that writes something that at least smells like a Tarantino movie, and two, the answer to the big question, can this model keep learning new things without forgetting the old ones or without degrading at some point? So, let’s start the training. Now, I’m training this on my M2 Max MacBook Pro with 32 GB of RAM,

02:46 and this repo was only ever tested on an Nvidia card. So, yeah, we’re already going off script here a bit. But, while it is learning, let me tell you what this thing actually is. So, what exactly is Mini AGI? Well, every model you’ve ever run locally was once trained on a giant GPU cluster and then frozen in time. And that’s why these older models, whenever you would ask them, for example, what is the current US president, it would give you an outdated answer. Basically, it stopped learning the day it shipped. And a much bigger

03:19 problem is that you can’t really train a big model at home yourself because training needs roughly three or four times more memory than just running a model. And that’s why you can chat with a 30-billion-parameter model on a MacBook, but training a model, even a small one, is a job for a giant hyperscaler data center. And there’s another problem. If you fine-tune a model on your own stuff, you get something called catastrophic forgetting. So, for example, if you would take a model trained on basic

03:49 English sentences and then teach this model chess, it would suddenly forget how to write English. Which, to be honest, is how my brain works sometimes when I have to learn a new JavaScript framework. But, MiniAGI is trying to fix all of these problems. So, how does it differ from any other model? Well, first, it never stops training. It reads text in order, one chunk at a time, like a person reading a book, and it takes a small learning step after every chunk. And there is no finish line. You can just do this forever. And second, it’s

04:21 pretty efficient with memory. It’s a mixture of experts model, and we’ve covered mixture of experts models a lot on this channel, so I’m not going to tell you how they work. But, basically, TLDR is it’s a team of small specialist networks, and only [snorts] some of them are active at any given moment. But, here, every expert is its own file on disk, holding everything needed to train it. And before each chunk, the model loads only the experts the text needs onto the GPU. So, the model can be way bigger than your graphics card. And the

04:53 specialist team grows by itself. New experts pop up as files in the folder, and the ones nobody uses gets deleted. So, it’s basically a very efficient, self-governing workspace. So, let’s check in on how it’s doing. 2 hours in, and it’s already cracking the most important sentence in all children’s literature. “Once upon a time, there was very down.” Close enough. And it was already producing some material, like, for example, “Timmy was sad because he was so happy.”

05:23 >> [snorts] >> Same, Timmy. And just for fun, every 5 minutes, I’m also giving the English-only model a little test, the opening of an imaginary Tarantino scene that we wrote ourselves. Two men sit in a booth, one of them leans in, “Vincent.” And the model has to write what Vincent says next. It doesn’t learn anything from this. It’s just a peek inside its brain on on it’s doing at that current training stage. But so far it has only ever read bedtime stories. So Vincent is saying things like, “They said you are mine. They said you are their mom. They say you are cone.”

06:01 Well, this isn’t going to make for a very good movie. So 4 hours in, I finally had a model that writes simple English. And remember I mentioned catastrophic forgetting? So the author of Mini AGI tested exactly that. He took the model that had learned eight different subjects, like stories, code, and math, and then fed it half a million characters of nothing but chess. And with a normal setup, the model lost roughly half of what it knew about everything else. But Mini AGI kept 99.84%

06:34 of it. And the trick to make this work was surprisingly simple. So the model has two parts, a shared core that every single letter passes through, and the experts, which are the specialists. When it learns something new, the experts are allowed to change quickly, but the shared core is only allowed to change 10 times slower. So the new stuff mostly lands in the specialists, while the foundation everything else relies on stays pretty much intact. It’s like renovating a few rooms without touching the foundation of the house, if that

07:06 makes sense. Okay, so let’s test that ourselves. Step two, let’s teach it how to write like Tarantino. So we’re going to add Pulp Fiction, Jackie Brown, Inglourious Basterds, Django Unchained, and his segment of Four Rooms to the data set, and that is basically what our screenplay data set mostly consists of. And within 5 minutes of running the second phase, it learned the layout of a screenplay, so that’s already nice. We have some indentation, the character names in caps. And now within half an hour, the cast is already showing up as

07:38 well, Ordell, Louis, Schultz. But now it’s kind of morphing with the children’s stories. So now the Tarantino movie characters are kind of apologizing like kindergarteners. Stephen, I’m soaring. Calvin’s them, I’m soared. Sounds like old English. Meanwhile, its English got worse and worse until even once upon a time came out as screenplay fragments. So you can see we have some problems here. So the problem was that after about 15 minutes it stopped getting better on the hidden reservoir script because we only have one script

08:12 to compare it against, but it kept getting better on the ones it was already reading. So it stopped learning Tarantino and started memorizing those specific scripts. It’s basically like learning to be funny from your uncle versus just memorizing his five favorite jokes. So then I decided to pause the training and pick it up the next day. And the next morning I switched it back to children’s stories. And this is the best thing I love about this model. 5 minutes in it remembered to speak English again. So remember learning for

08:43 the first time took the model hours, but now that span is a lot shorter because the knowledge was never erased. It was already on the disk. And after 2 more hours it had gotten even better at writing those children stories again. So now it was time to test it on the Tarantino data set again. But this time I added three more additional scripts so we have more data to work with and it snapped back into the Tarantino mode almost instantly. Bill from Kill Bill showed up and it got the best score yet on the hidden reservoir dogs script. And

09:15 the crossover between the children’s stories and Tarantino got even better. The bastards who should be more companion show up to the food court. I mean that’s that’s kind of nice. At least everyone’s getting along. So I kept training it for a while and after about an hour and a half I had a somewhat coherent Frankenstein model ready to go. So to test it out, I built this little screenplay page where you can test this model as a screen partner in a non-existent Tarantino movie. So, let’s take it for a spin. So, as you can

09:47 see here, this is our screenplay chat application here. And here you can just define what kind of character you’re going to play, who’s going to be your scene partner, and then we’re setting a scene with a heading and an action to begin with. And then we can also choose the reply length. And then we can also tone the temperature crazy we want the responses to be. What if we invent new characters? So, I’m going to be John. And I’m going to be talking to Freddy. I’m just going to ask Freddy, “So, what are you thinking, man? Should we rob the bank today or not?” What is Freddy going to reply? Tonics.

10:21 Freddy is setting some tonics. Arlene’s just about her. She’s almost her. Who’s Arlene, man? To him. To me. All right. She’s not but I a silence. You’re being very mysterious, Freddy. What? Arlene’s makes a way they’re making him. She’s cold. I mean, I do agree Arlene can be very cold sometimes. All right, Freddy. I have no chance. I have to rob this bank. Are you coming with me or not? Last chance. Freddy’s setting the tone again. Shrinks

10:53 car but when a silence. Oh my god, this this this model really likes using the word a silence. Interesting. Interesting. All right, I’m hopping in the car and I’m driving away. What is Freddy going to say? She’s notch and I’m sure a silence. He just had anyway still her. Man, this guy this guy is a poetic soul. I can I can feel it. >> [laughter] >> This this word just comes up so much. A silence. What if I play along? What if I just say a silence. There’s always die then looks

11:26 up. She’s them. Oh my god, it switched to a new place by itself. [laughter] Oh my god, that is interesting. I’ve never encountered this before. All right, so we’re now in a new scene. Where is my money, Freddy? Let’s see what Freddy replies. What? What do you mean, what? Where’s the money, Freddy? There’s somebody girls the car. She’s coming back. So, it may look quite dumb, and I understand that, but you got to realize this is a result

11:56 of 7 hours of training, maybe even more, maybe like 9 hours. It’s quite impressive that you can train a model from literally zero, and in 9 hours it will start spitting out some Tarantinisms. Although they don’t make sense, at least at least it’s something, you know? So, as we saw there, it can’t really hold a conversation. It learned what a Tarantino script looks like and who’s in it, but to actually talk like him, it would need hundreds of millions of characters of English, which means we

12:27 would need to train this model for several days non-stop just to get it to say basic grown-up sentences. So, I hope this shows that training models is really hard and time-consuming. But, the cool part is that with Mini AGI, you can do just that. You can now have a model that you can continuously train on your consumer hardware until you get something better out of it than the bear said, “I want to be a greatest fish.” So, there you have it, folks. That is Mini AGI in a nutshell. The author calls

13:00 it a toy-scale model, and the weights aren’t even out yet. So, is it AGI? Hell no. Can it write a coherent sentence? Well, barely. Should you use it in production? Absolutely not. But, on a simple M2 MacBook Pro, it was capable of getting from random gibberish to Tarantino cast apologizing to each other. So, I think this is a super cool training model. So, what would you teach your own Mini AGI? Let me know in the comments down below. And folks, if you like these types of technical breakdowns, please let me know by

13:32 smashing that like button underneath the video. And also, don’t forget to subscribe to our channel. This has been Andrus from BetterStack, and I will see you in the next videos. >> [music]