Flow matching for Beginners | @ariG23498

Author: Hugging Face · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Flow matching for Beginners | @ariG23498 · Channel: Hugging Face · 28:05 · Watch on YouTube

Topics: AI content generation, AI models

Summary

The video introduces flow matching as a way to transform samples from a known Gaussian noise distribution into data such as handwritten digits. It explains a straight-line path between noise and an image, whose target velocity is the data minus the noise, and shows how a model can learn that velocity. Generation then follows the learned velocity field using Euler’s method. Browser demonstrations illustrate training and sampling, followed by notes on the method’s limitations and suggested resources for further study.

Key points

A training example is formed by blending a data sample and independent Gaussian noise at a time between zero and one.
The video uses a straight-line trajectory from noise to data, with target velocity equal to the data sample minus the noise.
A model takes the intermediate point and time as input and predicts the velocity.
Generation starts with Gaussian noise and follows the predicted velocity field using Euler’s method.
Learning the velocity field is presented as a way to transport samples from a known noise distribution toward the data distribution.
The training example compares predicted velocity with the target velocity using an L2 loss.
The browser demos show velocity-field training on two-moons, spiral, and ring distributions, and an MNIST example using compressed image representations.
More Euler sampling steps can help a generated sample follow the velocity field more closely.
The speaker notes that the tutorial simplifies a more technically involved research area and points viewers to lectures and other learning resources.

Chapters

Start	Chapter	Summary
00:00	Flow matching and the noise-to-data path	The speaker introduces the topic and explains how noise and a training image define a straight-line trajectory with a target velocity.
06:10	Data distributions and Gaussian noise	The video describes handwritten digits as samples from a data distribution and explains why a known Gaussian distribution is useful as a starting point.
10:23	A short introduction to Gaussian distributions	The speaker explains the Gaussian mean, variance, and the uncertainty involved in drawing samples.
13:30	Transporting samples between distributions	The video connects following a velocity field for individual samples with transporting a whole noise distribution toward the data distribution.
15:37	Training the velocity field	The speaker walks through the training code, including sampling time and noise, constructing the intermediate point, and optimizing velocity prediction.
19:45	Training and sampling handwritten digits	A browser-based MNIST demonstration trains on compressed image representations and uses Euler steps to generate digit samples.
24:54	Takeaways and further learning	The speaker emphasizes that flow matching has greater technical depth than this introduction and recommends lectures and probability resources.
27:32	Closing remarks	The speaker points viewers to the online lectures and blog post and invites questions and suggestions.

Quotes

the target velocity is very simply put z - e — 02:35

This following the trajectory is something which we have used here is called the Euler method — 05:08

we want to go from a distribution that we know to a distribution which we don’t. — 09:52

that transport is what is um underlying all of this flow matching bit. — 15:04

flow matching is not as easy as this blog post tries to be. — 24:54

Concepts and entities

flow matching, Gaussian distribution, Euler method, vector field, MNIST, PCA, PyTorch, Claude, Codex, Hugging Face

Full transcript (original language)

00:00 Hello everybody. I am Auditory and today I’m going to talk about flow matching. Consider this video as an addendum to the blog post that I have recently written on the same topic. And in this video we are going to talk about a few things. Number one is going from noise to images. That is what we want, right? And then we talk about how to train this objective. What is the objective and then how to infer on it. We will also see in the browser how to train a very simple example. We go from noise to handwritten MNIST digits, which is

00:31 going to be super cool. Look at the video. Look at the blog post as well. And whenever you think you are not able to understand, just quickly map from the video to the blog post and I I hope it’s going to make sense. And that’s how I would want to flow in the video. So, what to train? Choose a training image Z. Z is our training image. It’s one of the data from the data sets. And an independent noise added epsilon. Epsilon is a draw from the Gaussian

01:02 noise distribution that we have. Of the same shape. Now choose a number T. And T is time essentially, which goes from zero to one. And blend the two. So, this is a weighted sum of epsilon that is noise and Z, which is our data. What happens at time T? At time T, which is X T is equal to zero or X of zero, it’s just plain noise. And at T is equal

01:33 to one, X of one, it’s the data. At halfway, which is X of T, you will have an equal mixture of both noise and data. And T or time is simply a progress bar. So, from zero to one, you go from noise to data. And this traces a straight line trajectory from noise to to our image. And why this is a straight line trajectory, why is this not something non-linear, or why is this very

02:04 straightforward is because we assume it to be, we want this to be. Um going further down the weeds of diffusion papers and flow papers, you will find that this this is a very special case. But this is very intuitive to understand and this is what I went on with. And with this trajectory, you will also see that the target velocity, or just the rate of change of this of this noise to data, let’s say, or the trajectory, or the rate of change of

02:35 this trajectory, or the target velocity is very simply put z - e, that is the data, subtract it with the noise that you have sampled out. And what this does is if you have x at any point and you know which direction and with what speed you need to go uh you need to go to you know where you will end up, and that where is essentially your data. So, if

03:05 you have noise and you have the velocity, that is the direction and the speed you will end up with data. And this is the entire um interactive embed that happens. So, let’s play. You go from noise, that is 0.0. And as time increases and you follow this target velocity you go to a data. Let’s pick out a different digit. And you go to and you just play.

03:36 It’s following the velocity and it goes from noise to image. And this is extremely intuitive. You have this uh straight line, which is given by 1 - t epsilon + t of z. And as you have the target velocity, you can understand that if you are at a point and you have a target velocity, and if you follow that target velocity, you end up into the data. This is nice. This gives us everything that we need to

04:06 construct, essentially. You give the model the XT and the time T, and the model will predict the velocity, which is Z - E. And once you have the model give out the prediction, you will understand what to infer. And that takes me to the next section, which is what to infer. If you have the velocity, that is if the model gives you the velocity, you can simply go one step towards the velocity, or or use use the velocity to go one step from

04:38 where you are in that direction in that speed. And as you do that, or as you go in that direction of velocity, you essentially go end up somewhere, and that somewhere is going to be your data distribution. And the entire thing that that we all came across is vector fields, that is a field with velocities, and how you can map out a trajectory by just following this velocity field, and this

05:08 following the trajectory is something which we which we have used here is called the Euler method or Euler’s method of going to going from one place to another following this vector field. The generation in Python looks like this. So, if you have a model, let’s forget shape, let’s just talk about steps, that that is the how many steps you have to move. You start with Gaussian. You have

05:39 one by the number of steps is how small the step size is, and for each step size, you just predict the velocity and move in that direction and speed. That’s That’s essentially it, and it’s very simple to wrap your heads around. And this also takes me to the optional OD trajectories and flow. And if you are a researcher, you want to understand more about it, please feel free to read this. Uh in the video, I don’t think this is quite necessary, and it’s going to be a little

06:10 advanced. So, I’m going to skip this part, but be sure to read this. Uh play with the play with the fields, and see what this all means. In the section from noise to images, I’m going to formulate what we mean by generation and how uh flow matching comes into play. So, imagine a data set containing thousands of handwritten digits, and each image is uh 28 by 28, and when flattened can be of 784 pixels or 784

06:43 dimensions. And each of this image can be thought of as a point in this very high-dimensional, that is 784, dimensional space. Our data Our data set gives us 1,000 such points, and these points are not scattered uniformly, which is very important to uh point out because if it would have been, the entire section or the entire blog post would not come into existence. The entire essence is these points are not scattered uniformly, and

07:15 and handwritten digits occupy some regions more densely than others. And what we can describe this intuition, or the tool with which we can describe this intuition, is called probability distribution. Now, the probability distribution tells us how probability is spread, which is good. Some regions with high handwritten digit probability are more plausible, while regions that contain meaningless images are less probable or have little probability.

07:46 Now think of the entire thing together. This is the handwritten digit probability distribution. It’s a very specific probability distribution which only talks about handwritten digits. And this is the data distribution. Now, if you have the data distribution, what you what you can do is you can now draw novel novel images, novel handwritten images from this data distribution because you know which is plausible, you know what is highly probable to happen, and you also know what is not that probable to happen.

08:17 Once you have this information with you, you can sample or draw from this data distribution novel handwritten images, which is good. Now, do you have it? That’s the question I’m going to ask. Do you Do you as a user or do you as a researcher have the probability distribution in your hand? If you had it, we wouldn’t have had the research that a lot of people are doing on figuring out these novel images or these realistic images. They don’t have the

08:49 distribution and they want to map a distribution which which we know, and spoiler alert, that is going to be the normal distribution at the Gaussian distribution, and we are going to talk about Gaussian distribution a little later. It’s It’s a very known It’s very well drawn out. The mathematical equations are there. So, in essence, normal distribution, good. We can draw from it. We know the probability density function. It’s It’s It’s elegant.

09:20 Now, if you have the normal distribution, how can you map or mold this normal distribution into your data distribution? That is the entire essence of anything diffusion that you see. Not only flow models, but also diffusion models, score matching, um or or DDPMs, DDIMs. I’m just out a lot of jargons, but you don’t have to remember all of this. Think about the intuition that I’m trying to build. We have If we had data distribution at hand, if we had the probability data distribution, we would not have to go

09:52 through so much of research. Due to the fact that we don’t have it, we want a way to map from something we know, that is the Gaussian distribution, to the data distribution. We want to go from this to this. And looking at this, you you kind of understand where I’m going. We want with diffusion, with flow matching, with score matching, we want to go from a distribution that we know to a distribution which we don’t.

10:23 As I said, I’m a man of my words. I’m going to talk about Gaussian a little bit. Um, if you if you are good with Gaussian, please skip this section as you won’t need it. So, a Gaussian distribution is Let me clear this. A Gaussian distribution looks like this. It’s a bell-shaped curve. And uh it’s centered around the mean, that is mu. And the spread is uh devised with its variance, that is sigma squared. What do we mean by spread? So, if you draw this, you will see that

10:55 the bell either shrinks or spreads out with the variance. The variance goes down, the bell shrinks. If the variance goes up, the bell spreads out. And uh a mean shifts the entire bell. So, if you move the mean, you’ll see that the bell moves along with it, with the variance being the same. Okay? Now, what we want to do is we want to draw from this distribution. So, what does draw mean? Draw is to sample from a distribution along with the intuition of

11:27 the probability. So, always think about the x-axis as the samples that are there and corresponding to each sample, what is the likelihood of that sample being present. So, if it is surrounded if it is centered at zero, let me move to 0 0. Okay? And the variance is one, maybe. So, what I mean by this is if we draw,

11:57 we’re going to see um a a value between 0.5 to -0.5 a lot. And uh if we draw it, you see that a green point comes into position. We also see another green point. And you you’ll you’ll be able to figure out that it’s always between the entire spread. Does that mean it it does not overshoot and it goes to outliers? It definitely goes to outliers. That’s the whole point of probability. Probability is the math

12:29 of uncertainty. You’re never certain of what is going to happen. You’re just more certain of certain things because it’s probable. It’s it’s not definite. That’s the entire essence. And once you have the idea wrapped around your head, it’s going to be so much better. So, if you draw 50, you’ll see that there are values that overshoot the uh variance. There are values that overshoot the variance on the positive and the negative side. But,

13:00 most of the values are centered around the variance. And as you move the variance and you draw, you’ll see that the draws are more spread out now. You’ll be able to figure out that they are more towards the tail end because it’s very draw it’s very spread out. I’ve blabbered a lot and we go to the next section. Now, before going to training the velocity field, let me talk to you for a second. We We understood

13:30 a major thing about noise distribution and going from noise distribution or Gaussian to the probability data distribution. We also understood that if there is one single straight line from one data point of noise to one data point of data, we can essentially infer by using the Euler method given that we have the velocity. But how does moving from one point to another with the velocity also bridge

14:01 moving from one probability distribution to another? Now, this is a very intuitive setting that I think a little differently. If you start with noise, you essentially are starting with noise distribution. So, it’s one draw from the noise distribution. But as soon as you move or you move your no or or you move all of your samples of the noise al- along a velocity or along a velocity field, you’re essentially also transporting

14:32 your noise distribution to the data distribution. So, the vector field or the field with a lot of velocity vectors, as you might see here, this is a field with a lot of these arrows. These arrows are vector fields. The vector velocity vectors. I beg your pardon. And once you have this one point sample from a noise distribution, you are moving from noise and you’re going into

15:04 the data distribution. So, if you sample a lot of this, you have a lot of these noise data’s that you start off from. And following along this velocity, you go into the probability distribution of the data. And that transport is what is um underlying all of this flow matching bit. You have a velocity field. You want to train it, and we’ll see how to train this. And why do we train it? It’s because that helps us transport from P noise or

15:37 um P init or P Gaussian or P normal. P is probability to P of data, the data distribution that we always wanted to achieve but never had. Having said that, let’s come uh to the section. And in the section, we train the velocity field. Let’s straight away go to the code. The code is so much easier uh than the paragraphs of text. A data loader, this is very PyTorch-like. So, it’s a data loader, and you get Z. Z is one sample from the

16:08 data set or a bunch of mini batch of this data set. Epsilons are um a draw from the random normal, randn. N stands for normal. T is a sample of uh from a uniform distribution, 0 to 1. This is the time. It can be anything between 0 to 1. 0 and 1 both considered inclusive. Xt is the straight line from uh noise to data, and Xt gives you

16:39 one point in between this line, which is a mixture of or a weighted mixture of uh epsilon, that is our noise, and Z, that is our data. And target is Z minus epsilon. So, this is the ground truth. This is the known velocity. And the prediction of the model would be the target velocity. It It’s to achieve the target velocity. The loss is prediction minus target, which is the velocity which we know versus the velocity which we predict. And this is just L2 norm, which

17:10 is very simple. It’s just one the prediction minus the target squared, which is always going to give you a positive result and you mean it. And now with PyTorch, you zero grad the optimizers, you do a loss.backward, you update the gradients, and you essentially go one step with your optimizer and optimize your neural network. So, essentially what we do is we are trying to emulate the velocity

17:42 field, the entire field from P normal to P of data. And in this embed, two moons is the target distribution that we want to achieve. And once we hit train, you will see that the training loss goes down. And there will be a there will be at the very beginning a sample at as a as in a sample a lot of draws from the normal distribution, and the arrows are something that we are

18:12 training. The arrows will be the velocity field or the vectors, and it will push the samples to these two target distributions. Let’s see that in action. So, essentially it went from normal and it’s still trying it’s still trying to go to these two moons. Let’s reset. Let’s do a spiral. And then train again. See the velocities are changing and it kind of maps, but is not completely mapping to this spiral. It’s not

18:44 overfitting on the spiral, which is good. Let’s reset. Let’s do a ring. And the same thing happens all over again. Please feel free to play with the field at time T to check at which time step what the velocity is going to look like, and also see how the training loss is going down. You can also uncheck check the fields, show the target, and so so much in in this embed.

19:14 And the training objective is essentially really dense equation as as we see in the papers, but it essentially tells you that the target the target is Z minus E. You need to subtract the target from the prediction and the prediction is just UT of theta and UT is the velocity given the point of XT. So, the XT is this, prediction is XT and T given the model. The model is just going

19:45 to give you the velocity and you do a prediction minus target. Exactly this, but so much easier in code. This is the end game. We have done everything. We have looked at theory. We have looked at embeds which take uh P D P in it to P data and we want to wrap our heads around to something which we actually need. We need to train with the same objective a neural network

20:15 which lets us create images that look like handwritten digits. And this is something which Claude and Codex has helped me build. This is not me at all. Very fair disclaimer, but I have been the human in the loop to guide it more and more to what it is now. So, let’s just let’s just train. So, what we see here is a model that is getting trained in your browser with the flow objective that is

20:45 predict the velocity that is Z minus epsilon. And as the as the training updates the prediction error goes down, the training time is also something which we have here and you see that it’s saturated because there are two things that are happening. One is we are and doing a pixel wise um training error. That is we are not doing a Z minus epsilon where Z is an Z is an image and epsilon is another image of

21:17 noise. But what we do is we do a PCA of this image which compresses its an algorithm of compressing uh to only features that we need or highly impactful features. And then we do a target velocity of the PCA which is a little unintuitive but it it it’ll make sense. So you have pixels, you compress it into something which is not pixels anymore but these can be en- these are encoded which can be decoded

21:47 to the pixels again. And uh these encodings now formulate your data set and the noise is of the same space as the encodings and you have from noise to these encodings and you do the entire theory and or or better to say that you just apply the entire theory that we learned to these encodings and you would train this. So due to the fact that these compressions are not that great,

22:18 the prediction error saturates. So better the compression or better representations might um give you better errors. That is the error might come down very quickly. But we have something which is working. So that’s one. Number two that I think is this is a very naive kind of a setup. So if you have better um optimizers or if you have better ways to optimize it or to look into the gradients and figure

22:49 out what is happening, you might be able to figure out how uh you want the loss to go down easily. So that’s the first part of training, which is good. So every uh handwritten digit is somewhat noticeable. Threes are three looks looks very like eights, but you can fairly figure out zeros, but zeros look like two. Sixes look like zeros and so on. Now, once you have this vector field, that is the that is a

23:20 model that gives you the vector field, you can sample using the Euler method. And let’s target seven. And sampling steps are whatever. Like this does not really matter, but as soon as you do a sampling step of 30, you see that seven is like the Euler step as soon as you give more sampling steps to the Euler step, it goes closer to the it follows the velocity field better. It

23:50 does not overshoot, it does not undershoot. And as as you go to another target digit, say two, figure out another noise, you get two sometimes, you don’t get two sometimes. And this is essentially what happens with probability. Ah, there you have it. I was I was clicking on new noise so many times to figure out what is what is wrong, but here it looks more or less like two. But

24:23 with more sampling steps and again a new noise, looks like three. Again, a good two, but it goes from eight, apparently, two two, which is uh Don’t ask me how this happens. It just happens and it blows my mind every time that that it that it does. And with that, I want to close, but before that, I also want to make sure I give you some key takeaways. The takeaways would be

24:54 that flow matching is not as easy as this blog post tries to be. It’s much more complicated. There are a bunch of technical jargons, mathematical jargons that one would want to know when they’re doing research, but the entire essence of this blog post is for you to understand some of it. And as soon as you understand and as soon as things click, you might want to switch to something more difficult and to

25:25 switch to something more difficult, these lectures are just what you need. So, by Stanford and one of them is by MIT. So, these two lectures as I have taken a lot of help from and they are they are top class. Of course, they have to be. But, if you are a newbie, if you want to know about flow matching, just know or wrap your head around it. Julia does a

25:56 tremendous task of making sure you understand the entire diffusion. I’ll also link the playlist she has. She is incredible. And another thing that I wanted to mention, which might sound a little um how do you call it? A a little intimidating is all the notations of probability are not that complicated. It’s just that uh mathematicians want to communicate

26:29 or or engineers or whoever, if you are in a field, you want to communicate with a high efficiency and less jargon way. And to understand all of this, I had to go through a lot of papers. All of these papers had very intimidating math. And I took this course from Stanford. It’s uh introduction to probability and this changed the entire perspective of me for not only probability but also teaching and this way of communication

27:00 and I’m sorry if this video goes to ends up being a little longer but my opinion would be to also look at this course and this suddenly changes your entire perspective of teaching of probability of passionate teaching and also how do you how do you want to respect each other’s time. So to respect each other’s time mathematicians have devised symbols. The symbols are not very complicated. They just want to

27:32 signify certain things which once you know it just forms so well and so effectively so efficiently in one line or two lines that it’s crazy. So what do you take away from this? One is the lectures are all online. Go go ahead if you want to follow along. This blog post remains in hugging face as a space. If you like what I did do help me with up voting it and also if

28:05 you have any other queries if you want me to cover something else do let me know in the comments or in my Twitter DMs or just post in Twitter. That would be great. And with that uh see you next time.