Diffusion Models — Part 0: Preface
For the mathematically curious
1 A Note Before We Begin
I’ve been fascinated by unsupervised learning for quite some time. Even though much of my current work involves supervised learning and AI agents, I’ve always been drawn to the idea of learning from data without labels. The whole idea that you could train a model to discover the underlying structure of data without being assigned targets is fascinating to me.
Algorithms like k-means and PCA are still useful, but I never found them particularly exciting. Clustering observations was one thing, but the idea of generating new observations was something else altogether. It wasn’t until I stumbled upon generative adversarial networks (GANs) that my fascination with unsupervised learning really took off. Imagine my shock and awe when I first saw a GAN generate realistic images of people who never existed. Those were not photographs retrieved from a database, but images that a neural network had learned to generate from scratch. I was hooked.
By late 2022, models like Stable Diffusion, DALL-E were generating remarkably realistic images from text prompts, sometimes with minor inaccuracies. What really blew my mind was seeing the earliest version of OpenAI’s Sora. Watching a model generate a moving picture sequence was an out-of-this-world experience. Of course, Sora had its flaws and limitations, but it turned out that I wanted to understand the core mathematical principles behind these models. Image and video generation were probably the application areas where I wanted to see the math in action. How was a neural network able to learn the structure of a complicated, high-dimensional probability distribution? How was it able to generate new samples from it? What mathematical machinery made generation possible?
I wanted to dive deep into the mathematics of these models and not just import a library and call a function to generate an image or video. As I began reading, I discovered that diffusion models were an important part of the story.
There was another reason that drew me to diffusion models, almost by instinct. I had encountered diffusion processes years earlier in my thermal/statistical physics courses. Brownian motion, random walks, particles diffusing through a medium, and their evolution were all fascinating. Unfortunately, I never understood the mathematical machinery behind them. I also tried approaching stochastic calculus from a mathematical finance perspective but that did not go too well either.
So when I read about diffusion models in generative AI, there was something strangely familiar about the subject. Here was this complicated mathematical machinery that I had seen before, but this time it was being used to generate images and videos. It was a strange and exciting feeling, and I wanted to understand it better.
To my horror, I discovered that the mathematical literature on diffusion models is quite dense and requires a fair amount of mathematical maturity. The “easier” parts of diffusion models were approachable with a basic understanding of probability, calculus, and linear algebra. But the broader theory seemed to required more- stochastic calculus, stochastic differential equations, and measure theoretic probability- the bane of my existence. My previous attempts to study measure-theoretic probability and stochastic calculus had ended in frustration, hatred, and despair. I had more or less promised myself that I would never voluntarily return to those subjects.
Apparently, the universe had other plans. I didn’t want to abandon diffusion models, but I also didn’t want to spend two years studying prerequisites before beginning. I decided to take a different approach: begin with the mathematics I already knew, introduce additional machinery just in time, and be explicit about the limitations of my approach. Honestly, I am not sure if this approach will work, but I am willing to try.
2 Who Is This Series For?
This series is for anyone curious about the mathematics behind diffusion models but who may not have an eternity to spend studying life-destroying or soul-crushing mathematics like measure-theoretic probability and stochastic calculus. At the same time, this series is not for those who are looking for a quick and easy way to implement diffusion models. My goal is to help you understand the mathematics behind diffusion models, not to teach you how to use a library to generate images or videos.
Equations will be extensively used in this series, and for the most part, the mathematics will be the explanation. I will try to keep the mathematics grounded in conceptss typically encountered in undergraduate-level courses in mathematics or engineering, introducing the more advanced concepts as needed. What do I mean by that? I will assume that you are familiar with calculus, linear algebra, and probability. We will use the concepts of random variables, random vectors, probability density functions, distributions, conditional probability, and most importantly, conditional expectation a lot. For the most part, an undergraduate-level understanding of these concepts will suffice. I will also assume that you are familiar with the basics of machine learning and neural networks. If you are not, I would recommend that you first read a good introductory book on machine learning and neural networks before proceeding with this series.
Of course, the more mathematics you know, the better. But I guarantee that the mathematics will be largely accessible to those who are willing to put in the required effort. And yes, that includes the advanced concepts too. In fact, that’s how I learned the more advanced material myself. I don’t expect you to have studied graduate-level probability or stochastic calculus, but I will try to introduce the necessary concepts as we go along.
3 Who Is This Series Not For?
If you are looking for a tutorial-style introduction to using or training diffusion models, you will be disappointed. There are excellent tutorials or online articles for that. I will certainly use PyTorch and numerical experiments wherever appropriate, but the implementations exist only to illustrate the mathematics. To add some humor, I may generate high-quality AI slop images too, but they will be backed by whatever mathematics is relevant for generation.
On the other hand, this series is also not intended for diffusion models researchers looking for novel algorithms, state-of-the-art (SOTA) architectures or new ideas. I am still learning the subject myself, and my understanding continues to evolve. I will occasionally make simplifying assumptions, leave certain advanced results without proofs, and in many cases, admit my own ignorance.
Finally, if you find mathematical derivations tedious and would rather skip to the final results, you may find my pace frustrating. I intend to spend considerable time understanding why an equation takes a certain form, what assumptions went into deriving it, and what it actually tells us. After all, the mathematics is the whole point of this exercise.
4 A Note on the book: The Principles of Diffusion Models
There is a fantastic book available on diffusion models called The Principles of Diffusion Models by Chieh-Hsin Lai, Yang Song, Dongjun Kim, Yuki Mitsufuji, and Stefano Ermon. Much of what I’ve learned (and what I’m learning) about diffusion models so far has come from this masterpiece. For those who want to study this subject seriously, this book is a must-read.
My notes, on the other hand, are not a substitute for this book. Think of this series as a set of notes entitled Diffusion Models for Dummies Who Know Some Math, written by one of those dummies. I am not an AI researcher and my understanding OF diffusion models is quite limited. I am simply trying to work through the mathematics, one equation at a time.
5 A Note on AI Usage
I have used ChatGPT extensively as a learning aid while studying diffusion models, particularly when I struggled with concepts and ideas. I often used it as a tutor and a study companion. I asked questions, challenged assumptions, requested more explanations or simpler examples until I could make sense of them. I worked through the mathematics myself (by hand), often filling in 10+ pages for equations that would fit in half a page of a notebook, and checked important results against the literature.
As an AI engineer myself, I am not opposed to the use of AI tools. But I believe there’s a difference between using AI as a learning aid to sharpen your understanding and letting it do all the thinking and brainstorming for you. I want you to experience the struggle of working through the mathematics yourself, getting stuck, making mistakes and, on bad days, questioning your life choices until you reap the rewards of finally understanding something that seems impossible.
6 Enough Talk. Let’s Study Diffusion Models.
At this point, I’ve accumulated a lot of handwritten notes on diffusion models that start with the simplest model- DDPM and extend to topics involving the multidimensional Fokker-Planck equation. And these notes will keep growing as and when I add more topics into my diffusion models toolbox.
Turning these notes into something readable is another challenge altogether. Through this series, I hope to help readers understand how diffusion models work, grasp the mathematical ideas behind them, and appreciate how concepts from a wide spectrum of topics in mathematics and machine learning come together to make these remarkable generative models possible.
Enough about my notes and motivations. Let’s get to work.