Generative Models Explained生成式模型讲解

Seen from the outside, the scene has three layers:

• Input and output: a prompt goes in and a picture that fits it comes out.
• Randomness: the same prompt can produce many different pictures, so besides the prompt the model also takes a random input, usually a set of random numbers. Dropping the prompt and keeping only the random numbers, the model simply produces some plausible picture; this is called unconditional generation. The prompt is a condition added on top, so this guide studies unconditional generation first and brings the prompt back in the diffusion chapter.
• Learning: the model is not a set of rules written by hand but is learned from a large collection of training images.

从外部看，这个场景可以分成三层：

• 输入与输出：输入一句提示词，输出一张符合描述的图片。
• 随机性：同一句提示词可以得到许多张不同的图片，说明模型除了提示词，还有一份随机输入，通常是一组随机数。去掉提示词、只保留随机数，模型就随意生成一张像样的图片，这称为无条件生成。提示词只是在此基础上加的条件，所以本文先研究无条件生成，到 Diffusion 一章再把提示词加回来。
• 学习：模型不是人写好的规则，而是从大量训练图片中学出来的。

The problem therefore becomes: given a large pile of training images, how can we build a machine that, each time it runs, produces a new image that looks as if it belonged with the training images? The next chapter makes “looks as if it belonged” precise.

于是问题变成：给定一大堆训练图片，怎样造出一台机器，每运行一次就产出一张新图片，而且看起来与训练图片属于同一类？下一章先把“看起来属于同一类”说得更精确。

The new image we want is not just any image. A 28×28 image has 784 pixels, and filling them in at random almost always gives noise. What we want is an image that looks as if it belonged with the training images, for example another handwritten digit.

Imagine every possible image sitting in a huge lottery box, each with a different share: images that look like real handwritten digits have large shares, and noise has almost none. The ten thousand training images are draws from this box, and generating a new image means drawing one more. Mathematically the box is a probability distribution, written $p_{\text{data}}(x)$, and an image's share is its probability. The difficulty is that the box itself cannot be seen; only the draws are at hand. A generative model therefore builds, from the draws, a box $p_\theta(x)$ that resembles the real one as closely as possible, and then draws from it. This problem is called density estimation. It belongs to unsupervised learning, because the training data consist only of the samples themselves, with no answers labelled by people. The appendix at the end reviews the probability ideas used throughout this guide.

For continuous values, such as pixel brightness, the probability of drawing any single exact value is zero, so another quantity is needed to describe how crowded the draws are near a point: the probability density. A density becomes a probability only after it is integrated over an interval, so the density itself can exceed 1, while the total area under the curve is always 1.

我们想要的“新图片”并不是任意一张图片。28×28 的图片有 784 个像素，随手给每个像素填一个值，几乎总是一片噪声；我们想要的是看起来与训练图片属于同一类的图片，比如又一个手写数字。

可以把所有可能的图片想象成装在一个巨大的抽奖箱里，每张图片占的份量不同：像真实手写数字的图片份量大，噪声图片的份量几乎为 0。训练集里的一万张图片，就是从这个箱子里抽出来的；生成一张新图片，就是再从箱子里抽一张。在数学上，这个箱子就是一个概率分布，记作 $p_{\text{data}}(x)$，每张图片的份量就是它的概率。困难在于箱子本身看不见，手里只有抽出来的样本。生成模型要做的，就是根据这些样本造一个尽量相像的箱子 $p_\theta(x)$，再从中抽取。这类问题称为 density estimation（密度估计）。它属于无监督学习，因为训练数据只有样本本身，没有人工标注的答案。全文用到的概率知识，在文末附录中有单独的介绍。

对于连续取值的量，例如像素的亮度，抽中某一个精确值的概率是 0，因此需要另一个量来描述“这个值附近的样本有多密集”，即概率密度。密度只有在一段区间上积分之后才成为概率，所以它本身可以大于 1，而曲线下的总面积始终等于 1。

The previous chapter described generation as building a lottery box $p_\theta(x)$ that resembles the real one and drawing from it. Building it well requires a way to judge the resemblance, and a natural test is this: real training images, looked up in the model's box, should turn out to be common rather than rare. To run this test, the most direct approach is to write down a formula for $p_\theta(x)$, so that the share of any image can be computed. For one-dimensional data, for instance, one may assume a Gaussian, whose parameters θ are its mean μ and standard deviation σ; each choice of θ gives a different $p_\theta(x)$. The standard way to choose θ is maximum likelihood estimation, which makes the training data as probable as possible under the model. The formula uses the notation argmax, which differs from max:

• $\max_\theta f(\theta)$: the largest value the function reaches, an output.
• $\arg\max_\theta f(\theta)$: the θ at which that largest value is reached, an input. Read it as “the θ that makes $f(\theta)$ largest”.

For example, $f(\theta)=-(\theta-2)^2$ is largest at θ = 2, where it equals 0, so $\max_\theta f(\theta)=0$ while $\arg\max_\theta f(\theta)=2$. Training needs the parameters themselves, not the best score, so the formula uses argmax:

上一章把生成描述为：造一个尽量像真实箱子的 $p_\theta(x)$，再从中抽取。要把箱子造好，先得有办法判断它像不像。一个自然的标准是：把真实的训练图片放到模型的箱子里查一查，它们应该是常见的，而不是罕见的。为了做这个检查，最直接的做法是写出 $p_\theta(x)$ 的具体公式，这样任何一张图片的份量都能算出来。例如对一维数据，可以假设它服从高斯分布，参数 θ 就是均值 μ 和标准差 σ；不同的 θ 给出不同的 $p_\theta(x)$。挑选 θ 的标准做法是 maximum likelihood estimation（最大似然估计），即让训练数据在模型下出现的概率尽可能大。式中用到记号 argmax，它与 max 不同：

• $\max_\theta f(\theta)$：函数能达到的最大值，是一个输出。
• $\arg\max_\theta f(\theta)$：取得这个最大值时的 θ，是一个输入，读作“使 $f(\theta)$ 最大的 θ”。

例如 $f(\theta)=-(\theta-2)^2$ 在 θ = 2 时最大，最大值为 0，于是 $\max_\theta f(\theta)=0$，而 $\arg\max_\theta f(\theta)=2$。训练要的是参数本身，而不是最好的分数，所以公式里写的是 argmax：

Taking the logarithm does not move the maximum, but it turns a product of per-sample probabilities into a sum and is numerically more stable. In the figure below, the height of each vertical line is one sample's $p_\theta(x^{(i)})$, and the log-likelihood is the sum of the logarithms of these heights. The method has one precondition: $p_\theta(x)$ must be written down and computed. For one-dimensional data a Gaussian takes a single line; for 784-dimensional images it is far harder, as the next chapter discusses.

取对数不改变最大值的位置，却把各样本概率的连乘变成求和，数值上也更稳定。下图中每条竖线的高度就是一个样本的 $p_\theta(x^{(i)})$，对数似然是这些高度取对数后的总和。这个方法有一个前提：必须能写出 $p_\theta(x)$ 并算出它的值。对一维数据，高斯分布的公式一行就能写完；换成 784 维的图片就难得多，下一章讨论这一点。

Putting the two previous chapters together: to generate, we draw from $p_\theta(x)$; to train, we check how probable the training images are under $p_\theta(x)$ and adjust the parameters by maximum likelihood. Both rely on “writing down $p_\theta(x)$”, which means giving a function that takes any $x$ and returns how probable it is. For one-dimensional data the Gaussian formula is such a function. For images it would be a function that takes an image and returns the probability of that image.

The difficulty is a requirement every probability must meet: over all possible images, the probabilities add up to exactly 1. Getting a neural network to give an image a score is easy, but a score is not yet a probability. To turn scores into probabilities, each score must be divided by the total score of all images, just as a candidate's votes give a share only once the total number of votes is known. There are far too many possible images to add them all up, as the experiment below shows.

把前两章合起来看：生成时，从 $p_\theta(x)$ 中抽取；训练时，检查训练图片在 $p_\theta(x)$ 下有多常见，再用最大似然调整参数。两者都依赖“写出 $p_\theta(x)$”，也就是给出一个函数，输入任意一个 $x$，输出它出现的概率。对一维数据，高斯分布的公式就是这样的函数。换成图片，就需要一个函数：输入一张图片，输出这张图片出现的概率。

难点在于概率必须满足一个要求：所有可能的图片，概率加起来恰好等于 1。让神经网络给一张图片打一个分数并不难，但分数还不是概率。要把分数换算成概率，必须除以所有图片分数的总和，就像只知道某位候选人的票数还算不出得票率，还要知道总票数。而可能的图片多到根本加不完，下面的小实验演示这一点。

Two routes grew out of this difficulty, and the first split of the taxonomy follows them:

• Explicit density: keep the plan above. The model can compute the probability of any image, and a special structure makes the probabilities add up to 1 automatically, so the total never has to be computed. Training uses maximum likelihood; generation draws images according to these probabilities. The price is that the special structure limits how the model can be built; later chapters show two such structures.
• Implicit density: skip computing probabilities and build the generating machine directly. It takes a set of random numbers and outputs an image, so each run is one draw from the box, without the box ever being written down. This sidesteps the total entirely, and the network can be built any way we like. The price is that the model cannot say how probable an image is and cannot use maximum likelihood, so training needs some other way to judge whether its images look real.

The figure below compares how the two routes train and generate.

由这个难点分出了两条路线，分类图的第一层正是按此划分的：

• Explicit density（显式密度）：坚持上面的方案。模型能算出任意一张图片的概率，并通过一种特殊的结构，使所有概率自动加起来等于 1，因而不必真的去求总和。训练时用最大似然，生成时按这些概率抽取图片。代价是特殊结构限制了模型的设计方式，后面的章节会介绍两种这样的结构。
• Implicit density（隐式密度）：跳过“算概率”这一步，直接造那台生成机器：输入一组随机数，输出一张图片，每运行一次就相当于从箱子里抽一张，而箱子本身从未被写出来。这样完全绕开了求总和的问题，网络也可以随意设计；代价是模型无法回答某张图片出现的可能性有多大，也不能用最大似然训练，需要另找办法判断生成的图片像不像真的。

下图对比两条路线怎样训练、怎样生成。

The two models below have learned the same two-dimensional distribution. The explicit model on the left can report the density at any point; the implicit model on the right can only keep sampling, and to learn the density it has to approximate it with a histogram of its samples.

下面两个模型学到的是同一个二维分布。左侧的 explicit 模型能回答任意一点的密度；右侧的 implicit 模型只能不断采样，若想知道密度，只能用样本的直方图去逼近。

Following these two routes, and the specific techniques within each, the common generative models can be arranged into the map below¹. The next three parts go through it in order.

按照这两条路线，以及每条路线内部的具体做法，常见的生成模型可以整理成下面这张分类图¹。后面三部分将依次介绍。

Both routes are built from neural networks. For the purposes of this guide, a neural network can be treated as a function with adjustable parameters: numbers go in, other numbers come out, and the parameters θ decide how one becomes the other. A 28×28 grayscale image is simply 784 numbers, and training means adjusting θ again and again until the outputs come closer to what is wanted. Two kinds of network appear throughout generative models. An encoder compresses an image into a few numbers, called a latent vector; a decoder does the reverse and expands a few numbers into an image. The encoder's output is smaller than its input and the decoder's is larger, so the diagrams draw them as trapezoids that narrow or widen from left to right.

Generative models are built by combining networks like these. Each later chapter that introduces a model includes a structure diagram showing which networks it uses, what each takes in and puts out, and how training differs from generation. The boxes in these diagrams are colored by what they hold:

• Data (blue): real images from the training set.
• Networks (purple): the parts with parameters that training adjusts.
• Latent vectors and noise (gray): a few numbers, or random numbers drawn from a standard normal distribution.
• Loss (red): a single number measuring how far an output is from what is wanted; training makes it smaller.
• Generated output (green): what the model produces.

两条路线都由神经网络搭建而成。在本文中，神经网络可以先看作一个带可调参数的函数：输入一组数，输出另一组数，参数 θ 决定输入怎样变成输出。一张 28×28 的灰度图片就是 784 个数；训练就是反复调整 θ，让输出更接近要求。生成模型里反复出现两类网络。Encoder（编码器）把一张图片压缩成少数几个数，称为 latent 向量；decoder（解码器）反过来，把少数几个数展开成一张图片。前者的输出比输入小，后者的输出比输入大，所以图中分别画成从左到右收窄和展开的梯形。

生成模型就是由这样的网络组合而成的。后面各章介绍具体模型时，都附有一张结构图，说明它由哪些网络组成、每个网络输入什么、输出什么，以及训练时和生成时的路线有何不同。结构图中方框的颜色表示其中的内容：

• 数据（蓝色）：训练集中的真实图片。
• 网络（紫色）：带参数、由训练调整的部分。
• latent 向量与噪声（灰色）：少数几个数，或从标准正态分布抽出的随机数。
• 损失（红色）：衡量输出离要求有多远的一个数，训练让它变小。
• 生成结果（绿色）：模型产出的内容。

An image has hundreds of pixels, and writing down their joint distribution directly is very hard. An autoregressive model takes another angle: it lays the pixels out in a sequence and predicts each pixel from the ones before it. The figure below shows its structure.

一张图片有几百个像素，直接写出它们的联合分布十分困难。Autoregressive 模型换了一个角度：把像素排成一个序列，每次根据前面的像素预测下一个像素。下图是它的结构。

The approach rests on the chain rule of probability, which guarantees that any joint distribution can be factored into a product of conditional distributions:

这种做法的依据是概率论的链式法则：任何联合分布都可以拆成一串条件分布的乘积：

Generating an image then becomes predicting pixels one at a time. Each step only has to answer “given the pixels so far, how likely is each value of the next one?”, which is exactly what an RNN or a masked Transformer does well³. Every factor can be computed exactly, so their product $p(x)$ can be too, and these models therefore have a tractable density. The price is that generation must proceed in order, so T pixels take T steps; and an image is two-dimensional, so forcing it into a single line is unnatural.

于是生成一张图片变成逐个预测像素，每一步只需回答“已知前面的像素，下一个像素取各个值的概率是多少”，这正是 RNN 或带掩码的 Transformer 擅长的事³。每个因子都能精确算出，它们的乘积 $p(x)$ 也就能精确算出，这类模型因而属于 tractable density（可精确计算的密度）。代价是生成必须按顺序进行，T 个像素就需要 T 步；而且图像本身是二维的，硬排成一行并不自然。

A 28×28 image has 784 pixel values, but a random point in this 784-dimensional space is almost always noise. Real images occupy a tiny part of it, and they usually differ from one another along only a few factors, such as shape, stroke width and slant. A latent space is a low-dimensional coordinate system for these factors: each image corresponds to a point $z$ in it, and a decoder network turns $z$ back into pixels.

In the example below the latent space has only two dimensions: the horizontal axis controls shape and the vertical axis controls aspect ratio. The axes learned by a real model rarely have such tidy meanings, but the principle is the same. Nearby points in latent space decode to similar images, and moving along a straight line produces a smooth change in the image.

28×28 的图片有 784 个像素值，但在这 784 维空间里随手取一个点，几乎总是一片噪声；真实图片只占据其中极小的一部分，而且彼此的差别往往只体现在少数几个因素上，比如形状、粗细和倾斜。Latent space（潜空间）就是为这些因素建立的低维坐标系：每张图片对应其中的一个点 $z$，再由一个 decoder 网络把 $z$ 还原成像素。

下面的例子中 latent space 只有两维，横轴控制形状，纵轴控制横纵比。真实模型学到的坐标轴通常没有这样整齐的含义，但道理相同：latent space 中相距很近的点解码出相似的图片，沿一条直线移动对应图片的平滑变化。

The benefit of this structure is clearest in interpolation. Averaging two images in pixel space gives only a double exposure of two translucent shapes, which is not an image the dataset would contain. Taking points along the line between two latent codes and decoding them gives genuine intermediate shapes.

这种结构的好处在插值时最为明显。在像素空间里直接对两张图片取平均，得到的只是两个半透明形状的叠影，并不是数据集里会出现的图片；在 latent space 里取两点连线上的中间点再解码，得到的则是一个真正的中间形状。

The previous chapter assumed a decoder that turns a point in latent space back into an image. An autoencoder is the most direct way to obtain one: an encoder compresses the image into a few numbers $z$, a decoder expands $z$ back into an image, and training asks only that the result be as close to the original as possible. The figure below shows its structure.

上一章假定已经有一个 decoder，能把 latent space 中的点还原成图片。Autoencoder（自编码器）是得到这样一个 decoder 最直接的办法：encoder 把图片压缩成少数几个数 $z$，decoder 再把 $z$ 还原成图片，训练只要求还原出的图片与原图尽量接近。下图是它的结构。

A trained encoder assigns each training image one fixed point, so the same image lands in the same place however many times it is encoded. The training objective cares only whether these points decode back to their images, not about the regions between them, so the codes end up scattered across the plane with large gaps the decoder has never seen. This is where generation runs into trouble: a $z$ picked at random usually falls into a gap and decodes to something that looks like nothing.

训练好的 encoder 给每张训练图片分配一个固定的点，同一张图片无论编码多少次，都落在同一个位置。训练目标只关心这些点能否还原成原图，并不关心点与点之间的区域，于是编码散落在平面各处，中间留着大片 decoder 从未见过的空白。用它生成新图片时，问题就出在这里：随手取一个 $z$，多半落在空白处，解码出来什么也不像。

A plain autoencoder is therefore good for compressing images and extracting features, but it cannot simply be used to generate. For a randomly picked $z$ to decode to a reasonable image as well, the codes have to fill a region agreed on in advance. That is what the VAE in the next chapter does.

因此普通 autoencoder 适合压缩图片、提取特征，却不能直接用来生成。要让随手取的 $z$ 也能解码成像样的图片，编码就必须铺满一块事先约定的区域，下一章的 VAE 正是这样做的。

A VAE (variational autoencoder) keeps the encoder and decoder of an autoencoder and makes two changes, both aimed at filling the gaps seen in the previous chapter⁴:

• Encoder output: a distribution $\mathcal N(\mu,\sigma^2)$ instead of a point, and each encoding draws a fresh $z$ from it.
• Training objective: besides accurate reconstruction, the distribution must also stay close to a prior agreed on in advance, $\mathcal N(0,I)$.

The figure below shows the structure of a VAE; Compare with Autoencoder sets the two models one above the other.

VAE（variational autoencoder，变分自编码器）保留了 autoencoder 的 encoder 与 decoder，只做两处改动，目的都是填满上一章看到的空白⁴：

• Encoder 的输出：一个分布 $\mathcal N(\mu,\sigma^2)$，而不是一个点，每次编码都从中抽一个新的 $z$。
• 训练目标：除了要求重建准确，还要求这个分布接近事先约定的先验 $\mathcal N(0,I)$。

下图是 VAE 的结构，“与 Autoencoder 对比”把两种模型上下并列。

The first change means the decoder is trained not on a few isolated points but on a small region around each one. The second pulls all of these regions toward the origin until they join up. Together they leave no large gaps in the latent space, so a $z$ drawn at random from the prior still decodes to a reasonable image. The experiment below feeds the same image into both kinds of encoder again and again.

第一处改动使 decoder 在训练中见到的不是孤立的几个点，而是每个点周围的一小片区域；第二处改动把这些区域都拉向原点，直到彼此衔接。两者合起来，latent space 中就不再有大片空白，从先验中随手抽一个 $z$ 也能解码成像样的图片。下面的小实验把同一张图片反复送进两种 encoder。

The two requirements of the training objective, accurate reconstruction and closeness to the prior, can be derived from maximum likelihood. With a latent variable in the model, the probability of an image becomes $p_\theta(x)=\int p_\theta(x\mid z)\,p(z)\,dz$, an integral over every possible $z$, which cannot be computed exactly. The VAE therefore maximizes a lower bound on its logarithm, the ELBO (evidence lower bound):

训练目标的两项要求，即重建准确、分布接近先验，可以从最大似然推导出来。模型中有了 latent 变量之后，一张图片的概率变成 $p_\theta(x)=\int p_\theta(x\mid z)\,p(z)\,dz$，需要对所有可能的 $z$ 积分，无法精确计算。VAE 因此转而最大化它的对数的一个下界，即 ELBO（evidence lower bound，证据下界）：

Why is the right-hand side a lower bound? The exact relation is $\log p_\theta(x)=\text{ELBO}+D_{\mathrm{KL}}\big(q_\phi(z\mid x)\,\|\,p_\theta(z\mid x)\big)$. The last term measures how far the encoder's guess is from the true posterior $p_\theta(z\mid x)$, which cannot be computed; but a KL divergence is never negative, so the ELBO can never exceed $\log p_\theta(x)$. Raising the ELBO therefore either raises $\log p_\theta(x)$ itself or narrows the gap between the two.

右边为什么是下界？精确的关系是 $\log p_\theta(x)=\text{ELBO}+D_{\mathrm{KL}}\big(q_\phi(z\mid x)\,\|\,p_\theta(z\mid x)\big)$。最后一项衡量 encoder 的估计离真正的后验 $p_\theta(z\mid x)$ 有多远，这个后验算不出来；但 KL 散度永远不小于 0，所以 ELBO 永远不超过 $\log p_\theta(x)$。推高 ELBO，要么推高 $\log p_\theta(x)$ 本身，要么缩小两者之间的差距。

The VAE is therefore an approximate density model. The second term is a KL divergence, and the two terms pull against each other:

• The reconstruction term wants σ near 0 for every $x$ and the μ values well apart, which makes decoding easiest.
• The prior term wants every μ near 0 and every σ near 1, squeezing all codes into a standard normal distribution.

The experiment below uses a weight β to set the balance between the two terms; β = 1 is the standard VAE¹⁰. At β = 0 the model reduces to a plain autoencoder, and the gaps between codes decode to noise. At moderate β the distributions join up, and every $z$ drawn from the prior decodes to a reasonable image. At large β all the distributions pile on top of one another, and the decoder can only output an average of the images, which looks blurry. This is also why VAE samples tend to be blurry.

VAE 因此属于 approximate density（近似密度）。式中第二项是 KL 散度，两项之间存在拉锯：

• 重建项希望每个 $x$ 的 σ 趋于 0、μ 彼此分开，这样 decoder 最容易还原。
• 先验项希望每个 μ 趋于 0、σ 趋于 1，让所有编码都挤成标准正态分布。

下面的小实验用一个权重 β 调节两项的比例，β = 1 即标准 VAE¹⁰。β 为 0 时退化为普通 autoencoder，编码之间的空白处解码出噪声；β 适中时各分布彼此衔接，从先验中采样的 $z$ 都能解码成像样的图片；β 过大时所有分布挤在一起，decoder 只好输出各图片的平均，结果变得模糊。VAE 样本普遍偏模糊，原因也在于此。

Training meets one more technical obstacle. The reconstruction term needs a sample $z$ from $q_\phi(z\mid x)$, and sampling has no derivative, so gradients cannot flow back to the μ and σ produced by the encoder. The reparameterization trick moves the randomness into an independent noise variable ε:

训练时还有一个技术障碍。重建项需要从 $q_\phi(z\mid x)$ 中采样 $z$，而“采样”这个操作没有导数，梯度无法传回 encoder 输出的 μ 和 σ。重参数化技巧把随机性挪到一个独立的噪声 ε 上：

Now $z$ is an ordinary function of μ and σ, with $\partial z/\partial\mu=1$ and $\partial z/\partial\sigma=\epsilon$, so gradients pass through; ε is just an input that needs no gradient. Each forward pass draws a fresh ε, and the backward pass treats that ε as a constant.

这样 $z$ 成为 μ 和 σ 的普通函数，$\partial z/\partial\mu=1$，$\partial z/\partial\sigma=\epsilon$，梯度可以顺利回传；ε 只是一个不需要求导的输入。每次前向传播重新抽一个 ε，反向传播时把这次的 ε 当作常数即可。

Once $p_\theta(x)$ is given up, maximum likelihood is no longer available, and another way is needed to judge whether the samples are good. A GAN trains a second network, the discriminator D, to act as a judge. The generator G turns random noise $z\sim p(z)$ into a sample $x=G(z)$, and D outputs $D(x)$, the probability that $x$ came from the real data⁵. The figure below shows its structure.

放弃写出 $p_\theta(x)$ 之后，最大似然无从谈起，需要另一种方式判断生成得好不好。GAN 的思路是再训练一个判别器 D 充当裁判：generator G 把随机噪声 $z\sim p(z)$ 变成样本 $x=G(z)$，D 输出 $D(x)$，即 $x$ 来自真实数据的概率⁵。下图是它的结构。

The objectives of the two networks are exactly opposed:

两个网络的目标正好相反：

D wants to label real samples 1 and generated samples 0; G wants D to label generated samples 1 as well. At the ideal equilibrium $p_G=p_{\text{data}}$, and D outputs 0.5 everywhere. G is a one-way map $z\to x$ with no matching encoder, so a GAN neither gives $p(x)$ nor maps an image back to $z$. What it gains in return is a sharp sample in a single step.

D 希望把真实样本判为 1、生成样本判为 0；G 希望 D 把生成样本也判为 1。理想的平衡点上 $p_G=p_{\text{data}}$，D 处处输出 0.5。G 是一个只能向前走的映射 $z\to x$，没有对应的 encoder，所以 GAN 既不给出 $p(x)$，也无法把一张图片映射回 $z$；它换来的是一步即可得到的清晰样本。

Training has no loss curve that shows progress, since the two networks gain and lose ground against each other. A common failure is mode collapse, in which G learns to produce only one of the peaks. When the discriminator is strong from the start, G is first pushed toward the nearest peak; from then on it can lower its loss simply by fooling D on that peak, so it never goes looking for the other one. Choosing “D trains 300 steps first” in the experiment above reproduces this.

A second problem lies in G's objective. Early in training the samples are poor, D sees through them easily, and $D(G(z))$ is close to 0. At exactly this end $\log(1-D)$ is almost flat, so G receives a gradient near 0 and learns almost nothing. In practice G minimizes $-\log D(G(z))$ instead. The two objectives share the same optimum, but the new one is steepest when $D(G(z))$ is near 0, providing the largest gradient exactly when it is most needed. The GAN experiment above uses this form.

训练过程没有一条能说明进展的损失曲线，两个网络此消彼长。常见的问题是 mode collapse（模式坍塌），即 G 只学会生成其中一个峰：判别器一开始就很强时，G 先被推向离它最近的那个峰，之后只要在这个峰上骗过 D 就能降低损失，便不再去探索另一个峰。在上面的实验中选择“D 先训练 300 步”即可复现这种情形。

另一个问题出在 G 的目标函数上。训练初期生成样本很差，D 很容易识破，$D(G(z))$ 接近 0；而 $\log(1-D)$ 恰好在这一端几乎是平的，G 得到的梯度接近 0，几乎学不到东西。实践中把 G 的目标换成最小化 $-\log D(G(z))$，两者的最优点相同，但新目标在 $D(G(z))$ 接近 0 时最陡，正好在最需要的时候提供最大的梯度。上面的 GAN 实验用的就是这一写法。

A GAN asks the generator to turn noise into data in a single step, which is hard and makes training unstable. Diffusion models split this into many small steps: first fix a way to add noise to the data gradually, then train a network to remove only a little noise at a time⁶. The figure below shows its structure.

GAN 要求 generator 一步就把噪声变成数据，这件事很难，训练也不稳定。Diffusion 模型把它拆成许多小步：先规定如何逐渐给数据加噪，再训练网络每次只去掉一点噪声⁶。下图是它的结构。

The idea has several mathematical formulations, of which rectified flow is the simplest⁷. It draws a straight line between the data $x$ and noise of the same shape $z\sim\mathcal N(0,I)$:

这一思路有多种数学表述，其中 rectified flow 最为简洁⁷。它在数据 $x$ 与同形状的噪声 $z\sim\mathcal N(0,I)$ 之间连一条直线：

The network $f_\theta(x_t,t)$ learns to predict the velocity $v$, with loss $\|f_\theta(x_t,t)-v\|_2^2$. This is an ordinary regression problem, and far more stable than the adversarial training of a GAN.

网络 $f_\theta(x_t,t)$ 的任务是预测速度 $v$，损失为 $\|f_\theta(x_t,t)-v\|_2^2$，这是一个普通的回归问题，比 GAN 的对抗训练稳定得多。

The same $x_t$ can arise from many different pairs $(x,z)$, so the network can only learn the conditional expectation of the velocity, $\mathbb E[v\mid x_t,t]$. At $t=1$ pure noise carries no information about the data, so the velocity can only point toward the mean of all the data; a single step would send every sample to somewhere near that mean, the same source of blur as in a VAE. With more steps, each step re-estimates the direction from the current position, and the samples gradually separate and settle onto the data. Diffusion sampling is therefore indirect: there is no direct map $z\to x$, only repeated calls to $f_\theta$ starting from noise.

To control which class is generated, the condition $y$ can also be fed to the network, and during training $y$ is randomly replaced by an empty condition $\varnothing$, so that a single network learns both the conditional and the unconditional velocity. At sampling time, classifier-free guidance extrapolates between the two⁸:

同一个 $x_t$ 可能来自许多不同的 $(x,z)$ 组合，因此网络只能学到速度的条件期望 $\mathbb E[v\mid x_t,t]$。在 $t=1$ 处，纯噪声不含任何关于数据的信息，速度只能指向全体数据的平均位置；若只走一步，所有样本都会落在平均值附近，这与 VAE 的模糊同出一源。步数增加之后，每一步都根据当前位置重新估计方向，样本才逐渐分开、落到数据上。所以 diffusion 的采样是间接的：它没有一个直接的 $z\to x$ 映射，而是从噪声出发反复调用 $f_\theta$。

若要控制生成的类别，可以把条件 $y$ 一并输入网络，训练时随机把 $y$ 替换为空条件 $\varnothing$，使同一个网络同时学会有条件与无条件两种速度。采样时用 classifier-free guidance（无分类器引导）把二者外推⁸：

$1+w=0$ gives unconditional generation, $1+w=1$ ordinary conditional generation, and $1+w>1$ strengthens the condition further.

$1+w=0$ 为无条件生成，$1+w=1$ 为普通的条件生成，$1+w>1$ 则进一步强化条件。

Running diffusion directly in pixel space is expensive, because the noise must have the same shape as the image. A latent diffusion model combines the earlier families⁹. A VAE first trains an encoder and a decoder that compress images into a smaller latent space, with a GAN discriminator loss making the decoder's output sharper. The encoder is then frozen and a diffusion model is trained on the latents, with conditions such as text injected through cross-attention. To generate, the model denoises iteratively in latent space starting from noise, and the decoder turns the result back into an image.

直接在像素空间做 diffusion 的计算量很大，因为噪声必须与图片同形状。Latent diffusion model 把前面几类模型组合在一起⁹：先用 VAE 训练一对 encoder 与 decoder，把图片压缩到较小的 latent space，并借助 GAN 的判别器损失让 decoder 的输出更清晰；随后冻结 encoder，在 latent 上训练 diffusion 模型，文本等条件通过 cross-attention 注入。生成时在 latent space 中从噪声开始迭代去噪，最后交给 decoder 还原成图片。

The table below compares the four families.下表把四类模型放在一起比较。