Denoising Diffusion Implicit Models

Abandon Bayes and Markov

In Review DDPM and DDPM, to get the distribution $p({\mathbf{x}}_{t-1}|{\mathbf{x}}_t,{\mathbf{x}}_0)$, we applied Bayes' Theorem:

$$p({\mathbf{x}}_{t-1}|{\mathbf{x}}_t,{\mathbf{x}}_0)=\frac{p({\mathbf{x}}_t|{\mathbf{x}}_{t-1})p({\mathbf{x}}_{t-1}|{\mathbf{x}}_0)}{p({\mathbf{x}}_t|{\mathbf{x}}_0)}$$

where we set $p({\mathbf{x}}_t|{\mathbf{x}}_{t-1})$ as a series of Gaussian distribution, so that we can derive $p({\mathbf{x}}_{t-1}|{\mathbf{x}}_0)$ and $p({\mathbf{x}}_t|{\mathbf{x}}_0)$ from it and therefore obtain $p({\mathbf{x}}_{t-1}|{\mathbf{x}}_t,{\mathbf{x}}_0)$.

The Gaussian we chose for $p({\mathbf{x}}_t|{\mathbf{x}}_{t-1})$ is

$$p({\mathbf{x}}_t|{\mathbf{x}}_{t-1})=\mathcal{N}({\mathbf{x}}_t\mid \sqrt{{\alpha }_t}{\mathbf{x}}_{t-1},(1-{\alpha }_t)\text{I})$$

This transition satisfies Markov Property.

In DDIM, we choose a another way to obtain $p({\mathbf{x}}_{t-1}|{\mathbf{x}}_t,{\mathbf{x}}_0)$ by assuming it as a Gaussian

$$p({\mathbf{x}}_{t-1}|{\mathbf{x}}_t,{\mathbf{x}}_0)=\mathcal{N}({\mathbf{x}}_{t-1}\mid {\kappa }_t{\mathbf{x}}_t+{\lambda }_t{\mathbf{x}}_0,{\sigma }_t^2\text{I})$$

This is a general form for $p({\mathbf{x}}_t|{\mathbf{x}}_{t-1},{\mathbf{x}}_0)$, which should satisfy

$$\int p({\mathbf{x}}_{t-1}|{\mathbf{x}}_t,{\mathbf{x}}_0)p({\mathbf{x}}_t|{\mathbf{x}}_0)\mathrm{d}{\mathbf{x}}_t=p({\mathbf{x}}_{t-1}|{\mathbf{x}}_0)$$

Now we could get the form of $p({\mathbf{x}}_t|{\mathbf{x}}_{t-1},{\mathbf{x}}_0)$ by setting up $p({\mathbf{x}}_t|{\mathbf{x}}_0)$ and $p({\mathbf{x}}_{t-1}|{\mathbf{x}}_0)$. To reuse the model we trained in DDPM, we remain the same distribution of them

$$\begin{array}{rl}p({\mathbf{x}}_{t-1}|{\mathbf{x}}_0)& =\mathcal{N}({\mathbf{x}}_{t-1}|\sqrt{{\bar{\alpha }}_{t-1}}{\mathbf{x}}_{t-1},(1-{\bar{\alpha }}_{t-1})\text{I})\\ p({\mathbf{x}}_t|{\mathbf{x}}_0)& =\mathcal{N}({\mathbf{x}}_t|\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0,(1-{\bar{\alpha }}_t)\text{I}\end{array}$$

Then we can solve

$${\kappa }_t=\frac{\sqrt{1-{\bar{\alpha }}_{t-1}-{\sigma }_t^2}}{\sqrt{1-{\bar{\alpha }}_t}},{\lambda }_t=\sqrt{{\bar{\alpha }}_{t-1}}-\frac{\sqrt{{\bar{\alpha }}_t\left(1-{\bar{\alpha }}_{t-1}-{\sigma }_t^2\right)}}{\sqrt{1-{\bar{\alpha }}_t}}$$

So $p({\mathbf{x}}_{t-1}|{\mathbf{x}}_t,{\mathbf{x}}_0)$ is a set of distributions depend on ${\sigma }_t^2$. If we chose ${\sigma }_t^2$ as the same in DDPM, the $p({\mathbf{x}}_{t-1}|{\mathbf{x}}_t,{\mathbf{x}}_0)$ will also be the same.

Now, by Bayes' Theorem we could obtain the forward process $p({\mathbf{x}}_t|{\mathbf{x}}_{t-1},{\mathbf{x}}_0)$, which is no longer Markovian as $p({\mathbf{x}}_t|{\mathbf{x}}_{t-1},{\mathbf{x}}_0)=p({\mathbf{x}}_t|{\mathbf{x}}_{t-1})$ in DDPM

Tip

The general idea of DDIM is, specific $p({\mathbf{x}}_t|{\mathbf{x}}_0)$ and $p({\mathbf{x}}_{t-1}|{\mathbf{x}}_0)$ by ${\bar{\alpha }}_t$ and ${\bar{\alpha }}_{t-1}$ to get $p({\mathbf{x}}_{t-1}|{\mathbf{x}}_t,{\mathbf{x}}_0)$, rather than by defining $p({\mathbf{x}}_t|{\mathbf{x}}_{t-1})$ to get all other distributions in DDPM

Training and Inference

As same in DDPM, DDIM trains a denoise network ${\hat{\mathbf{x}}}_0={\hat{\mathbf{x}}}_{\theta }({\mathbf{x}}_t)$ to get $p({\mathbf{x}}_{t-1}|{\mathbf{x}}_t)$ from $p({\mathbf{x}}_{t-1}|{\mathbf{x}}_t,{\mathbf{x}}_0)$. Since the loss function, as the same in DDPM, does not include ${\sigma }_t^2$, so we can use a pre-trained DDPM model for DDIM inference by changing the value of ${\sigma }_t^2$.

In particular, if we set ${\sigma }_t^2=0$, the inference process would be determinant, and this is what the Implicit in DDIM mean.

Accelerate Generation

In DDPM, the generative process is considered as the approximation to the reverse process; since of the forward process has $T$ steps, the generative process is also forced to sample $T$ steps.

However, the denoising network we trained in DDPM does not rely on any specific forward procedure. That is, as long as ${\mathbf{x}}_t$ is fixed by $p({\mathbf{x}}_t|{\mathbf{x}}_0)$ with parameter ${\bar{\alpha }}_t$, the denoising network ${\hat{\mathbf{x}}}_0={\hat{\mathbf{x}}}_{\mathbf{\theta },t}({\mathbf{x}}_t)$ would work regardless ${\mathbf{x}}_t$ is sampled from ${\mathbf{x}}_{t-1}$ or ${\mathbf{x}}_{t-100}$. Therefore, a DDPM trained for $[1,\dots ,T]$ includes all the parameters we need for any subsequence $[{\tau }_1,\dots ,{\tau }_S]\subset [1,\dots ,T]$, and we could build a DDIM model on this new sequence without training a new model.

By this way, we can directly accelerate the generation process to $S$ steps by recalculating ${\bar{\alpha }}_1,\dots ,{\bar{\alpha }}_S$ from ${\alpha }_{{\tau }_1},\dots ,{\alpha }_{{\tau }_S}$.