Review DDPM

This article is a simplified interpretation of DDPM.

The general target of diffusion models is to learn a real world distribution $p(\mathbf{x})$. DDPM chooses a step-by-step Markov Chain to do this

$$p_{\mathbf{\theta }}({\mathbf{x}}_0)=\int p_{\mathbf{\theta }}({\mathbf{x}}_{0:T})\mathrm{d}{\mathbf{x}}_{1:T}$$

where we set $p({\mathbf{x}}_T)=\mathcal{N}({\mathbf{x}}_T|0,\text{I})$, and

$$p_{\mathbf{\theta }}({\mathbf{x}}_{0:T})=p_{\mathbf{\theta }}({\mathbf{x}}_T)\prod _{t=1}^T{p}_{\mathbf{\theta }}({\mathbf{x}}_{t-1}|{\mathbf{x}}_t)$$

We don't know what is $p_{\mathbf{\theta }}({\mathbf{x}}_{t-1}|{\mathbf{x}}_t)$, so we apply Bayes' Theorem,

$$p_{\mathbf{\theta }}({\mathbf{x}}_{t-1}|{\mathbf{x}}_t)=\frac{p_{\mathbf{\theta }}({\mathbf{x}}_t|{\mathbf{x}}_{t-1})p_{\mathbf{\theta }}({\mathbf{x}}_{t-1})}{p_{\mathbf{\theta }}({\mathbf{x}}_t)}$$

Recall the definition of the transition distribution

$$p_{\mathbf{\theta }}({\mathbf{x}}_t|{\mathbf{x}}_{t-1})=\mathcal{N}({\mathbf{x}}_t\mid \sqrt{{\alpha }_t}{\mathbf{x}}_{t-1},(1-{\alpha }_t)\text{I})$$

So we therefore can conclude $p_{\mathbf{\theta }}({\mathbf{x}}_{t-1}|{\mathbf{x}}_0)$ and $p_{\mathbf{\theta }}({\mathbf{x}}_t|{\mathbf{x}}_0)$, than we would have

$$\begin{array}{rl}{p}_{\mathbf{\theta }}({\mathbf{x}}_{t-1}|{\mathbf{x}}_t,{\mathbf{x}}_0)& =\frac{p_{\mathbf{\theta }}({\mathbf{x}}_t|{\mathbf{x}}_{t-1})p_{\mathbf{\theta }}({\mathbf{x}}_{t-1}|{\mathbf{x}}_0)}{p_{\mathbf{\theta }}({\mathbf{x}}_t|{\mathbf{x}}_0)}\\ & =\frac{\mathcal{N}({\mathbf{x}}_t|\sqrt{{\alpha }_t}{\mathbf{x}}_{t-1},(1-{\alpha }_t)\text{I})\mathcal{N}({\mathbf{x}}_{t-1}|\sqrt{{\bar{\alpha }}_{t-1}}{\mathbf{x}}_{t-1},(1-{\bar{\alpha }}_{t-1})\text{I})}{\mathcal{N}({\mathbf{x}}_t|\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0,(1-{\bar{\alpha }}_t)\text{I})}\\ & =\mathcal{N}({\mathbf{x}}_{t-1}\mid \mathbf{\mu }({\mathbf{x}}_t,{\mathbf{x}}_0),{\sigma }_t^2\text{I})\end{array}$$

where

$$\begin{array}{rl}\mathbf{\mu }({\mathbf{x}}_t,{\mathbf{x}}_0)& =\frac{(1-{\overline{\alpha }}_{t-1})\sqrt{{\alpha }_t}}{1-{\overline{\alpha }}_t}{\mathbf{x}}_t+\frac{(1-{\alpha }_t)\sqrt{{\overline{\alpha }}_{t-1}}}{1-{\overline{\alpha }}_t}{\mathbf{x}}_0\\ {\sigma }_t^2& =\frac{(1-{\alpha }_t)(1-{\overline{\alpha }}_{t-1})}{1-{\overline{\alpha }}_t}\end{array}$$

However, we want $p_{\mathbf{\theta }}({\mathbf{x}}_{t-1}|{\mathbf{x}}_t)$ other than $p_{\mathbf{\theta }}({\mathbf{x}}_{t-1}|{\mathbf{x}}_t,{\mathbf{x}}_0)$. To solve this, we train a model $\hat{\mathbf{x}}={\hat{\mathbf{x}}}_{\mathbf{\theta }}({\mathbf{x}}_t)$to predict ${\mathbf{x}}_0$ from ${\mathbf{x}}_t$, so that the probability would only depend on ${\mathbf{x}}_t$. Therefore

$$p_{\mathbf{\theta }}({\mathbf{x}}_{t-1}|{\mathbf{x}}_t)\approx p_{\mathbf{\theta }}({\mathbf{x}}_{t-1}|{\mathbf{x}}_t,{\mathbf{x}}_0={\hat{\mathbf{x}}}_{\mathbf{\theta }}({\mathbf{x}}_t))=\mathcal{N}({\mathbf{x}}_{t-1}\mid \mathbf{\mu }({\mathbf{x}}_t,{\hat{\mathbf{x}}}_{\mathbf{\theta }}({\mathbf{x}}_t)),{\sigma }_t^2\text{I})$$

This is what denoising means.