Evidence Lower Bound (ELBO)

Derive the ELBO

In variational inference we want to approximate the true posterior distribution $p_{\theta }(z|x)$ from a known distribution family $q_{\varphi }(z|x)$.

We often choose KL divergence as the loss function to measure the difference between two probability distributions. Therefore, we need to minimize

$$\begin{array}{rl}{\mathbb{D}}_{\mathrm{KL}}(q_{\varphi }(z|x)\Vert p_{\theta }(z|x))& ={\mathbb{E}}_{q_{\varphi }(z|x)}\left[\log \frac{q_{\varphi }(z|x)}{p_{\theta }(z|x)}\right]\\ & ={\mathbb{E}}_{q_{\varphi }(z|x)}\log q_{\varphi }(z|x)-{\mathbb{E}}_{q_{\varphi }(z|x)}\log p_{\theta }(z|x)\\ & ={\mathbb{E}}_{q_{\varphi }(z|x)}\log q_{\varphi }(z|x)-{\mathbb{E}}_{q_{\varphi }(z|x)}\left\{\log \left[\frac{p_{\theta }(x,z)}{p_{\theta }(x)}\right]\right\}\\ & ={\mathbb{E}}_{q_{\varphi }(z|x)}\log q_{\varphi }(z|x)-{\mathbb{E}}_{q_{\varphi }(z|x)}\left[\log p_{\theta }(x,z)\right]+{\mathbb{E}}_{q_{\varphi }(z|x)}\left[\log p_{\theta }(x)\right]\\ & =\underset{-\mathrm{ELBO}}{\underbrace{{\mathbb{E}}_{q_{\varphi }(z|x)}\log q_{\varphi }(z|x)-{\mathbb{E}}_{q_{\varphi }(z|x)}\left[\log p_{\theta }(x,z)\right]}}+\log p_{\theta }(x)\end{array}$$

where we define

$$\mathrm{ELBO}(x)={\mathbb{E}}_{q_{\varphi }(z|x)}\left[\log p_{\theta }(x,z)\right]-{\mathbb{E}}_{q_{\varphi }(z|x)}\log q_{\varphi }(z|x)$$

Then we have

$$\log p_{\theta }(x)=\mathrm{ELBO}(x)+{\mathbb{D}}_{\mathrm{KL}}(q_{\varphi }(z|x)\Vert p_{\theta }(z|x))$$

Since the left hand side is constant (not depending on $\varphi$), too minimize KL divergence is equivalent to maximize $\mathrm{ELBO}(\varphi )$

Why Evidence Lower Bound

The name of ELBO comes from ${\mathbb{D}}_{\mathrm{DL}}(\cdot )\ge 0⟹\log p_{\theta }(x)\ge \mathrm{ELBO}(x)$