Normal Distribution (Gaussian Distribution)

Standard Normal Distribution

A continuous r.v. $Z$ is said to have the standard normal distribution if its PDF $\phi$ is given by

$$\phi (z)=\frac{1}{\sqrt{2\pi }}{e}^{-z^2/2},-\infty <z<\infty$$

We write this as $Z\sim \mathcal{N}(0,1)$ since, as we could obtain $Z$ has mean $0$ and variance $1$

The corresponding CDF is

$$\Phi (x)=\frac{1}{\sqrt{2\pi }}{\int }_{-\infty }^x\exp \left(-\frac{t^2}{2}\right)\mathrm{d}t$$

Normal Distribution

If $Z\sim \mathcal{N}(0,1)$, then

$$X=\mu +\sigma Z$$

is said to have the normal distribution with mean $\mu$ and variance ${\sigma }^2$, for any real $\mu$ and ${\sigma }^2$ with $\sigma >0$. We denote this by $X\sim \mathcal{N}(\mu ,{\sigma }^2)$

And the PDF of Normal distribution is

$$f(x)=\frac{1}{\sqrt{2\pi }\sigma }\exp \left[-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right]$$

Moments

Non-central

$$\mathrm{E}[(X-\mu {)}^p]=\{\begin{array}{ll}0,& \text{if p is odd},\\ {\sigma }^p(p-1)!!,& \text{if p is even.}\end{array}$$

Central

Order	Non-central moment	Central moment
1	$\mu$	0
2	${\mu }^2+{\sigma }^2$	${\sigma }^2$
3	${\mu }^3+3\mu {\sigma }^2$	0
4	${\mu }^4+6{\mu }^2{\sigma }^2+3{\sigma }^4$	$3{\sigma }^4$
5	${\mu }^5+10{\mu }^3{\sigma }^2+15\mu {\sigma }^4$	0
6	${\mu }^6+15{\mu }^4{\sigma }^2+45{\mu }^2{\sigma }^4+15{\sigma }^6$	$15{\sigma }^6$
7	${\mu }^7+21{\mu }^5{\sigma }^2+105{\mu }^3{\sigma }^4+105\mu {\sigma }^6$	0
8	${\mu }^8+28{\mu }^6{\sigma }^2+210{\mu }^4{\sigma }^4+420{\mu }^2{\sigma }^6+105{\sigma }^8$	$105{\sigma }^8$

Log Normal

Let $Z$ be a standard normal variable, and let $\mu$ and $\sigma$ be two real numbers, with $\sigma >0$. Then, the distribution of the random variable

$$X=\exp (\mu +\sigma Z)$$

is called the log-normal distribution with parameters $\mu$ and $\sigma$. We can also write

$$\ln (X)\sim \mathcal{N}(\mu ,{\sigma }^2)⟺X\sim \mathrm{LN}(\mu ,{\sigma }^2)$$

Multivariate Normal

A $k$-dimensional random vector $\text{X}=(X_1,\dots ,X_k)$ is said to have a Multivariate Normal (MVN) distribution if every linear combination of the $X_j$ has a Normal distribution. That is, we require

$$t_1{X}_1+\cdots +t_k{X}_k$$

to have a Normal distribution for any constants $t_1,\dots ,t_k$. If $t_1{X}_1+\cdots +t_k{X}_k$ is a constant, we also consider it to have a Normal distribution with variance $0$. An important special case is $k=2$; this distribution is called the Bivariate Normal (BVN)

If $(X_1,\dots ,X_k)$ is MVN, then the marginal distribution of $X_i$ is Normal, since we can take $t_i$ to be $1$ and all other $t_j$ to be $0$. However, the converse is false: it is possible to have Normally distributed r.v.s. $X_1,\dots ,X_k$ such that $(X_1,\dots ,X_k)$ is not Multivariate Normal.

If $\text{X}=(X_1,\dots ,X_n)$ and $\text{Y}=(Y_1,\dots ,Y_m)$ are Multivariate Normal random vectors with $\text{X}$ independent of $\text{Y}$, then the concatenated random vector $\text{W}=(X_1,\dots ,X_n,Y_1,\dots ,Y_m)$ is Multivariate Normal

To specify an MVN random vector $(X_1,\dots ,X_k)$, we need the parameters as follows:

• the mean vector $({\mu }_1,\dots ,{\mu }_k)$ where $\mathbb{E}[X_j]={\mu }_j$
• the covariance matrix, which is the $k\times k$ matrix of covariances between components, arranged so that the $(i,j)$ entry is $\mathrm{Cov}(X_i,X_j)$

Bivariate Normal

Let $(Z,W)$ be BVN with $\mathcal{N}(0,1)$ marginals and $\mathrm{Corr}(Z,W)=\rho$. By the definition of Multivariate Normal, any $(Z,W)$ of the form

$$\begin{array}{rl}Z& =aX+bY\\ W& =cX+dY\end{array}$$

will be Bivariate Normal, where $X,Y$ i.i.d. $\mathcal{N}(0,1)$. The means are already $0$. Setting the variance qual to $1$ gives

$$a^2+b^2=1,c^2+d^2=1$$

Setting the covariance of $Z$ and $W$ equal to $\rho$ gives $ac+bd=\rho$. There are more unknowns than equations here, and we just need one solution. To simplify, let’s look for a solution with $b=0$, then we have

$$\begin{array}{rl}Z& =X\\ W& =\rho X+\sqrt{1-{\rho }^2}Y\end{array}$$

Let $\tau =\sqrt{1-{\rho }^2}$, we get

$$\begin{array}{rl}X& =Z\\ Y& =-\frac{\rho }{\tau }Z+\frac{1}{\tau }W\end{array}$$

By the changes of variables, we have the Jacobian

$$\frac{\partial (x,y)}{\partial (z,w)}=\left(\begin{array}{cc}1& 0\\ -\frac{\rho }{\tau }& \frac{1}{\tau }\end{array}\right)$$

which has absolute determinant $\frac{1}{\tau }$, therefore

$$\begin{array}{rl}{f}_{Z,W}(z,w)& =f_{X,Y}(x,y)\cdot \left|\left|\frac{\partial (x,y)}{\partial (z,w)}\right|\right|\\ & =\frac{1}{\tau }{f}_X(x)f_Y(y)\\ & =\frac{1}{2\pi \tau }\exp \left[-\frac{1}{2}(x^2+y^2)\right]\\ & =\frac{1}{2\pi \tau }\exp \left\{-\frac{1}{2}\left[z^2+{\left(-\frac{\rho }{\tau }z+\frac{1}{\tau }w\right)}^2\right]\right\}\\ & =\frac{1}{2\pi \tau }\exp \left[-\frac{1}{2{\tau }^2}\left(z^2+w^2-2\rho zw\right)\right]\end{array}$$

Properties

Any Linear Transformation of Multi-variate Normal is Multi-variate Normal

$$\mathbf{X}\sim \mathcal{N}(\mathbf{\mu },\mathbf{\Sigma }),\mathbf{Y}=\text{A}\mathbf{X}+\mathbf{b}⟹\mathbf{Y}\sim \mathcal{N}(\text{A}\mathbf{\mu }+\mathbf{b},\text{A}\mathbf{\Sigma }{\text{A}}^{-1})$$