Covariance and Correlation

Covariance

Definition

The covariance between r.v.s $X$ and $Y$ is

$$\mathrm{Cov}(X,Y)=\mathrm{E}\left[(X-\mathrm{E}[X])(Y-\mathrm{E}[Y])\right]$$

Multiplying this out and using linearity, we have an equivalent expression

$$\mathrm{Cov}(X,Y)=\mathrm{E}[XY]-\mathrm{E}[X]\mathrm{E}[Y]$$

Think about the definition intuitively: If $X$ and $Y$ tend to move in the same direction, then $X-\mathrm{E}[X]$ and $Y-\mathrm{E}[Y]$ will be positive on average, giving a positive covariance. If $X$ and $Y$ tend to move in opposite directions, then the covariance would to negative.

If $X$ and $Y$ are independent, then their covariance is zero. We say that r.v.s with zero covariance are uncorrelated

Warning

While independence implies a zero covariance, $X$ and $Y$ are uncorrelated does not mean they are independent.

For example, let $X\sim \mathcal{N}(0,1)$, and let $Y=X^2$. Then $\mathrm{E}[XY]=\mathrm{E}[X^3]=0$ because the odd moments of the standard Normal distribution are equal to $0$ by symmetry. Thus

$$\mathrm{Cov}(X,Y)=\mathrm{E}[XY]-\mathrm{E}[X]\mathrm{E}[Y]=0$$

Covariance is a measure of linear association, so r.v.s can be dependent in nonlinear ways and still have zero covariance.

If $X,Y$ are square integrable, the following four statements are equivalent:

1. $X,Y$ are uncorrelated
2. $\mathrm{Cov}(X,Y)$ = 1
3. $\mathrm{E}[XY]=\mathrm{E}[X]\mathrm{E}[Y]$ (need not to be independent)
4. $\mathrm{Var}(X+Y)=\mathrm{Var}(X)+\mathrm{Var}(Y)$ (need not to be independent)

Property

If $X,T,W,V$ are real-valued random variables and $a,b,c,d$ are real-valued constants we have

1. $\mathrm{Cov}(X,a)=0$
2. $\mathrm{Cov}(X,X)=\mathrm{Var}(X)$
3. $\mathrm{Cov}(X,Y)=\mathrm{Cov}(Y,X)$
4. $\mathrm{Cov}(X+a,Y+b)=\mathrm{Cov}(X,Y)$

$$\begin{array}{rl}\mathrm{Cov}(aX+bY,cW+dV)& =ac\mathrm{Cov}(X,W)+ad\mathrm{Cov}(X,V)\\ & +bc\mathrm{Cov}(Y,W)+bd\mathrm{Cov}(Y,V)\end{array}$$

6. Cauchy-Schwarz Inequality

$$\mathrm{Cov}(X,Y)\le \sqrt{\mathrm{Var}(X)}\sqrt{\mathrm{Var}(Y)}$$

this can be proved by

$$\cos ⟨X,Y⟩=\frac{\mathrm{E}[XY]}{\sqrt{\mathrm{E}[X^2]}\sqrt{\mathrm{E}[Y^2]}}\le 1$$

where we define $⟨X,Y⟩=\mathrm{E}[XY]$ as the inner product

Correlation

The correlation between r.v.s $X$ and $Y$ is

$$\mathrm{Corr}(X,Y)=\frac{\mathrm{Cov}(X,Y)}{\sqrt{\mathrm{Var}(X)\mathrm{Var}(Y)}}$$

Therefore, for any r.v.s $X$ and $Y$

$$-1\le \mathrm{Corr}(X,Y)\le 1$$

Covariance Matrix

Definition

For multi-variate r.v.s, for example, $\mathbf{X}=(X_1,\dots ,X_n)$, $\mathbf{Y}=(Y_1,\dots ,Y_n)$, we can define the covariance matrix

$$\begin{array}{rl}\mathrm{Cov}(\mathbf{X},\mathbf{Y})& =\mathrm{E}\left[(\mathbf{X}-\mathrm{E}[\mathbf{X}])(\mathbf{Y}-\mathrm{E}[\mathbf{Y}])\right]\\ & =\left(\begin{array}{ccc}\mathrm{Cov}(X_1,Y_1)& \cdots & \mathrm{Cov}(X_1,Y_n)\\ ⋮& \ddots & ⋮\\ \mathrm{Cov}(X_n,Y_1)& \cdots & \mathrm{Cov}(X_n,Y_n)\end{array}\right)\end{array}$$

In particular, if $\mathbf{Y}=\mathbf{X}$, we call $\mathrm{Var}(\mathbf{X})=\mathrm{Cov}(\mathbf{X},\mathbf{X})$ the variance matrix of $\mathbf{X}$.

Properties

$$\mathbf{Z}=\text{A}\mathbf{X}+\mathbf{Y}⟹\mathrm{E}\left[\mathbf{Z}\right]=\text{A}\mathrm{E}\left[\mathbf{X}\right]+\mathrm{E}\left[\mathbf{Y}\right]$$

$$\mathrm{Cov}(\text{A}\mathbf{X},\text{B}\mathbf{Y})=\text{A}\mathrm{Cov}(\mathbf{X},\mathbf{Y}){\text{B}}^T$$

3. $\mathrm{Var}(\mathbf{X})$ is positive definite
4. If $\text{A}\in {\mathrm{SM}}_n(\mathbb{R})$ is a symmetric matrix, we call ${\mathbf{X}}^T\text{A}\mathbf{X}$ a quadratic form of $\mathbf{X}$. And we have

$$\mathrm{E}\left[{\mathbf{X}}^T\text{A}\mathbf{X}\right]=\mathrm{E}{\left[\mathbf{X}\right]}^T\text{A}\mathrm{E}\left[\mathbf{X}\right]+\mathrm{tr}\left(\text{A}\mathrm{Var}(\mathbf{X})\right)$$