Uniformly Minimum Variance Unbiased Estimate, UMVUE

Definition

In all the unbiased estimation of $g(\theta )$, we want to find a best estimation $\tilde{g}(\theta )$ such that it has the lowest variance (that is, most efficient) among all these unbiased estimations. We call such $\tilde{g}(\theta )$ the uniformly minimum variance unbiased estimate (UMVUE) of $g(\theta )$

To find a UMVUE of $g(\theta )$, we can apply the two methods below

Cramer-Rao Inequality

Fisher Information Matrix

Let $f(x;\mathbf{\theta })$ be the PDF of the population and $X_1,X_2,\dots ,X_n$ i.i.d. $\sim f(x;\mathbf{\theta })$. We define

$$I(\mathbf{\theta })=\mathrm{E}\left[(\nabla \log f(x;\mathbf{\theta }))(\nabla \log f(x;\mathbf{\theta }){)}^T\right]$$

as the Fisher Information Matrix (FIM).

If $\mathbf{\theta }=\theta$, the matrix becomes a number

$$I(\theta )=\mathrm{E}\left[{\left(\frac{\partial \log f(x;\theta )}{\partial \theta }\right)}^2\right]$$

Assume we can change order the gradient and integral without any impacts on the result for $f(x;\mathbf{\theta })$, that is

$$\nabla {\int }_{-\infty }^{\infty }f(x;\mathbf{\theta })\mathrm{d}x={\int }_{-\infty }^{\infty }\nabla f(x;\mathbf{\theta })\mathrm{d}x$$

and

$$\nabla {\int }_{-\infty }^{\infty }\nabla f(x;\mathbf{\theta })\mathrm{d}x={\int }_{-\infty }^{\infty }{\nabla }^{2}f(x;\mathbf{\theta })\mathrm{d}x$$

Then we will have

$$\begin{array}{rl}\mathrm{E}\left[\nabla \log f(x;\mathbf{\theta })\right]& ={\int }_x\frac{\nabla f(x;\mathbf{\theta })}{f(x;\mathbf{\theta })}p(x;\mathbf{\theta })\mathrm{d}x\\ & ={\int }_x\nabla f(x;\mathbf{\theta })\mathrm{d}x\\ & =\nabla {\int }_{x}f(x;\mathbf{\theta })\mathrm{d}x\\ & =\nabla (1)=0\end{array}$$

where we call $\nabla \log f(x;\mathbf{\theta })$ the score function of the population. Therefore, we get

$$\mathrm{Var}(\nabla \log f(x;\mathbf{\theta }))=I(\theta )-0^2=I(\theta )$$

and

$$\begin{array}{rl}\mathrm{E}\left[{\nabla }^2\log f(x;\mathbf{\theta })\right]& =\mathrm{E}\left[\frac{{\nabla }^{2}f(\cdot )f(\cdot )}{f^2(\cdot )}-{\left(\frac{\nabla f(\cdot )}{f(\cdot )}\right)}^2\right]\\ & =-\mathrm{E}\left[{\left(\frac{\nabla f(\cdot )}{f(\cdot )}\right)}^2\right]\\ & =-I(\mathbf{\theta })\end{array}$$

For $\mathbf{\theta }=\theta$, this is

$$I(\theta )=-\mathrm{E}\left[\frac{{\partial }^2}{\partial {\theta }^2}\log f(x;\theta )\right]$$

C-R Inequality

Let $\hat{\mathbf{g}}(X)$ be an estimator of any vector function of parameters $\mathbf{\theta }$ from the observed sample $X=X_1,\dots ,X_n$, and denote its expectation vector $\mathrm{E}\left[\hat{\mathbf{g}}(X)\right]$ by $\mathbf{\Gamma }(\mathbf{\theta })$. The C-R Inequality then states the covariance matrix of $\hat{\mathbf{g}}(X)$ satisfies

$$\mathrm{Cov}\left(\hat{\mathbf{g}}(X)\right)\ge \frac{1}{n}\mathbf{\gamma }(\theta )I(\mathbf{\theta }{)}^{-1}\mathbf{\gamma }(\theta {)}^T$$

where

• The matrix inequality $A\ge B$ is understood that the matrix $A-B$ is positive semidefinite
• ${\mathbf{\gamma }}^{\prime }(\theta )=\frac{\partial {\mathbf{\Gamma }}^{\prime }(\mathbf{\theta })}{\partial \mathbf{\theta }}$ is the Jacobian matrix whose $i,j$ element is given by $\frac{\partial {\Gamma }_i(\mathbf{\theta })}{\partial {\theta }_j}$

If $\hat{\mathbf{g}}(X)$ is unbiased (i.e. $\mathbf{\Gamma }(\mathbf{\theta })=\mathrm{E}[\hat{\mathbf{g}}(X)]=\mathbf{g}(\mathbf{\theta })$), then the inequality reduces to

$$\mathrm{Cov}(\mathbf{g}(X))\ge \frac{1}{n}{\mathbf{g}}^{\prime }(\mathbf{\theta })I(\mathbf{\theta }{)}^{-1}{\mathbf{g}}^{\prime }(\mathbf{\theta }{)}^T$$

If $\mathbf{\theta }=\theta$, the inequality becomes

$$\mathrm{Var}(\hat{g}(X))\ge \frac{{\gamma }^2(\theta )}{nI(\theta )}$$

and

$$\mathrm{Var}(\hat{g}(X))\ge \frac{g^{\prime }(\theta {)}^2}{nI(\theta )}$$

at the unbiased case.

Particularly, if $g(\theta )=\theta$, we would have

$$\mathrm{Var}(\hat{\theta }(X))\ge \frac{1}{nI(\theta )}$$

From the C-R inequality, we see if the variance of a unbiased estimate is its lower bound, then it is the UMVUE.