Unbiasedness and Efficiency

Unbiasedness

An point estimator is considered unbiased if its expected value equals the true value of the parameter begin estimated. For example, if $\hat{\theta }$ is an estimator for a parameter $\theta$, it is unbiased if

$$\mathrm{E}[\hat{\theta }]=\theta$$

Common examples of unbiased estimators include the sample mean for estimating the population mean and the sample variance calculated the Bessel's correction (dividing by $n-1$ instead of $n$)

To get a unbiased estimator, we can first use the techniques in point estimation such as method of moments and MLE to get a estimation $\hat{g}(X_1,\dots ,X_n)$ of the target parameter $g(\theta )$. Then, we correct our estimation by adding a coefficient to it and solve then equation

$$\lambda \mathrm{E}\left[\hat{g}(X_1,\dots ,X_n)\right]=g(\theta )⟹\hat{\theta }={\hat{g}}_{\text{unbiased}}(X_1,\dots ,X_n)=\lambda \hat{g}(X_1,\dots ,X_n)$$

However, unbiasedness does not always guarantee that an estimator is the best choice. In some cases, biased estimators may perform better in terms of mean squared error (MSE) compared to unbiased ones. This is particularly true when biased estimators have lower variance, which can lead to more reliable estimates in practice. Mathematically, we have

$$\begin{array}{rl}\mathrm{MSE}(\hat{g},g(\theta ))& =\mathrm{E}\left[(\hat{g}-g(\theta {)}^2)\right]\\ & =\mathrm{E}\left[{\left(\hat{g}-\mathrm{E}[\hat{g}]+\mathrm{E}[\hat{g}]-g(\theta )\right)}^2\right]\\ & =\mathrm{E}\left[{\left(\hat{g}-\mathrm{E}[\hat{g}]\right)}^2\right]+2\mathrm{E}\left[\left(\hat{g}-\mathrm{E}[\hat{g}]\right)\left(\mathrm{E}[\hat{g}]-g(\theta )\right)\right]+{\left(\mathrm{E}[\hat{g}]-g(\theta )\right)}^2\\ & =\mathrm{Var}(\hat{g})+{\left(\mathrm{E}[\hat{g}]-g(\theta )\right)}^2\end{array}$$

since $2\mathrm{E}\left[\left(\hat{g}-\mathrm{E}[\hat{g}]\right)\left(\mathrm{E}[\hat{g}]-g(\theta )\right)\right]=2{\mathrm{E}}^2[\hat{g}]-2g(\theta )\mathrm{E}[\hat{\mathrm{g}}]-2{\mathrm{E}}^2[\hat{g}]+2g(\theta )\mathrm{E}[\hat{g}]=0$. In this formula, we could see that the MSE of the estimator is the sum of its variance error and the square of its bias

Efficiency

Efficiency measures how well an estimator performs relative to other estimators. An efficient estimator achieves the lowest possible variance among all unbiased estimators for a given parameter. Mathematically, if for $\forall \theta \in \Theta$ we have

$$\mathrm{Var}({\hat{g}}_1(\theta ))\le \mathrm{Var}({\hat{g}}_2(\theta ))$$

and there exist at least one $\theta \in \Theta$ such that $\mathrm{Var}({\hat{g}}_1(\theta ))<\mathrm{Var}({\hat{g}}_2(\theta ))$. Then we say ${\hat{g}}_1$ is more efficient than ${\hat{g}}_2$.