Consistency and Asymptotic Normality

Consistency

Definition

Consistency in statistics refers to an estimator's fundamental property where estimates converge in probability to the true parameter values as the sample size increase indefinitely.

In other words, if for all $\epsilon >0,\theta \in \Theta$, we have

$$\underset{n\to \infty }{\lim }P\left(\left|\hat{g}-g(\theta )\right|\ge \epsilon \right)=0$$

Then we call $\hat{g}$ is a consistent estimator of $g(\theta )$. This type of consistency is also called weak consistency.

For strong consistency, we require the estimator to converge almost surely to the true parameter as the sample size approaches infinity.

Conditions

Method of moments estimators are usually consistent because they are based on sample moments that converge to population moments as the sample size increases.

MLE estimators are consistent if

1. The log likelihood function is differentiable
2. The likelihood function is identifiable, that is
   • Different parameter values must correspond to different probability distributions + the parameter space is compact
   • Or the Kullback–Leibler Divergence between two distributions with different parameters be strictly positive

Note

These conditions are weaker than the condition to convergence in first order norm

Asymptotic Normality

Definition

Asymptotic Normality refers to the property of an estimator whereby, as the sample size $n$ approaches infinity, the distribution of the estimator converges to a normal distribution. This means that for sufficiently large samples, the distribution of the estimator can be approximated by a normal distribution, regardless of the original distribution of the data from which it was derived.

The asymptotic normality of a estimator can been seen as a generalization of central limit theorem (CLT), where the estimator, in particular, is the sample mean.

An MLE with the true parameter ${\theta }_0$ has asymptotic normality if

1. For every $\epsilon >0$, $\theta \in ({\theta }_0-\epsilon ,\theta +\epsilon )$, $\frac{\partial \ell (\theta )}{\partial \theta }$, $\frac{{\partial }^2\ell (\theta )}{\partial {\theta }^2}$ and $\frac{{\partial }^3\ell (\theta )}{\partial {\theta }^3}$ exist.
2. For every $\epsilon >0$, $\theta \in ({\theta }_0-\epsilon ,\theta +\epsilon )$, $\frac{{\partial }^3\ell (\theta )}{\partial {\theta }^3}\le H(x)$, where $H(x)$ is integrable and $\mathrm{E}[H(X)]<\infty$.
3. The order of the partial derivative and the integral of $\theta$ can be exchanged.

Then we will have

$$\sqrt{n}\left(\hat{\theta }-{\theta }_0\right)\stackrel{d}{\to }\mathcal{N}\left(0,\frac{1}{I({\theta }_0)}\right)$$

where $I({\theta }_0)$ is the Fisher Information. From this formula we can see the asymptotic variance of $\hat{\theta }$ is $\frac{1}{nI({\theta }_0)}$ (if $\hat{\theta }$ is an MLE), which happens to be the lower bound of C-R inequality.

Delta Method

The Delta Method is a statistical technique used to derive the asymptotic distribution of a function of an estimator that is asymptotically normal.

If we have an estimator $T_n$ that is asymptotically normal, i.e.

$$\sqrt{n}\left(T_n-\mu \right)\stackrel{d}{\to }\mathcal{N}(0,{\sigma }^2)$$

We can utilize a first-order Taylor series expansion around the point

$$g(T_n)\approx g(\mu )+g^{\prime }(\mu )(T_n-\mu )$$

Then

• The asymptotic mean is $\mathrm{E}[g(T_n)]\approx g(\mu )$
• The asymptotic variance is $\mathrm{Var}(g(T_n))\approx (g^{\prime }(\mu ){)}^2\cdot {\sigma }^2$. Therefore

$$g(T_n)\stackrel{d}{\to }\mathcal{N}(g(\mu ),(g^{\prime }(\mu ){)}^2{\sigma }^2)$$

That is

$$\sqrt{n}\left[g(T_n)-g(\mu )\right]\stackrel{d}{\to }\mathcal{N}(0,(g^{\prime }(\mu ){)}^2{\sigma }^2)$$