Sufficient and Complete Statistics

Sufficient Statistic

A statistic is said to be sufficient for a parameter if it captures all the information needed to estimate that parameter from the sample data.

More formally, a statistic $T(X)$ is sufficient for a parameter $\theta$ is the conditional distribution of the sample data $X$ given $T(X)$ does not depend on $\theta$. For example, in the case of a normal distribution with known variance and unknown mean, the sample mean is a sufficient statistic for the mean.

Fisher-Neyman Factorization Theorem

A statistic $T(X)$ is sufficient for $\theta$ if and only if the likelihood function can be factored as $f_{\theta }(x)=g_{\theta }(T(x))h(x)$, where $g_{\theta }$ depends only on $T(x)$ and $\theta$, and $h(x)$ does not depend on $\theta$.

Complete Statistic

A statistic is complete of no non-trivial function of that statistic has an expected value of zero for all parameter values unless the function itself is almost zero.

In simper terms, if for any measurable function $\phi$, $\mathrm{E}[\phi (T)]=0$ for all values of $\theta$, then it must be that $P(\phi (T)=0)=1$. This means that the complete statistic contains all possible information about the parameter.

Rao-Blackwell Theorem

The conditional expected value of an unbiased estimator given a sufficient statistic is another unbiased estimator that's at least as good.

That is, if $T$ is a sufficient statistic, $\hat{g}$ is an unbiased estimator of $g(\theta )$, then we have

• $\mathrm{E}[g(\theta )|T]$ is unbiased: $\mathrm{E}[\mathrm{E}[g(\theta )|T]]=g(\theta )$.
• At least as good: $\mathrm{Var}(\mathrm{E}[\hat{g}|T])\le \mathrm{Var}(\hat{g})$

Lehmann-Scheffé Theorem

The Lehmann–Scheffé theorem has an additional hypothesis that the sufficient statistic is complete, i.e. it admits no unbiased estimators of zero. It also has an additional conclusion: the estimator you get is the unique best unbiased estimator.

That is, if $T$ is a sufficient complete statistic, $\hat{g}$ is an unbiased estimator of $g(\theta )$, then we have

• $\mathrm{E}[g(\theta )|T]$ is unbiased: $\mathrm{E}[\mathrm{E}[g(\theta )|T]]=g(\theta )$.
• Best: $\mathrm{Var}(\mathrm{E}[\hat{g}|T])\le \mathrm{Var}(\hat{g})$ for any unbiased estimator $\hat{g}$, i.e. $\mathrm{E}[\hat{g}|T]$ is the UMVUE of $g(\theta )$
• Unique: $\mathrm{E}[\hat{g}|T]$ is the only UMVUE of $g(\theta )$

Exponential Distribution Family

For the exponential family, which includes distributions like the normal, Poisson, and exponential distributions, a sufficient statistic can often be expressed in terms of a function of the data. For instance, if $X_1,X_2,\dots ,X_n$ are i.i.d. r.v.s from an exponential family with PDF

$$p(x;\theta )=h(x)\psi (\theta )\exp \left(\sum _{i=1}^n{\eta }_i(\theta )T_i(x)\right)$$

where ${\eta }_i(\theta )$ are functions of $\theta$, then $T(X)=\left(T_1(X),T_2(X),\dots ,T_s(X)\right)$ is a sufficient complete statistic of $\theta$

For example, for $X_1,X_2,\dots ,X_n$ i.i.d. $\mathcal{N}(\mu ,{\sigma }^2)$, a sufficient complete statistic of $(\mu ,{\sigma }^2)$ is $\left({\sum }_{i=1}^n{X}_i,{\sum }_{i=1}^n{X}_i^2\right)$, then $(\bar{X},S^2)$ is also a sufficient complete statistic. Therefore $\mathrm{E}[\bar{X}|\bar{X}]=\bar{X}$ is the UMVUE of $\mu$, and $\mathrm{E}[S^2|S^2]=S^2$ is the UMVUE of ${\sigma }^2$.