Point Estimation

Point estimation involves providing a single value, called a point estimate, as the best guess or approximation of an unknown population parameter.

Method of Moments

Estimating population parameters by equating sample moments to their corresponding population moments.

For i.i.d. samples $X_1,\dots ,X_n$ from a population with PDF $f(x\mid {\theta }_1,\dots ,{\theta }_k)$, where ${\theta }_1,\dots ,{\theta }_k$ are $k$ unknown parameters. We can

1. Calculate the first $k$ sample moments

$$\begin{array}{rl}{m}_1& =\frac{1}{n-1}\sum _{i=1}^n\left(X_i-\bar{X}\right)\\ m_2& =\frac{1}{n-1}\sum _{i=1}^n{\left(X_i-\bar{X}\right)}^2\\ & ⋮\\ m_k& =\frac{1}{n-1}\sum _{i=1}^n{\left(X_i-\bar{X}\right)}^k\end{array}$$

2. Express the first $k$ population moments as functions of the parameters

$$\begin{array}{rl}\mathrm{E}\left[X-\mathrm{E}[X]\right]& =g_1({\theta }_1,\dots ,{\theta }_k)\\ \mathrm{E}\left[(X-\mathrm{E}[X]{)}^2\right]& =g_2({\theta }_1,\dots ,{\theta }_k)\\ & ⋮\\ \mathrm{E}\left[(X-\mathrm{E}[X]{)}^k\right]& =g_k({\theta }_1,\dots ,{\theta }_k)\end{array}$$

3. Estimate the population moments via the sample moments (Law of Large Numbers)

$$\begin{array}{rl}{m}_1& =g_1({\theta }_1,\dots ,{\theta }_k)\\ m_2& =g_2({\theta }_1,\dots ,{\theta }_k)\\ & ⋮\\ m_k& =g_k({\theta }_1,\dots ,{\theta }_k)\end{array}$$

4. Solve the equations to obtain ${\hat{\theta }}_1,\dots ,{\hat{\theta }}_k$

For example, if $X_1,\dots ,X_n$ i.i.d. $\mathrm{Unif}(a,b)$, then

$$\mathrm{E}[X]=\frac{a+b}{2},\mathrm{Var}(X)=\frac{(b-a{)}^2}{12}$$

Can we could solve the estimation of $a,b$

$$a=\mathrm{E}[X]-\sqrt{3\mathrm{Var}(\mathrm{X})},b=\mathrm{E}[X]+\sqrt{3\mathrm{Var}(X)}$$

Therefore

$$\hat{a}=\mu -\sqrt{3s^2},\hat{b}=\mu +\sqrt{3s^2}$$

Maximum Likelihood Estimation

See Maximum Likelihood Estimation (MLE)