Maximum Likelihood Estimation (MLE)

If we model a set of observations as a random sample from an unknown joint probability distribution which is express in terms of s set of parameters. The goal of maximum likelihood estimation is to determine the parameters for which the observed data have the highest joint probability.

We write the parameters governing the joint distribution as a vector $\theta =[{\theta }_1,{\theta }_2,\dots ,{\theta }_k{]}^T$ so that this distribution falls within a parametric family $\left\{f(\cdot ;\theta )|\theta \in \Theta \right\}$, where $\Theta$ is called the parameter space, a finite-dimensional subset of Euclidean space. Evaluating the joint density at the observed data sample $\mathbf{y}=(y_1,y_2,\dots ,y_n)$ gives a real-valued function

$${\mathcal{L}}_n(\theta )={\mathcal{L}}_n(\theta ;\mathbf{y})=f_n(\mathbf{y};\theta )$$

which is called the likelihood function. For i.i.d. random variables, $f_n(\mathbf{y};\theta )$ will be the product of univariate (that is, only one variable) PDF:

$$f_n(\mathbf{y};\theta )=\prod _{k=1}^n{f}_k(y_k;\theta )$$

The goal of maximum likelihood estimation is to find the values of the model parameters that maximize the likelihood function over the parameter space, that is

$$\hat{\theta }=\underset{\theta \in \Theta }{\mathrm{argmax}}{\mathcal{L}}_n(\theta ;\mathbf{y})$$

Tip

Sometimes we use the log likelihood function, that is $l(\theta )=\ln \mathcal{L}(\theta )={\sum }_{k=1}^n\ln f_k(y_k;\theta )$

Invariant Property

$$\hat{\theta }=\underset{\theta \in \Theta }{\mathrm{argmax}}{\mathcal{L}}_n(\theta ;\mathbf{y})⟹g(\hat{\theta })=\underset{g(\theta )\in g(\Theta )}{\mathrm{argmax}}{\mathcal{L}}_n(g(\theta );\mathbf{y})$$