Pearson's chi-squared Test

Pearson’s chi-squared test (${\chi }^2$) is a fundamental statistical method used to analyze categorical data by evaluating whether observed differences between data sets occurred by chance.

Let $F_0(x)$ be the distribution function of a known discrete population. We have $n$ independent and identically distributed observation samples

$$X_1,X_2,\cdots ,X_n.$$

Assume $X_1\sim F(x)$. The hypothesis testing problem is:

$$H_0:F(x)=F_0(x),x\in \mathbb{R}⟷H_1:\exists x\in \mathbb{R}\text{ such that }F(x)\ne F_0(x)$$

When the sample size $n$ is large, Karl Pearson proposed the ${\chi }^2$ goodness-of-fit test in 1900. The following discussion addresses two scenarios: whether $F_0(x)$ contains unknown parameters or not.

Case 1: Probability Function Does Not Contain Unknown Parameters First, consider the case where $F_0(x)$ does not contain unknown parameters. Let the probability function of $F_0(x)$ be:

$$p_j=P(X_1=a_j),j=1,2,\cdots ,k.$$

Let $n_j={\sum }_{i=1}^n{I}_{\{X_i=a_j\}},j=1,2,\cdots ,k$. The observed frequency $n_j$ and the expected frequency $n{p}_j$ are shown in the table below.

Value	$a_1$	$a_2$	$\cdots$	$a_k$
Expected Frequency	$n{p}_1$	$n{p}_2$	$\cdots$	$n{p}_k$
Observed Frequency	$n_1$	$n_2$	$\cdots$	$n_k$

By the law of large numbers, when $H_0$ holds, $n_j/n$ converges in probability to $p_j$, $j=1,2,\cdots ,k$. Therefore, the expected frequency $n{p}_j$ is close to the observed frequency $n_j$. When $H_1$ holds, $n_j/n$ does not converge in probability to $p_j$, $j=1,2,\cdots ,k$, so the expected frequency $n{p}_j$ is far from the observed frequency $n_j$. The test statistic is:

$$T=\sum _{j=1}^k\frac{(n_j-n{p}_j{)}^2}{n{p}_j}.$$

When $H_0$ holds, it can be proven (details omitted),

$$T\stackrel{d}{\to }{\chi }_{k-1}^2.$$

This test method is called the goodness-of-fit test.

Case 2: The probability function contains several unknown parameters Let the probability function of the distribution function $F_0(x)$ be

$$P(X=a_j)=p_j({\theta }_1,{\theta }_2,\cdots ,{\theta }_r),j=1,2,\cdots ,k,$$

where ${\theta }_1,{\theta }_2,\cdots ,{\theta }_r$ are unknown, $r<k$.

When $H_0$ holds, the maximum likelihood estimate of $({\theta }_1,{\theta }_2,\cdots ,{\theta }_r)$ is $({\hat{\theta }}_1,{\hat{\theta }}_2,\cdots ,{\hat{\theta }}_r)$. According to the invariance of the maximum likelihood estimate, when $H_0$ holds, the maximum likelihood estimate of $p_j$ is

$${\hat{p}}_j={\hat{p}}_j({\hat{\theta }}_1,{\hat{\theta }}_2,\cdots ,{\hat{\theta }}_r),j=1,2,\cdots ,k.$$

Construct the test statistic

$$T=\sum _{j=1}^k\frac{(n_j-n{\hat{p}}_j{)}^2}{n{\hat{p}}_j}.$$

When $H_0$ holds, $T\stackrel{d}{\to }{\chi }_{k-r-1}^2$, where $r$ is the number of unknown parameters. The degrees of freedom can be understood as follows: there are $k$ degrees of freedom in total. To find the maximum likelihood estimates of ${\theta }_1,{\theta }_2,\cdots ,{\theta }_r$, $r$ equations are needed, i.e., there are $r$ constraints, reducing $r$ degrees of freedom. The constraint ${\sum }_{j=1}^k{\hat{p}}_j=1$ further reduces one degree of freedom, leaving $k-1-r$ degrees of freedom.

Contingence Table

Let there be two attributes: $A$ and $B$. Attribute $A$ has $k$ levels: $a_1,a_2,\cdots ,a_k$, and attribute $B$ has $m$ levels: $b_1,b_2,\cdots ,b_m$. There is a sample of $n$ observations, where the frequency of attribute $A$ having level $a_i$ and attribute $B$ having level $b_j$ is $n_{ij}$, $i=1,2,\cdots ,k,j=1,2,\cdots ,m$. The data is presented in the following contingency table:

$$\begin{array}{cccccc}A\backslash B& b_1& b_2& \cdots & b_m& \text{Row Totals}\\ a_1& n_{11}& n_{12}& \cdots & n_{1m}& n_{1\cdot }\\ a_2& n_{21}& n_{22}& \cdots & n_{2m}& n_{2\cdot }\\ ⋮& ⋮& ⋮& \ddots & ⋮& ⋮\\ a_k& n_{k1}& n_{k2}& \cdots & n_{km}& n_{k\cdot }\\ \text{Column Totals}& n_{\cdot 1}& n_{\cdot 2}& \cdots & n_{\cdot m}& n\end{array}$$

We want to examine whether attributes $A$ and $B$ are independent. This is a problem of testing for independence in a contingency table (testing if the random variables corresponding to the rows and columns are independent). The hypothesis testing problem is:

$H_0$: Attributes $A$ and $B$ are independent $\leftrightarrow H_1$: Attributes $A$ and $B$ are not independent

Let

$$\begin{array}{rl}{u}_i& =P(\text{Attribute }A\text{ has level }{a}_i),i=1,2,\dots ,k,\\ v_j& =P(\text{Attribute }B\text{ has level }{b}_j),j=1,2,\dots ,m.\end{array}$$

When $H_0$ holds,

$$P\left(\text{Attribute }A\text{ has level }{a}_i,\text{Attribute }B\text{ has level }{b}_j\right)=u_i{v}_j,i\in [k],j\in [m].$$

From the problem description, the maximum likelihood estimates of $u_i$ and $v_j$ are:

$${\hat{u}}_i=\frac{n_{i\cdot }}{n},{\hat{v}}_j=\frac{n_{\cdot j}}{n},$$

where $n_{i\cdot }$ is the sum of $n_{ij}$ over $j$ (i.e., the row sum), and $n_{\cdot j}$ is the sum of $n_{ij}$ over $i$ (i.e., the column sum).

The observed frequency of attribute $A$ having level $a_i$ and attribute $B$ having level $b_j$ is $n_{ij}$. The expected frequency when $H_0$ holds is $n_{i\cdot }{n}_{\cdot j}/n$. Therefore, the test statistic is defined as:

$$T=\sum _{i=1}^k\sum _{j=1}^m\frac{(n_{ij}-n_{i\cdot }{n}_{\cdot j}/n{)}^2}{n_{i\cdot }{n}_{\cdot j}/n}$$

When $H_0$ holds, it can be proven (details omitted here) that:

$$T\stackrel{d}{⟶}{\chi }_{(k-1)(m-1)}^2.$$

The degrees of freedom can be calculated as follows: There are a total of $km$ degrees of freedom. Estimating the maximum likelihood estimates of $u_1,u_2,\cdots ,u_k$ requires $k-1$ equations (because $u_1+u_2+\cdots +u_k=1$), which reduces $k-1$ degrees of freedom. Estimating the maximum likelihood estimates of $v_1,v_2,\cdots ,v_m$ requires $m-1$ equations, which reduces $m-1$ degrees of freedom. Furthermore, since the null hypothesis holds:

$$\sum _{i=1}^k\sum _{j=1}^m\frac{n_{i\cdot }{n}_{\cdot j}}{n^2}=1$$

This reduces another 1 degree of freedom. Thus, the remaining degrees of freedom are $km-(k-1)-(m-1)-1=(k-1)(m-1)$.