Cross Entropy Loss

Idea

The cross entropy loss is often used in classifier models in machine learning, where the model are supposed to obtain a probability distribution among the suspected categories which the input image may belong to.

For a input image $x_i$, we have the predicted class distribution obtained by a model with parameters $\theta$, $P(y\mid x_i;\theta )$; and the true class distribution, $P^{\ast }(y\mid x_i)$. The loss function should indicate the similarity between these two distributions, and a natural way to do this is by KL divergence:

$$\begin{array}{rl}{D}_{\mathrm{KL}}(P^{\ast }\parallel P)& =\sum _y{P}^{\ast }(y\mid x_i)\log \frac{P^{\ast }(y\mid x_i)}{P(y\mid x_i;\theta )}\\ & =\sum _y{P}^{\ast }(y\mid x_i)\log P^{\ast }(y\mid x_i)-\sum _y{P}^{\ast }(y\mid x_i)\log P(y\mid x_i;\theta )\end{array}$$

Note that the first term is constant, so we just have

$$\underset{\theta }{\mathrm{argmin}}{D}_{\mathrm{KL}}(P^{\ast }\parallel P)=\underset{\theta }{\mathrm{argmin}}\left[-\sum _y{P}^{\ast }(y\mid x_i)\log P(y\mid x_i;\theta )\right]$$

and we could define the cross entropy loss as

$$H(P^{\ast },P)=-\sum _y{P}^{\ast }(y\mid x_i)\log P(y\mid x_i;\theta )$$

In conclusion, the cross entropy loss is the natural loss based on KL divergence.

Form with Softmax

If we define $y_i$ as the true distribution $P^{\ast }(y\mid x_i)$, and ${\hat{y}}_i$ as the predicted distribution generalized by softmax activation function $\mathrm{softmax}(o_i)$, then the cross entropy loss is

$$\begin{array}{rl}H(y\mid \hat{y})& =-\sum _i{y}_i\log {\hat{y}}_i\\ & =-\sum _i{y}_i\log \left[\frac{\exp (o_i)}{{\sum }_j\exp (o_j)}\right]\\ & =\sum _i{y}_i\log \left[\sum _j\exp (o_j)\right]-\sum _i{y}_i{o}_i\end{array}$$

And the derivative of $H(y\mid \hat{y})$ with respect to any logit $o_i$ would be

$${\partial }_{o_i}H(y\mid \hat{y})=\frac{\exp (o_i)}{{\sum }_j\exp (o_j)}-y_i=\mathrm{softmax}(o_i)-y_i$$