Context Encoding for Semantic Segmentation

Idea

Improve the semantic segmentation results by effectively utilizing global context, which captures the semantic context of scenes and selectively enhances class-dependent feature maps.

Context Encoding Module

Encoding Layer

Input

Considering the input feature map with the shape of $C\times H\times W$ as a set of $C$-dimensional input features $X=\left\{x_1,\dots ,x_N\right\}$, where $N=H\times W$

Codebook

The Encoding Layer learns a codebook $D$ containing $K$ codewords (or visual centers)

$$D=\left\{d_1,d_2,\dots ,d_K\right\}$$

Each $d_k$ represents a distinct visual center in the feature space that captures a specific semantic meaning or category in the input data

Smoothing Factor

Alongside the codebook, the layer also learns a set of smoothing factors

$$S=\left\{s_1,s_2,\dots ,s_K\right\}$$

Each $s_k$ corresponds to a smoothing factor for the visual center $d_k$. These factors are used to control the influence of each visual center on the feature representation

Output

The layer outputs the residual encoder by aggregating the residuals with soft-assignment weights $e_k={\sum }_{i=1}^N{e}_{ik}$, where

$$e_{ik}=\frac{\exp (-s_k\Vert r_{ik}{\Vert }^2)}{{\sum }_{j=1}^K\exp (-s_j\Vert r_{ij}{\Vert }^2)}{r}_{ik}$$

and the residuals are given by $r_{ik}=x_i-d_k$. The final output is $e={\sum }_{k=1}^K\varphi (e_k)$, where $\varphi$ denotes Batch Normalization with ReLU activation. ($e$ is a $C$-dimensional vector)

Feature Map Attention

Use a fully connected layer to predict feature map scaling factors $\gamma =\delta (We)$, where $W$ denotes the layer weights and $\delta$ is the sigmoid function. Then the module output is given by $Y=X\otimes \gamma$.