Non-local Neural Networks

Idea

Local Attention

Attention mechanism implemented through convolutional layers, which operate on small patches of the input (e.g., $3\times 3$ or $5\times 5$ kernels)

Non-local Attention

Extends the concept of attention by allowing the model to consider relationships across the entire input space, regardless of distance.

Method

See PSANet - Point-wise Spatial Attention Network for Scene Parsing