Object detection has produced many algorithms because of the complexity of combining the localization and classification branches. These methods improve algorithm performance from different angles, but are hard to measure in a unified way. This paper uses multiple attention mechanisms to unify different object detection heads: attention over the feature level, over the spatial position and across channels, corresponding respectively to scale awareness, spatial awareness and task awareness. This method can noticeably improve model performance without increasing the computational cost.
Given a set of feature maps extracted from $L$ different layers of a feature pyramid (that is, the backbone), $F_{in} = \lbrace F_i\rbrace^L_{i=1}$, the feature maps in the set are upsampled or downsampled to an intermediate size to obtain a feature tensor $F\in R^{L\times H\times W\times C}$, where $L$ denotes the number of extracted layers and $H$, $W$, $C$ denote the height, width and number of channels. Let $S=H\times W$ and reshape the features to $F\in R^{L\times S\times C}$. These three dimensions correspond to three effects:
- Differences in object scale correspond to features at different scales;
- Different sets of transformations for different object shapes correspond to different spatial positions of the features;
- The variety of object representations and tasks corresponds to different channel features
Learning an attention function directly over all dimensions would be very expensive, so the paper converts the attention function into a sequence of attentions over the 3 corresponding dimensions, each of which only attends to its current dimension:
$$ W(F) = \pi_C(\pi_S(\pi_L(F)\cdot F)\cdot F)\cdot F $$Scale-aware Attention
First, attention over scale is introduced to dynamically fuse features from different scales based on semantic importance:
$$ \pi_L(F)\cdot F = \sigma\left(f\left(\frac{1}{SC}\sum_{S,C}F\right)\right)\cdot F $$$f(\cdot)$ is a linear function, approximated here with a $1\times 1$ convolution layer. $\sigma(\cdot)$ is the hard-sigmoid function, as follows:
$$ \sigma(x) = \max (0,\min (1,\frac{x+1}{2})) $$Spatial-aware Attention
Next, spatial-aware attention is used to improve the model’s discriminative ability at different spatial positions. Because the $S$ dimension is large, the paper decouples it: deformable convolution is first used to sparsify the attention learning, and then features are integrated across scales:
$$ \pi_S(F)\cdot F = \frac{1}{L}\sum^L_{l=1}\sum^K_{k=1}w_{l,k}\cdot F(l;p_k+\Delta p_k;c)\cdot \Delta m_k $$where $K$ is the number of sparse sampling positions and the remaining parameters are similar to those in deformable convolution.
Task-aware Attention
To promote joint learning and the generalization of object representations, the paper also designs a task-aware attention. It can dynamically switch feature channels on and off to assist different tasks:
$$ \pi_C(F)\cdot F = \max(\alpha^1(F)\cdot F_c+\beta^1(F),\alpha^2(F)\cdot F_c+\beta^2(F)) $$where $[\alpha^1,\alpha^2,\beta^1,\beta^2]^T = \theta(\cdot)$ are hyperparameters that control the activation threshold, and the implementation of $\theta(\cdot)$ is similar to dynamic ReLU.
Structure
The attention mechanisms above are used in sequence and can also be stacked several times. The structure figure of DynamicHead is roughly as follows:
Any type of backbone can be used to extract the feature pyramid and further scale it to a unified size to build the feature tensor, which is then used as the input of DynamicHead and passed through several DyHead modules containing scale-aware, spatial-aware and task-aware attention, stacked in series. The output of DyHead can be used for different tasks, such as classification, center/box regression and so on.
one stage: Single-stage detectors obtain object positions by densely sampling the feature maps, which simplifies the design of the object detector. The integration of DynamicHead with a single-stage detector is shown below:
two stage: Two-stage detectors extract intermediate representations from the feature pyramid through the RPN layer and the ROI pooling layer. To couple this characteristic, the paper uses scale-aware and spatial-aware attention before ROI pooling, and replaces the original fully connected layers with task-aware attention, as follows:
Reference
Dynamic Head: Unifying Object Detection Heads with Attentions