Most current object detection/segmentation models are based on Fast/Faster RCNN and FPN; the authors mainly provide a comparable framework for instance segmentation, namely Mask R-CNN.

Introduction

mask rcnn works by adding a branch on top of faster rcnn to predict segmentation masks in parallel. This branch is in fact a small FCN network applied to each ROI region.

  • A brief note on fast/faster rcnn here: fast rcnn introduced ROIPool on top of rcnn, and faster rcnn introduced RPN on top of fast rcnn.

Instance segmentation delineates object contours at the pixel level. There are two commonly used strategies:

  1. Segmenting proposals, that is, segmentation before recognition: candidate segmentation regions are delineated first, and class recognition is then performed on the regions to achieve instance segmentation. This approach is slow and has low accuracy.
  2. Semantic segmentation. This differs from instance segmentation: semantic segmentation only separates objects of the same class from each other, whereas instance segmentation must also separate objects within the same class. So this approach starts from the result of per-pixel classification and tries to split pixels of the same class into different instances, that is, it achieves instance segmentation on top of semantic segmentation.

mask rcnn is based on the instance-first strategy, and performs it simultaneously with classification and box regression.

Key points of the model

mask rcnn adopts the same two-stage structure as faster rcnn. The first stage is identical to faster rcnn, namely the RPN structure; the second stage has three parallel branches: classification, box regression and segmentation mask.

During training, the model uses a multi-task loss function for each roi:

$$ L = L_{cls}+L_{box}+L_{mask} $$

where $L_{cls}$ is the class loss, $L_{box}$ is the object box regression loss and $L_{mask}$ is the mask loss. The mask branch produces a $Km^2$-dimensional result for each RoI region, where $K$ is the number of classes and $m$ is the resolution of the RoI. The authors apply a sigmoid to every pixel and define the mask loss as the average binary cross-entropy loss. $L_{mask}$ is only computed on the ground-truth instances associated with the current RoI.

Unlike the other branches, the mask branch preserves the spatial information of the input image and achieves pixel-level correspondence, which requires the roi features to be well aligned and to preserve the previous spatial positions corresponding to pixels. Previously faster rcnn had a roipool step that quantized the roi feature maps. This quantization introduces misalignment between the roi region and the extracted features; it may not affect the classification result, but it has a large negative impact on pixel-level segmentation tasks. So the authors removed the quantization operations in roipool and proposed the ROIAlign layer. This network layer can align the input image with the extracted features. It avoids all the quantization operations in roipool and instead uses bilinear interpolation to compute the pixel values of the corresponding points, as shown below:

The proposal of roialign can be said to be a major contribution of this paper: the core operation that the earlier roipool performed on instances was coarse spatial quantization for feature extraction, whereas roialign faithfully preserves the exact spatial location, and roialign is also insensitive to max/average pooling. It largely solves the long-standing problem of detection and segmentation with large-stride features

Main structure

To demonstrate the generality of the model, the authors instantiated several mask rcnn models with different structures. The main difference lies in the backbone of the model and the detection head network.

As for the backbone. The authors mainly evaluated resnet and resnext networks with depths of 50 and 101 layers. In the paper the authors use the $network-depth-features$ naming convention to name the networks.

  • Here is an example of the $network-depth-features$ naming convention. In the original faster rcnn implementation, feature extraction is performed at the fourth-to-last stage of the backbone, called C4, and since that backbone is resnet-50, the whole backbone is denoted $ResNet-50-C4$

The authors also tried an FPN structure to extract ROI features.

As for the model detection head network. The authors chose the detection heads of faster rcnn with resnet and faster rcnn with fpn, plus a mask branch that contains only convolution layers. The paper figure is included below for details:

Implementation details

Model training stage. The hyperparameters used for training are consistent with fast/faster rcnn. The mask loss $L_{mask}$ is only defined on positive rois (with a positive to negative roi ratio of 1:3). The rpn anchors use 5 scales and 3 aspect ratios. For the convenience of later experiments, the authors trained the rpn separately.

Model inference stage. After nms, the mask branch of the model is applied only to the 100 highest-scoring detection boxes. The mxm floating-point mask output is resized back to the roi size and binarized with a threshold of 0.5. Since the mask branch is not applied to all detection boxes, mask rcnn adds only a small amount of overhead on top of faster rcnn.

For more specific details, refer back to the original paper and the related source code

Reference

This post is mainly based on the following

Mask R-CNN