Image segmentation is pixel-level classification of an image, and mainly includes semantic segmentation and instance segmentation. Semantic segmentation only requires segmentation at the pixel level, while instance segmentation additionally requires separating different individuals on top of semantic segmentation. This post briefly introduces several basic semantic segmentation networks.

FCN

FCN (Full Convolution Network) is a classic image segmentation network whose main structure is an encoder-decoder, where the encoder extracts image feature information and the decoder interprets the extracted features. FCN replaces the fully connected layers in the network with convolution layers, so that 2D feature maps are obtained; these are then upsampled through deconvolution (transposed convolution), and finally a softmax layer produces the mask for each class. It can also handle input images of different sizes. The rough structure is shown below.

The structure figure above contains 3 FCN models, namely FCN-8s, FCN-16s and FCN-32s.

FCN-32s segments the input image directly by 32x downsampling followed by 32x upsampling. FCN-16s performs 2x upsampling, adds the result element-wise (Eltwise) to the 16x downsampled feature map of the previous layer, and then performs 16x upsampling for segmentation. FCN-8s performs 2x upsampling after the previous Eltwise operation, applies Eltwise with the 8x downsampled feature map, and then performs 8x upsampling for segmentation.

The paper’s comparison shows that FCN-8s performs better than the other two models, that is, fusing feature maps from multiple layers helps improve segmentation accuracy.

SegNet

From the FCN structure above we can see that in the decoder stage the model uses only a single deconvolution layer for upsampling, which is very coarse. SegNet also has an encoder-decoder structure, but the two parts are symmetric: the encoder follows the Vgg-16 network for feature extraction, while the decoder uses unpooling to upsample the feature maps and then convolves the upsampled images, with the aim of refining the geometric shape of objects and making up for the detail lost when the pooling layers in the encoder shrink the objects. The upsampling also makes use of pooling indices, that is, the weight indices saved during the pooling operations in the encoder, so that during upsampling the data is placed directly at the index positions and then passed through convolution layers for learning. The structure is as follows.

DeconvNet

This network is similar to SegNet in that it also uses a symmetric encoder-decoder structure and also uses pooling indices for upsampling, but unlike SegNet the model uses fully connected layers between the encoder and the decoder. To obtain instance-level segmentation results, the trained model can be applied to the proposals of an image and the results finally combined into the final segmentation, which also solves the size problem of the original FCN network and gives more detailed object information. The rough structure of the model is as follows.

UNet

The structure of UNet is also a symmetric encoder-decoder. The structure is shown first:

From the structure figure we can see that each upsampling step (this model uses deconvolution for upsampling) is followed by channel concatenation with the corresponding part of the encoder, and before concatenation the feature map of the corresponding encoder part must be cropped; the cropping keeps the central part of the image and cuts away the border so that the image size matches the corresponding decoder part. In the end a feature map with 2 channels is obtained, and to further obtain a probability map a softmax is applied to the feature map.

Summary

For deep learning based semantic segmentation, a general framework is roughly as follows:

  • Front end: feature extraction with a fully convolutional network
    • Downsampling + upsampling: Convlution + Deconvlution/Resize
    • Multi-scale feature fusion: element-wise addition of features, feature concatenation
    • Obtaining a pixel-level segement map: determining the class of each pixel
  • Back end: using CRF (conditional random field) / MRF (Markov random field) to refine the output of the front end into an optimized segmentation map

Reference

Fully Convolutional Networks for Semantic Segmentation

SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation

Learning Deconvolution Network for Semantic Segmentation

U-Net: Convolutional Networks for Biomedical Image Segmentation