RCNN

Object recognition is much more complex than plain image classification: the algorithm model has to identify which objects are where in the image and draw boxes around them. Ross Girshick proposed an algorithm model that uses a convolutional neural network together with a support vector machine for object detection. Below is a brief analysis of this classic model.

Basic structure

The first problem is object localization, which the authors solve through the region recognition approach, using $selective \ search$. The second problem is the scarcity of training data; the paper proposes supervised pre-training on a large auxiliary dataset, followed by fine-tuning of specific parts on a small dataset. The experimental results also confirm that this approach is effective. The rough structure is as follows:

It mainly consists of three parts:

• Generating class-independent candidate regions;
• A convolutional network used to extract a fixed-length feature vector from each region;
• A set of class-specific linear SVMs

Module design

For selecting candidate regions there are many implementations, such as $objectness$, $selective \ search$, $category-independent \ object \ proposal$ and so on. The authors chose $selective \ search$; here it must be said that the way $R-CNN$ proposes candidate boxes is not very reasonable, because the $selective \ search$ method is simply too slow and seriously slows the model down. For feature extraction, the authors use a $CNN$ to extract a fixed-length feature from each candidate region. First, all the pixels of the tight bounding box around the candidate region are warped to the input size suitable for the $CNN$ network; before that, the authors expand the tight bounding box so that, at the warped size, the original surrounding warped image context has exactly enough pixels of the box. In the recognition stage, the score of each candidate box is obtained through the $SVM$, and then a greedy non-maximum suppression ($NMS$) is used to consolidate the results. Considering that the training data is too large, the authors also use standard hard negative mining to improve model performance.

The training process of the model is also divided into three parts:

• CNN fine-tuning
• SVM training
• Bounding box regression training

Since the training of the first two is basically the same as before, I will not go into detail here; let me mainly mention the bounding box regression training.

The input of this part is $N$ training data pairs $\{(P^i, G^i)\}_{i=1,\dots,N}$, where $P^i=(P^i_x,P^i_y,P^i_w,P^i_h)$ denotes the pixel coordinates of the center of the candidate box together with its length and width, and likewise $G=(G_x,G_y,G_w,G_h)$ corresponds to the ground-truth object bounding box. Our goal is to learn a mapping from candidate box $P$ to ground-truth box $G$. Define four variables $d_x(P)$, $d_y(P)$, $d_w(P)$, $d_h(P)$; the first two specify a scale-invariant translation of the center of the bounding box of $P$, and the last two specify the translation of the width and height of the bounding box of $P$ in log space. Then the mapping from candidate box $P$ to the predicted target box $\hat{G}$ can be written as:

$$ \begin{aligned} \hat{G}_x & = P_wd_x(P)+P_x \\ \hat{G}_y & = P_hd_y(P)+P_y \\ \hat{G}_w & = P_wexp(d_w(P)) \\ \hat{G}_h & = P_hexp(d_h(P)) \end{aligned} $$

This can be treated as a linear regression problem: each of $d_x(P)$, $d_y(P)$, $d_w(P)$, $d_h(P)$ is modeled by the feature vector $\phi(P)$ from the candidate box pooling layer of the convolutional network, that is:

$$ d_*(P)=w^T_*\phi(P) $$

where $w_*$ are the parameters of the model, which can be learned through the regularized least squares objective (ridge regression):

$$ w_*=argmax_{\hat{w}_*}\sum_i^N(t_*^i-\hat{w}_*^T\phi(P^i))^2+\lambda\vert\vert\hat{w}_*\vert\vert^2 $$

For this paper, the regression target $t_*$ is defined as:

$$ \begin{aligned} t_x & = (G_x-P_x)/P_w \\ t_y & = (G_y-P_y)/P_h \\ t_w & = log(G_w/P_w) \\ t_h & = log(G_h/P_h) \end{aligned} $$

That is, after inputting $\phi$, $(t_x,t_y,t_w,t_h)$ is used to correct the position of the candidate box.

Reference

This post refers to the following paper:

Region-based Convolutional Networks for Accurate Object Detection and Segmentation