Point clouds are sparse in distribution, irregular in structure and unordered geometrically, so applying convolution operations directly on point clouds may make the generated features deviate greatly from their original representation.
VoxelNet
This model voxelizes the point cloud and then uses 3D convolution to obtain features. Specifically:
- The point cloud space is divided equally into a voxel grid of size $v_D$, $v_H$, $v_W$, and at most $T$ points are randomly kept within each cell (extras are removed, fewer are left as they are).
- A VFE layer (voxel feature encoding layer) is applied to each voxel cell to extract voxel features, which are merged into a 4D tensor.
- 3D convolution is applied to the tensor obtained above to extract features further, and then an RPN structure is used to obtain the results
As shown below:
VFE
VFE is the most important structure in VoxelNet. For a single-layer VFE, there are mainly the following steps:
- Compute the mean coordinate of all points in the voxel and the deviation of each point from the mean to obtain the input features of the points, which are merged as the input features of VFE $\mathbf{V}_{in} = \{\hat{\mathbf{p}}_i = [x_i, y_i, z_i, r_i, x_i-v_x, y_i-v_y, z_i-v_z]^T \in \mathbb{R}^7\}_{i=1\dots t}$
- Each $\hat{\mathbf{p}}_i$ is passed through an FCN structure composed of fully connected layers, BN layers and ReLU layers to obtain the point feature $f_i \in \mathbb{R}^m$
- Element-wise max pooling is applied to the features of all points in $V_{in}$ to obtain an aggregated feature $f \in \mathbb{R}^m$. The element-wise max pooling here can be understood as
np.max(f, axis=0): an input feature of size(n_point_in_voxel, m)becomes a feature of size(1, m)after element-wise max pooling - Concatenating $f$ and $f_i$ gives the final feature of each point, $f^{out}_i = [f^T_i, f^T]^T \in \mathbb{R}^{2m}$. Merging the $f^{out}_i$ of every point in the voxel gives the final output feature of VFE, $V_{out}=\{f^{out}_i\}_{i=1\dots t}$
The concrete structure is shown below:
This per-point operation may not make full use of the computing device, so the paper also mentions an efficient implementation of VFE later on; see the original paper for details.
RPN
After the features are obtained through VFE, several structures composed of 3D convolution, BN layers and ReLU layers are used to extract features further. Finally the RPN structure below is used to obtain the final output:
SECOND
VoxelNet uses 3D convolution layers, and this convolution layer is computationally expensive, which makes the overall inference speed of the model slow. This paper proposes a sparse 3D convolution operation, and also proposes angle loss regression to address the insensitivity of VoxelNet to object orientation.
There are very detailed articles online about the sparse 3D convolution method and about SECOND; the key is to build the element index tables of the input and output. I recommend these two:
An analysis of the SECOND algorithm
An easy-to-understand explanation of the Sparse Convolution process
Data Augmentation
SECOND devotes about half a page specifically to explaining some of the data augmentation methods they use.
Sample Ground Truths from the Database
Because point cloud ground truth data is rather scarce, both the convergence speed of model training and the final performance are greatly limited. The authors augment the point cloud data with a data augmentation method similar to small object pasting.
- Use the existing data to build a database with point clouds and corresponding ground truth
- Introduce the objects in the database into the current point cloud data by concatenation
- Use collision detection when removing and concatenating objects to verify whether the current operation conforms to physical rules
This method can effectively increase the number of ground truth objects in the point cloud, and can also improve the diversity of the data.
Object Noise
Similar to VoxelNet, noise is added to the objects in the point cloud independently and randomly. But VoxelNet uses the same parameters for all points, whereas the authors randomly pick a value from $\Delta\theta \in [-\pi/2, \pi/2]$ for the random rotation, and randomly pick a value from a Gaussian distribution with mean 0 and standard deviation 1 for the random linear transformation.
Global Rotation and Scaling
The whole point cloud data and the corresponding annotation boxes are scaled and rotated, with a scaling range of $[0.95, 1.05]$ and a rotation range of $[-\pi/4, \pi/4]$.
Reference
VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection