LSS is an early pioneering work that used BEV for autonomous driving perception, and many of the later BEV-related algorithms are variations built on the LSS framework.
BEV perception
BEV (Bird’s Eye View) perception, as the name suggests, solves the perception problem of autonomous driving from a top-down viewpoint. Object detection, segmentation, tracking and other problems are much more straightforward in this view, and for autonomous driving scenarios the height information of objects is actually not that important; most downstream tasks such as decision making and planning also work with this top-down view, and fusing information from multiple cameras becomes much easier. That is why autonomous driving solutions under BEV are simpler and more direct.
For pure-vision autonomous driving, the key to the BEV approach is how to transform or encode the data from the forward-facing camera view into a top-down view. There are currently four main methods:
- IPM (Inverse Perspective Mapping), which inversely maps the camera perspective space back to BEV space. IPM assumes that the ground is a perfect plane with no road height variation. This assumption differs greatly from real applications, so IPM is generally used for lane detection, where the assumption is usually reasonable.
- Lift-Splat, which roughly means using a Lift operation to lift 2D image features into a 3D frustum, and then using Splat to flatten it in the BEV direction. This framework was proposed by LSS and has had a great influence on later algorithms.
- MLP (Multi-Layer Perceptron), that is, using an MLP to learn the transformation. See VPN for more. This kind of method is very deployment-friendly, but it relies on a large amount of data to learn the mapping.
- Transformer, which uses a transformer for view transformation and is not limited by the receptive field the way a CNN is. The autonomous driving solution Tesla shared earlier uses a transformer. This kind of method is somewhat difficult to train and deploy, but may achieve better results.
Lift-Splat-Shoot
LSS is an end-to-end framework that extracts features from one or more camera images and lifts them into a frustum for each corresponding camera, then splats these frustums into a BEV grid. This is roughly the lift-splat pipeline; the following shoot can be understood as a downstream task, for example path prediction in the case of LSS.
Lift
For pure vision, the depth corresponding to each pixel in the image is not certain, so LSS generates a representation at every possible depth of each pixel. Specifically, the Lift operation generates a discretized depth distribution $\alpha$ for every point in the feature map ($D$ values in total), then takes the outer product with the feature $\mathbf{c}$ of the corresponding point (of length $C$) and multiplies by the corresponding $\alpha$ value. This yields the camera frustum. The process is roughly illustrated below:
Splat
This step projects the frustum generated above into a BEV grid of a given size. In the original LSS paper, the perception range along the $x$ and $y$ axes is $-50m\sim 50m$ and along the $z$ axis is $-10m\sim 10m$; the BEV cell size is $0.5m$ along the $x$ and $y$ axes and $20m$ along the $z$ axis, so the resulting BEV grid has a size of $200\times 200\times 1$.
During projection, multiple points may fall into a single BEV cell, and LSS fuses them with sum pooling. Because the generated frustum contains many points, the model is not very efficient, so LSS optimizes the sum pooling by using a cumsum trick to improve efficiency.
For the details of LSS I recommend reading the code open-sourced by NVIDIA; it is not complicated, only 5 or 6 Python files. I briefly organized the algorithm flow, roughly as follows:
There are also some good code walkthroughs online:
Reference
Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D