Deep Learning

Note: VAE is not a model that only cares about reconstruction; its more important ability is image generation, so VAE is not an AE model — it has generative ability. Unlike the common Autoencoder, the variational autoencoder does not map the input to a fixed vector but to a distribution $p_\theta$ (with parameters $\theta$). The relationship between the input data $\mathbf{x}$ and the latent variable $\mathbf{z}$ is as follows: $p_\theta(\mathbf{z})$: prior probability $p_\theta(\mathbf{x}\vert\mathbf{z})$: likelihood $p_\theta(\mathbf{z}\vert\mathbf{x})$: posterior probability Assuming that after obtaining the optimal model parameters $\theta^{\star}$ we can generate samples that follow the real data distribution through the following steps: ...

VQGAN is a generative model that performs well on a variety of image generation tasks (such as unconditional image generation, image completion, conditionally guided image generation and so on), the biggest highlight being image generation at the megapixel level. The overall pipeline of the model is as follows: Overall it is similar to VQVAE, but the PixelCNN in VQVAE is replaced by a Transformer, and a PatchGAN discriminator is added during training to provide an adversarial loss. ...

Normally the pixel values of an image are discrete (between 0 and 255), so we can actually regard an image as a sequence of length $H\times W\times C$. When predicting the current pixel value, the preceding pixel values are taken as known information and participate in the prediction, and this proceeds recursively pixel by pixel, which is the autoregressive model: $$ p(x) = p(x_1)p(x_2\vert x_1)\dots p(x_n\vert x_1,x_2,\dots,x_{n-1}) $$But the process above generates one pixel at a time, so it runs slowly. VQ-VAE borrows the idea of the autoregressive model, and its most obvious characteristic is that the latent variable obtained through the encoder is a discrete variable, which compresses the image into a small feature space. The overall pipeline is roughly shown below: ...

LSS is an early pioneering work that used BEV for autonomous driving perception, and many of the later BEV-related algorithms are variations built on the LSS framework. BEV perception BEV (Bird’s Eye View) perception, as the name suggests, solves the perception problem of autonomous driving from a top-down viewpoint. Object detection, segmentation, tracking and other problems are much more straightforward in this view, and for autonomous driving scenarios the height information of objects is actually not that important; most downstream tasks such as decision making and planning also work with this top-down view, and fusing information from multiple cameras becomes much easier. That is why autonomous driving solutions under BEV are simpler and more direct. ...