2024

Note: VAE is not a model that only cares about reconstruction; its more important ability is image generation, so VAE is not an AE model — it has generative ability. Unlike the common Autoencoder, the variational autoencoder does not map the input to a fixed vector but to a distribution $p_\theta$ (with parameters $\theta$). The relationship between the input data $\mathbf{x}$ and the latent variable $\mathbf{z}$ is as follows: $p_\theta(\mathbf{z})$: prior probability $p_\theta(\mathbf{x}\vert\mathbf{z})$: likelihood $p_\theta(\mathbf{z}\vert\mathbf{x})$: posterior probability Assuming that after obtaining the optimal model parameters $\theta^{\star}$ we can generate samples that follow the real data distribution through the following steps: ...

VQGAN is a generative model that performs well on a variety of image generation tasks (such as unconditional image generation, image completion, conditionally guided image generation and so on), the biggest highlight being image generation at the megapixel level. The overall pipeline of the model is as follows: Overall it is similar to VQVAE, but the PixelCNN in VQVAE is replaced by a Transformer, and a PatchGAN discriminator is added during training to provide an adversarial loss. ...

Normally the pixel values of an image are discrete (between 0 and 255), so we can actually regard an image as a sequence of length $H\times W\times C$. When predicting the current pixel value, the preceding pixel values are taken as known information and participate in the prediction, and this proceeds recursively pixel by pixel, which is the autoregressive model: $$ p(x) = p(x_1)p(x_2\vert x_1)\dots p(x_n\vert x_1,x_2,\dots,x_{n-1}) $$But the process above generates one pixel at a time, so it runs slowly. VQ-VAE borrows the idea of the autoregressive model, and its most obvious characteristic is that the latent variable obtained through the encoder is a discrete variable, which compresses the image into a small feature space. The overall pipeline is roughly shown below: ...