For the classic diffusion model DDPM, during the reverse generation inference we need to restore random Gaussian noise $\mathbf{x}_T$ step by step into a meaningful image $\mathbf{x}_0$, requiring $T$ inference steps in total, so generation is slow. For this reason, DDIM (Denoising Diffusion Implicit Models) uses a smaller number of sampling steps to speed up inference, and the process of generating an image from random noise does not introduce additional noise, that is, the reverse process is deterministic. DDIM has the same optimization objective as DDPM, but does not require the forward process to be a Markov chain.
Prerequisites
The optimization objective $L_t$ of DDPM and the sampling process $q(\mathbf{x}_{t-1}\vert\mathbf{x}_t)$ (approximated by $p_\theta(\mathbf{x}_{t-1} \vert \mathbf{x}_t)$) are as follows:
$$ \begin{aligned} L_t &= \mathbb{E}_{t \sim [1, T], \mathbf{x}_0, \boldsymbol{\epsilon}_t} \Big[\|\boldsymbol{\epsilon}_t - \boldsymbol{\epsilon}_\theta(\sqrt{\bar{\alpha}_t}\mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t}\boldsymbol{\epsilon}_t, t)\|^2 \Big] \\ p_\theta(\mathbf{x}_{t-1} \vert \mathbf{x}_t) &= \mathcal{N}(\mathbf{x}_{t-1}; \frac{1}{\sqrt{\alpha_t}} \Big( \mathbf{x}_t - \frac{1 - \alpha_t}{\sqrt{1 - \bar{\alpha}_t}} \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t) \Big), \frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\beta_t) \\ \end{aligned} $$From these two equations we can see that the loss function of DDPM depends only on $q(\mathbf{x}_t\vert\mathbf{x}_0)$, while the sampling process depends only on $q(\mathbf{x}_{t-1}\vert\mathbf{x}_t)$. DDIM removes the $q(\mathbf{x}_{t}\vert\mathbf{x}_{t-1})$ used in the DDPM derivations, which means DDIM does not define the concrete operation of the forward process.
Reverse process
In DDPM, we use Bayes’ theorem to turn the reverse process into the forward process:
$$ q(\mathbf{x}_{t-1}\vert\mathbf{x}_t,\mathbf{x}_0) = \frac{q(\mathbf{x}_t\vert\mathbf{x}_{t-1},\mathbf{x}_0)q(\mathbf{x}_{t-1}\vert\mathbf{x}_0)}{q(\mathbf{x}_t\vert\mathbf{x}_0)} $$After rearranging, integrating both sides over $\mathbf{x}_t$ eliminates $q(\mathbf{x}_t\vert\mathbf{x}_0)$, giving:
$$ \begin{aligned} \int q(\mathbf{\mathbf{x}_{t-1}\vert\mathbf{x}_t,\mathbf{x}_0})q(\mathbf{x}_t\vert\mathbf{x}_0)d\mathbf{x}_t = q(\mathbf{x}_{t-1}\vert\mathbf{x}_0) \end{aligned} \tag{1} $$The equation above is the marginal distribution condition that the diffusion model must satisfy.
Since $q(\mathbf{x}_{t-1}\vert\mathbf{x}_t,\mathbf{x}_0)$ is a Gaussian distribution, we simply assume the following form:
$$ \begin{aligned} q(\mathbf{x}_{t-1}\vert\mathbf{x}_t,\mathbf{x}_0) = \mathcal{N}(\mathbf{x}_{t-1}; \kappa_t\mathbf{x}_t+\lambda_t\mathbf{x}_0,\sigma^2_t\mathbf{I}) \end{aligned} \tag{2} $$Because the optimization objective of DDIM is the same as that of DDPM, we can reuse the DDPM training process, that is, $q(\mathbf{x}_{t-1}\vert\mathbf{x}_0)$ and $q(\mathbf{x}_t\vert\mathbf{x}_0)$. Combining these two quantities with (1) and (2) gives:
$$ \begin{aligned} \sqrt{\overline\alpha_{t-1}} &= \kappa_t\sqrt{\overline\alpha_t} + \lambda_t \\ 1-\overline\alpha_t &= \kappa_t^2(1-\overline\alpha_t)+\sigma^2_t \end{aligned} $$Treating $\sigma_t$ as a parameter, we can solve:
$$ \begin{aligned} \kappa_t &= \frac{\sqrt{1-\overline\alpha_t-\sigma^2_t}}{1-\overline\alpha_t} \\ \lambda_t &= \sqrt{\overline\alpha_{t-1}} - \frac{\sqrt{1-\overline\alpha_t-\sigma^2_t}\sqrt{\overline\alpha_t}}{\sqrt{1-\overline\alpha_t}} \end{aligned} $$Substituting into (2) gives:
$$ \begin{aligned} &q(\mathbf{x}_{t-1}\vert\mathbf{x}_t,\mathbf{x}_0) \\ =& \mathcal{N}(\mathbf{x}_{t-1};\sqrt{\overline\alpha_{t-1}}\mathbf{x}_0+\sqrt{1-\overline\alpha_t-\sigma^2_t}\frac{\mathbf{x}_t-\sqrt{\overline\alpha_t}\mathbf{x}_0}{\sqrt{1-\overline\alpha_t}}, \sigma^2_t\mathbf{I}) \end{aligned} \tag{3} $$For a more detailed derivation and explanation of equation (3), see this blog; the original paper also gives another derivation method.
Since we know $q(\mathbf{x}_t\vert\mathbf{x}_0)$, that is, the transformation between $\mathbf{x}_t$ and $\mathbf{x}_0$, the reparameterization trick gives the transformation from $\mathbf{x}_{t}$ to $\mathbf{x}_{t-1}$:
$$ \mathbf{x}_{t-1} = \sqrt{\overline\alpha_{t-1}}\frac{\mathbf{x}_t-\sqrt{1-\overline\alpha_t}\epsilon_\theta(\mathbf{x}_t,t)}{\sqrt{\overline\alpha_t}} + \sqrt{1-\overline\alpha_{t-1}-\sigma^2_t}\cdot\epsilon_\theta(\mathbf{x}_t,t) + \sigma_t\epsilon_t $$As in DDPM, $\epsilon_\theta(\mathbf{x}_t,t)$ is the noise predicted by the neural network.
The authors of DDIM define $\sigma_t$ as follows:
$$ \begin{aligned} \sigma_t &= \eta\cdot\sqrt{\tilde{\beta}_t} \\ &= \eta\cdot\sqrt{\frac{1-\overline\alpha_{t-1}}{1-\overline\alpha_t}}\sqrt{1-\alpha_t} \end{aligned} \tag{4} $$When $\eta$ is 1, the formula above gives $\sigma^2_t = \tilde\beta_t$, that is, the reverse process is consistent with that of DDPM. When $\eta=0$, the reverse process has no random noise (the $\sigma_t\epsilon_t$ term is 0), so once the initial random noise $\mathbf{x}_T$ is fixed, the reverse process is deterministic; the model in this case is DDIM.
Overall, the training process of DDIM is the same as that of DDPM, so we can take a trained DDPM model and attach the DDIM reverse process for inference.
Accelerated sampling
DDIM still needs to sample step by step to obtain the generated image, but its sampling sequence $\tau=[\tau_1,\tau_2,\dots,\tau_{len(\tau)}]$ is an arbitrary subsequence of the DDPM sampling sequence $[1,2,\dots,T]$. If we use a model trained with DDPM sampling over $T$ steps, DDIM lets us reduce the $T$ steps to $len(\tau)$ steps at inference time, thereby accelerating inference.
Reference
Denoising Diffusion Implicit Models
Talking about generative diffusion models (4): DDIM = DDPM from a higher viewpoint