[{"content":"For the classic diffusion model DDPM, during the reverse generation inference we need to restore random Gaussian noise $\\mathbf{x}_T$ step by step into a meaningful image $\\mathbf{x}_0$, requiring $T$ inference steps in total, so generation is slow. For this reason, DDIM (Denoising Diffusion Implicit Models) uses a smaller number of sampling steps to speed up inference, and the process of generating an image from random noise does not introduce additional noise, that is, the reverse process is deterministic. DDIM has the same optimization objective as DDPM, but does not require the forward process to be a Markov chain.\nPrerequisites The optimization objective $L_t$ of DDPM and the sampling process $q(\\mathbf{x}_{t-1}\\vert\\mathbf{x}_t)$ (approximated by $p_\\theta(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t)$) are as follows:\n$$ \\begin{aligned} L_t \u0026= \\mathbb{E}_{t \\sim [1, T], \\mathbf{x}_0, \\boldsymbol{\\epsilon}_t} \\Big[\\|\\boldsymbol{\\epsilon}_t - \\boldsymbol{\\epsilon}_\\theta(\\sqrt{\\bar{\\alpha}_t}\\mathbf{x}_0 + \\sqrt{1 - \\bar{\\alpha}_t}\\boldsymbol{\\epsilon}_t, t)\\|^2 \\Big] \\\\ p_\\theta(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t) \u0026= \\mathcal{N}(\\mathbf{x}_{t-1}; \\frac{1}{\\sqrt{\\alpha_t}} \\Big( \\mathbf{x}_t - \\frac{1 - \\alpha_t}{\\sqrt{1 - \\bar{\\alpha}_t}} \\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t) \\Big), \\frac{1-\\bar\\alpha_{t-1}}{1-\\bar\\alpha_t}\\beta_t) \\\\ \\end{aligned} $$From these two equations we can see that the loss function of DDPM depends only on $q(\\mathbf{x}_t\\vert\\mathbf{x}_0)$, while the sampling process depends only on $q(\\mathbf{x}_{t-1}\\vert\\mathbf{x}_t)$. DDIM removes the $q(\\mathbf{x}_{t}\\vert\\mathbf{x}_{t-1})$ used in the DDPM derivations, which means DDIM does not define the concrete operation of the forward process.\nReverse process In DDPM, we use Bayes\u0026rsquo; theorem to turn the reverse process into the forward process:\n$$ q(\\mathbf{x}_{t-1}\\vert\\mathbf{x}_t,\\mathbf{x}_0) = \\frac{q(\\mathbf{x}_t\\vert\\mathbf{x}_{t-1},\\mathbf{x}_0)q(\\mathbf{x}_{t-1}\\vert\\mathbf{x}_0)}{q(\\mathbf{x}_t\\vert\\mathbf{x}_0)} $$After rearranging, integrating both sides over $\\mathbf{x}_t$ eliminates $q(\\mathbf{x}_t\\vert\\mathbf{x}_0)$, giving:\n$$ \\begin{aligned} \\int q(\\mathbf{\\mathbf{x}_{t-1}\\vert\\mathbf{x}_t,\\mathbf{x}_0})q(\\mathbf{x}_t\\vert\\mathbf{x}_0)d\\mathbf{x}_t = q(\\mathbf{x}_{t-1}\\vert\\mathbf{x}_0) \\end{aligned} \\tag{1} $$The equation above is the marginal distribution condition that the diffusion model must satisfy.\nSince $q(\\mathbf{x}_{t-1}\\vert\\mathbf{x}_t,\\mathbf{x}_0)$ is a Gaussian distribution, we simply assume the following form:\n$$ \\begin{aligned} q(\\mathbf{x}_{t-1}\\vert\\mathbf{x}_t,\\mathbf{x}_0) = \\mathcal{N}(\\mathbf{x}_{t-1}; \\kappa_t\\mathbf{x}_t+\\lambda_t\\mathbf{x}_0,\\sigma^2_t\\mathbf{I}) \\end{aligned} \\tag{2} $$Because the optimization objective of DDIM is the same as that of DDPM, we can reuse the DDPM training process, that is, $q(\\mathbf{x}_{t-1}\\vert\\mathbf{x}_0)$ and $q(\\mathbf{x}_t\\vert\\mathbf{x}_0)$. Combining these two quantities with (1) and (2) gives:\n$$ \\begin{aligned} \\sqrt{\\overline\\alpha_{t-1}} \u0026= \\kappa_t\\sqrt{\\overline\\alpha_t} + \\lambda_t \\\\ 1-\\overline\\alpha_t \u0026= \\kappa_t^2(1-\\overline\\alpha_t)+\\sigma^2_t \\end{aligned} $$Treating $\\sigma_t$ as a parameter, we can solve:\n$$ \\begin{aligned} \\kappa_t \u0026= \\frac{\\sqrt{1-\\overline\\alpha_t-\\sigma^2_t}}{1-\\overline\\alpha_t} \\\\ \\lambda_t \u0026= \\sqrt{\\overline\\alpha_{t-1}} - \\frac{\\sqrt{1-\\overline\\alpha_t-\\sigma^2_t}\\sqrt{\\overline\\alpha_t}}{\\sqrt{1-\\overline\\alpha_t}} \\end{aligned} $$Substituting into (2) gives:\n$$ \\begin{aligned} \u0026q(\\mathbf{x}_{t-1}\\vert\\mathbf{x}_t,\\mathbf{x}_0) \\\\ =\u0026 \\mathcal{N}(\\mathbf{x}_{t-1};\\sqrt{\\overline\\alpha_{t-1}}\\mathbf{x}_0+\\sqrt{1-\\overline\\alpha_t-\\sigma^2_t}\\frac{\\mathbf{x}_t-\\sqrt{\\overline\\alpha_t}\\mathbf{x}_0}{\\sqrt{1-\\overline\\alpha_t}}, \\sigma^2_t\\mathbf{I}) \\end{aligned} \\tag{3} $$For a more detailed derivation and explanation of equation (3), see this blog; the original paper also gives another derivation method.\nSince we know $q(\\mathbf{x}_t\\vert\\mathbf{x}_0)$, that is, the transformation between $\\mathbf{x}_t$ and $\\mathbf{x}_0$, the reparameterization trick gives the transformation from $\\mathbf{x}_{t}$ to $\\mathbf{x}_{t-1}$:\n$$ \\mathbf{x}_{t-1} = \\sqrt{\\overline\\alpha_{t-1}}\\frac{\\mathbf{x}_t-\\sqrt{1-\\overline\\alpha_t}\\epsilon_\\theta(\\mathbf{x}_t,t)}{\\sqrt{\\overline\\alpha_t}} + \\sqrt{1-\\overline\\alpha_{t-1}-\\sigma^2_t}\\cdot\\epsilon_\\theta(\\mathbf{x}_t,t) + \\sigma_t\\epsilon_t $$As in DDPM, $\\epsilon_\\theta(\\mathbf{x}_t,t)$ is the noise predicted by the neural network.\nThe authors of DDIM define $\\sigma_t$ as follows:\n$$ \\begin{aligned} \\sigma_t \u0026= \\eta\\cdot\\sqrt{\\tilde{\\beta}_t} \\\\ \u0026= \\eta\\cdot\\sqrt{\\frac{1-\\overline\\alpha_{t-1}}{1-\\overline\\alpha_t}}\\sqrt{1-\\alpha_t} \\end{aligned} \\tag{4} $$When $\\eta$ is 1, the formula above gives $\\sigma^2_t = \\tilde\\beta_t$, that is, the reverse process is consistent with that of DDPM. When $\\eta=0$, the reverse process has no random noise (the $\\sigma_t\\epsilon_t$ term is 0), so once the initial random noise $\\mathbf{x}_T$ is fixed, the reverse process is deterministic; the model in this case is DDIM.\nOverall, the training process of DDIM is the same as that of DDPM, so we can take a trained DDPM model and attach the DDIM reverse process for inference.\nAccelerated sampling DDIM still needs to sample step by step to obtain the generated image, but its sampling sequence $\\tau=[\\tau_1,\\tau_2,\\dots,\\tau_{len(\\tau)}]$ is an arbitrary subsequence of the DDPM sampling sequence $[1,2,\\dots,T]$. If we use a model trained with DDPM sampling over $T$ steps, DDIM lets us reduce the $T$ steps to $len(\\tau)$ steps at inference time, thereby accelerating inference.\nReference Denoising Diffusion Implicit Models\nDDIM of diffusion models\nTalking about generative diffusion models (4): DDIM = DDPM from a higher viewpoint\n","permalink":"https://simplestory.github.io/en/posts/20240817_d9d0d187039bec56d5fc1024c51a882c/","summary":"\u003cp\u003eFor the classic diffusion model DDPM, during the reverse generation inference we need to restore random Gaussian noise $\\mathbf{x}_T$ step by step into a meaningful image $\\mathbf{x}_0$, requiring $T$ inference steps in total, so generation is slow. For this reason, DDIM (Denoising Diffusion Implicit Models) uses a smaller number of sampling steps to speed up inference, and the process of generating an image from random noise does not introduce additional noise, that is, the reverse process is deterministic. DDIM has the same optimization objective as DDPM, but does not require the forward process to be a Markov chain.\u003c/p\u003e","title":"DDIM"},{"content":"This is an image generation model of the diffusion model type. To put it simply, by continuously adding Gaussian noise to the original image $\\mathbf{x}_0$ ($T$ rounds in total), $\\mathbf{x}_0$ can finally be turned into pure Gaussian noise $\\mathbf{x}_T$, and the model is responsible for restoring $\\mathbf{x}_T$ to $\\mathbf{x}_0$. This is illustrated below:\nHere the process from $\\mathbf{x}_0$ to $\\mathbf{x}_T$ is called the forward process, and the process from $\\mathbf{x}_T$ to $\\mathbf{x}_0$ is the reverse or inference process.\nMost of this post is based on What are Diffusion Models.\nForward process Given the input image $\\mathbf{x}_0 \\sim q(\\mathbf{x})$, $T$ rounds of Gaussian noise are added to obtain $\\mathbf{x}_1$, $\\mathbf{x}_2$, $\\dots$, $\\mathbf{x}_T$. In this process $\\mathbf{x}_t$ depends only on $\\mathbf{x}_{t-1}$, so it can be regarded as a Markov process. We have:\n$$ \\begin{aligned} q(\\mathbf{x}_t \\vert \\mathbf{x}_{t-1}) \u0026= \\mathcal{N}(\\mathbf{x}_t; \\sqrt{1 - \\beta_t} \\mathbf{x}_{t-1}, \\beta_t\\mathbf{I}) \\\\ q(\\mathbf{x}_{1:T} \\vert \\mathbf{x}_0) \u0026= \\prod^T_{t=1} q(\\mathbf{x}_t \\vert \\mathbf{x}_{t-1}) \\end{aligned} $$where $\\{\\beta_t\\in (0, 1)\\}^T_{t=1}$ are given hyperparameters denoting the variances of a series of Gaussian distributions. In practice $\\beta_t$ increases as $t$ grows.\nLet $\\alpha_t = 1 - \\beta_t$ and $\\overline{\\alpha}_t = \\Pi_{i=1}^t\\alpha_i$. Using the reparameterization trick we have:\n$$ \\begin{aligned} \\mathbf{x}_t \u0026= \\sqrt{\\alpha_t}\\mathbf{x}_{t-1} + \\sqrt{1 - \\alpha_t}\\boldsymbol{\\epsilon}_{t-1} \\\\ \u0026= \\sqrt{\\alpha_t \\alpha_{t-1}} \\mathbf{x}_{t-2} + \\sqrt{1 - \\alpha_t \\alpha_{t-1}} \\bar{\\boldsymbol{\\epsilon}}_{t-2} \u0026 \\\\ \u0026= \\dots \\\\ \u0026= \\sqrt{\\bar{\\alpha}_t}\\mathbf{x}_0 + \\sqrt{1 - \\bar{\\alpha}_t}\\boldsymbol{\\epsilon} \\\\ q(\\mathbf{x}_t \\vert \\mathbf{x}_0) \u0026= \\mathcal{N}(\\mathbf{x}_t; \\sqrt{\\bar{\\alpha}_t} \\mathbf{x}_0, (1 - \\bar{\\alpha}_t)\\mathbf{I}) \\end{aligned} $$where $\\boldsymbol{\\epsilon}_{t-1}, \\boldsymbol{\\epsilon}_{t-2}, \\dots \\sim \\mathcal{N}(\\mathbf{0}, \\mathbf{I})$ and $\\bar\\epsilon_{t-2}$ denotes the combination of two Gaussian distributions, that is, combining them through the additivity of independent Gaussian distributions:\n$$ \\mathcal{N}(\\mathbf{0}, \\sigma_2^2\\mathbf{I}) + \\mathcal{N}(\\mathbf{0}, \\sigma_2^2\\mathbf{I}) \\sim \\mathcal{N}(\\mathbf{0}, (\\sigma^2_1+\\sigma^2_2)\\mathbf{I}) $$In summary we obtain:\n$$ \\begin{aligned} \\mathbf{x}_t = \\sqrt{\\overline{\\alpha}_t}\\mathbf{x}_0 + \\sqrt{1-\\overline{\\alpha}_t}\\boldsymbol\\epsilon \\end{aligned} \\tag{1} $$that is, $\\mathbf{x}_t$ at any time step can be expressed in terms of $\\mathbf{x}_0$ and $\\beta$. At the same time, from (1) we can see that as $t$ grows, $\\beta_t$ also grows and $\\overline\\alpha_t$ gets closer to 0, so $\\mathbf{x}_t$ gets closer to the standard Gaussian distribution $\\mathcal{N}(\\mathbf{0}, \\mathbf{I})$.\nReverse process If we could obtain the reversed distribution $q(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t)$, we could restore $\\mathbf{x}_0$ from the completely standard Gaussian distribution $\\mathbf{x}_t\\sim\\mathcal{N}(\\mathbf{0}, \\mathbf{I})$. Note that when $\\beta_t$ is small enough, $q(\\mathbf{x}_{t-1}\\vert\\mathbf{x}_t)$ is still a Gaussian distribution, but usually we cannot easily infer $q(\\mathbf{x}_{t-1}\\vert\\mathbf{x}_t)$, so we use a deep learning model (parameterized by $\\theta$, with the current mainstream structure being Unet + Attention) to predict a distribution $p_\\theta$ that approximates $q(\\mathbf{x}_{t-1}\\vert\\mathbf{x}_t)$.\n$$ \\begin{aligned} p_\\theta(\\mathbf{x}_{0:T}) \u0026= p(\\mathbf{x}_T) \\prod^T_{t=1} p_\\theta(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t) \\\\ p_\\theta(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t) \u0026= \\mathcal{N}(\\mathbf{x}_{t-1}; \\boldsymbol{\\mu}_\\theta(\\mathbf{x}_t, t), \\boldsymbol{\\Sigma}_\\theta(\\mathbf{x}_t, t)) \\end{aligned} $$From the formulas above we can see that, to obtain the distribution $p_\\theta$, $\\boldsymbol{\\mu}_\\theta(\\mathbf{x}_t, t)$ and $\\boldsymbol{\\Sigma}_\\theta(\\mathbf{x}_t, t)$ are needed.\nFor the reversed distribution $q(\\mathbf{x}_{t-1}\\vert\\mathbf{x}_t)$, adding $\\mathbf{x}_0$ as a known condition gives:\n$$ q(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t, \\mathbf{x}_0) = \\mathcal{N}(\\mathbf{x}_{t-1}; {\\tilde{\\boldsymbol{\\mu}}}(\\mathbf{x}_t, \\mathbf{x}_0), {\\tilde{\\beta}_t} \\mathbf{I}) $$The variance is written as $\\tilde{\\beta_t}$ because the DDPM paper simplifies things by setting $\\boldsymbol{\\Sigma}_\\theta(\\mathbf{x}_t, t)$ directly to $\\tilde{\\beta_t}$.\nBy Bayes\u0026rsquo; formula we obtain:\n$$ q(\\mathbf{x}_{t-1}\\vert\\mathbf{x}_t,\\mathbf{x}_0) = \\frac{q(\\mathbf{x}_t\\vert\\mathbf{x}_{t-1},\\mathbf{x}_0)q(\\mathbf{x}_{t-1}\\vert\\mathbf{x}_0)}{q(\\mathbf{x}_t\\vert\\mathbf{x}_0)} $$that is, the reverse process is entirely converted into the forward process. Substituting the density function of the Gaussian distribution gives:\n$$ \\begin{aligned} \u0026 q(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t, \\mathbf{x}_0) \\\\ = \u0026 \\frac{q(\\mathbf{x}_t\\vert\\mathbf{x}_{t-1},\\mathbf{x}_0)q(\\mathbf{x}_{t-1}\\vert\\mathbf{x}_0)}{q(\\mathbf{x}_t\\vert\\mathbf{x}_0)} \\\\ \\propto \u0026 \\exp\\Big( -\\frac{1}{2} \\big( (\\frac{\\alpha_t}{\\beta_t} + \\frac{1}{1 - \\bar{\\alpha}_{t-1}}) \\mathbf{x}_{t-1}^2 - (\\frac{2\\sqrt{\\alpha_t}}{\\beta_t} \\mathbf{x}_t + \\frac{2\\sqrt{\\bar{\\alpha}_{t-1}}}{1 - \\bar{\\alpha}_{t-1}} \\mathbf{x}_0) \\mathbf{x}_{t-1} + C(\\mathbf{x}_t, \\mathbf{x}_0) \\big) \\Big) \\end{aligned} $$The density function of an ordinary Gaussian distribution $\\mathcal{N}(\\mu, \\sigma^2)$ is as follows:\n$$ \\begin{aligned} f(x) \u0026= \\frac{1}{\\sqrt{2\\pi}\\sigma}\\exp(-\\frac{(x-\\mu)^2}{2\\sigma^2}) \\\\ \u0026=\\frac{1}{\\sqrt{2\\pi}\\sigma}\\exp(-\\frac{1}{2}(\\frac{1}{\\sigma^2}x^2-\\frac{2\\mu}{\\sigma^2}x+\\frac{\\mu^2}{\\sigma^2})) \\end{aligned} $$Comparing with the formula above, we have:\n$$ \\begin{aligned} \\frac{1}{\\sigma^2} \u0026= \\frac{1}{\\tilde{\\beta}_t} = \\frac{\\alpha_t}{\\beta_t}+\\frac{1}{1-\\bar\\alpha_{t-1}} \\\\ \u0026\\Rightarrow \\tilde{\\beta}_t = \\frac{1-\\bar\\alpha_{t-1}}{1-\\bar\\alpha_t}\\cdot\\beta_t \\\\ \\frac{2\\mu}{\\sigma^2} \u0026= \\frac{2\\sqrt{\\alpha_t}}{\\beta_t} \\mathbf{x}_t + \\frac{2\\sqrt{\\bar{\\alpha}_{t-1}}}{1 - \\bar{\\alpha}_{t-1}} \\mathbf{x}_0 \\\\ \u0026\\Rightarrow \\tilde\\mu_t(\\mathbf{x}_t, \\mathbf{x}_0) = \\frac{\\sqrt{\\alpha_t}(1-\\bar\\alpha_{t-1})}{1-\\bar\\alpha_t}\\mathbf{x}_t+\\frac{\\sqrt{\\bar\\alpha_{t-1}}\\beta_t}{1-\\bar\\alpha_t}\\mathbf{x}_0 \\end{aligned} $$So the variance $\\tilde{\\beta}_t$ is a constant, while the mean $\\tilde\\mu_t$ depends on $\\mathbf{x}_t$ and $\\mathbf{x}_0$. From (1) we obtain $\\mathbf{x}_0 = \\frac{1}{\\sqrt{\\bar{\\alpha}_t}}(\\mathbf{x}_t - \\sqrt{1 - \\bar{\\alpha}_t}\\boldsymbol{\\epsilon}_t)$, and substituting into the formula above gives:\n$$ \\begin{aligned} \\tilde{\\boldsymbol{\\mu}}_t =\\frac{1}{\\sqrt{\\alpha_t}} \\Big( \\mathbf{x}_t - \\frac{1 - \\alpha_t}{\\sqrt{1 - \\bar{\\alpha}_t}} \\boldsymbol{\\epsilon}_t \\Big) \\end{aligned} \\tag{2} $$where the Gaussian distribution $\\boldsymbol{\\epsilon}_t$ in the model is the predicted noise distribution, $\\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t)$, so we have:\n$$ \\begin{aligned} \\boldsymbol{\\mu}_\\theta(\\mathbf{x}_t, t) =\\frac{1}{\\sqrt{\\alpha_t}} \\Big( \\mathbf{x}_t - \\frac{1 - \\alpha_t}{\\sqrt{1 - \\bar{\\alpha}_t}} \\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t) \\Big) \\end{aligned} \\tag{3} $$Putting the reverse process above together, we obtain the model inference steps as follows:\nThe model predicts the Gaussian noise $\\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t)$ from $\\mathbf{x}_t$ and $t$, and then $\\boldsymbol{\\mu}_\\theta(\\mathbf{x}_t, t)$ is obtained from (3) Compute the variance $\\Sigma_\\theta(\\mathbf{x}_t, t)$. DDPM replaces $\\Sigma_\\theta(\\mathbf{x}_t, t)$ with $\\tilde\\beta_t$ and considers $\\beta_t$ and $\\tilde\\beta_t$ to be approximately equal $\\mathbf{x}_{t-1}$ is obtained from $p_\\theta(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t) = \\mathcal{N}(\\mathbf{x}_{t-1}; \\boldsymbol{\\mu}_\\theta(\\mathbf{x}_t, t), \\boldsymbol{\\Sigma}_\\theta(\\mathbf{x}_t, t))$ using the reparameterization method The DDPM inference procedure is as follows:\nTraining process As mentioned above, we use a model to predict a distribution $p_\\theta(\\mathbf{x}_{t-1}\\vert\\mathbf{x}_t)$ that approximates the reversed distribution $q(\\mathbf{x}_{t-1}\\vert\\mathbf{x}_t)$. There are generally two ways to approach this problem: one is to optimize the maximum likelihood of the negative log, $-\\log p_\\theta(\\mathbf{x}_0)$; the other is to optimize the cross entropy between the predicted distribution and the true distribution, $\\mathbb{E}_{q(\\mathbf{x}_0)}[-\\log p_\\theta(\\mathbf{x}_0)]$. Both are rather difficult to optimize directly, so we consider optimizing their variational lower bound (VLB) instead.\nSince the KL divergence is non-negative, we have:\n$$ -\\log p_\\theta(\\mathbf{x}_0) \\leq -\\log p_\\theta(\\mathbf{x}_0) + D_{KL}(q(\\mathbf{x}_{1:T}\\vert\\mathbf{x}_0)\\vert\\vert p_\\theta(\\mathbf{x}_{1:T}\\vert\\mathbf{x}_0)) $$Expanding and simplifying the KL divergence, and taking the expectation $\\mathbb{E}_{q(\\mathbf{x}_0)}$ on both sides, gives:\n$$ \\begin{aligned} \\mathcal{L}_{VLB} \u0026= \\mathbb{E}_{q(\\mathbf{x}_0)}\\Big( -\\log p_\\theta(\\mathbf{x}_0) + \\mathbb{E}_{q(\\mathbf{x}_{1:T}\\vert\\mathbf{x}_0)}\\Big[\\log\\frac{q(\\mathbf{x}_{1:T}\\vert\\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_{0:T})} + \\log p_\\theta (\\mathbf{x}_0)\\Big]\\Big)\\\\ \u0026= \\mathbb{E}_{q(\\mathbf{x}_0)}\\Big( \\mathbb{E}_{q(\\mathbf{x}_{1:T}\\vert\\mathbf{x}_0)}\\Big[\\log\\frac{q(\\mathbf{x}_{1:T}\\vert\\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_{0:T})}\\Big]\\Big) \\\\ \u0026= \\mathbb{E}_{q(\\mathbf{x}_{0:T})}\\Big[\\log\\frac{q(\\mathbf{x}_{1:T}\\vert\\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_{0:T})}\\Big] \\\\ \u0026\\geq \\mathbb{E}_{q(\\mathbf{x}_0)}[-\\log p_\\theta(\\mathbf{x}_0)] \\end{aligned} $$The second equality above uses Fubini\u0026rsquo;s theorem, and Jensen\u0026rsquo;s inequality gives the same result:\n$$ \\begin{aligned} \u0026 \\mathbb{E}_{q(\\mathbf{x}_0)}[-\\log p_\\theta(\\mathbf{x}_0)] \\\\ =\u0026 -\\mathbb{E}_{q(\\mathbf{x}_0)}\\Big[\\log\\Big(p_\\theta(\\mathbf{x}_0)\\cdot\\int p_\\theta(\\mathbf{x}_{1:T})d\\mathbf{x}_{1:T}\\Big)\\Big] \\\\ =\u0026 -\\mathbb{E}_{q(\\mathbf{x}_0)}\\Big[\\log\\Big(\\int p_\\theta(\\mathbf{x}_{0:T})d\\mathbf{x}_{1:T}\\Big)\\Big] \\\\ =\u0026 -\\mathbb{E}_{q(\\mathbf{x}_0)}\\Big[\\log\\Big(\\int q(\\mathbf{x}_{1:T}\\vert\\mathbf{x}_0)\\frac{p_\\theta(\\mathbf{x}_{0:T})}{q(\\mathbf{x}_{1:T}\\vert\\mathbf{x}_0)}d\\mathbf{x}_{1:T}\\Big)\\Big] \\\\ =\u0026 -\\mathbb{E}_{q(\\mathbf{x}_{0:T})}\\Big[\\log\\Big(\\mathbb{E}_{q(\\mathbf{x}_{1:T}\\vert\\mathbf{x}_0)}\\frac{p_\\theta(\\mathbf{x}_{0:T})}{q(\\mathbf{x}_{1:T}\\vert\\mathbf{x}_0)}\\Big)\\Big] \\\\ \\leq \u0026 -\\mathbb{E}_{q(\\mathbf{x}_{0:T})}\\Big[\\log\\frac{p_\\theta(\\mathbf{x}_{0:T})}{q(\\mathbf{x}_{1:T}\\vert\\mathbf{x}_0)}\\Big] \\\\ =\u0026 \\mathbb{E}_{q(\\mathbf{x}_{0:T})}\\Big[\\log\\frac{q(\\mathbf{x}_{1:T}\\vert\\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_{0:T})}\\Big] = \\mathcal{L}_{VLB} \\end{aligned} $$Note that $\\log$ is a concave function, so $-\\log$ is convex.\nThus we finally obtain:\n$$ \\mathcal{L}_{VLB} = \\mathbb{E}_{q(\\mathbf{x}_{0:T})}\\Big[\\log\\frac{q(\\mathbf{x}_{1:T}\\vert\\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_{0:T})}\\Big] \\geq \\mathbb{E}_{q(\\mathbf{x}_0)}[-\\log p_\\theta(\\mathbf{x}_0)] $$So minimizing $\\mathbf{L}_{VLB}$ minimizes the target loss function $\\mathbb{E}_{q(\\mathbf{x}_0)}[-\\log p_\\theta(\\mathbf{x}_0)]$.\nDeriving $\\mathbf{L}_{VLB}$ gives:\n$$ \\begin{aligned} \\mathcal{L}_{VLB} \u0026= \\mathbb{E}_{q(\\mathbf{x}_{0:T})} \\Big[ \\log\\frac{q(\\mathbf{x}_{1:T}\\vert\\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_{0:T})} \\Big] \\\\ \u0026= \\mathbb{E}_q \\Big[ \\log\\frac{\\prod_{t=1}^T q(\\mathbf{x}_t\\vert\\mathbf{x}_{t-1})}{ p_\\theta(\\mathbf{x}_T) \\prod_{t=1}^T p_\\theta(\\mathbf{x}_{t-1} \\vert\\mathbf{x}_t) } \\Big] \\\\ \u0026= \\mathbb{E}_q \\Big[ -\\log p_\\theta(\\mathbf{x}_T) + \\sum_{t=1}^T \\log \\frac{q(\\mathbf{x}_t\\vert\\mathbf{x}_{t-1})}{p_\\theta(\\mathbf{x}_{t-1} \\vert\\mathbf{x}_t)} \\Big] \\\\ \u0026= \\mathbb{E}_q \\Big[ -\\log p_\\theta(\\mathbf{x}_T) + \\sum_{t=2}^T \\log \\frac{q(\\mathbf{x}_t\\vert\\mathbf{x}_{t-1})}{p_\\theta(\\mathbf{x}_{t-1} \\vert\\mathbf{x}_t)} + \\log\\frac{q(\\mathbf{x}_1 \\vert \\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_0 \\vert \\mathbf{x}_1)} \\Big] \\\\ \u0026= \\mathbb{E}_q \\Big[ -\\log p_\\theta(\\mathbf{x}_T) + \\sum_{t=2}^T \\log \\Big( \\frac{q(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t, \\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_{t-1} \\vert\\mathbf{x}_t)}\\cdot \\frac{q(\\mathbf{x}_t \\vert \\mathbf{x}_0)}{q(\\mathbf{x}_{t-1}\\vert\\mathbf{x}_0)} \\Big) + \\log \\frac{q(\\mathbf{x}_1 \\vert \\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_0 \\vert \\mathbf{x}_1)} \\Big] \\\\ \u0026= \\mathbb{E}_q \\Big[ -\\log p_\\theta(\\mathbf{x}_T) + \\sum_{t=2}^T \\log \\frac{q(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t, \\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_{t-1} \\vert\\mathbf{x}_t)} + \\sum_{t=2}^T \\log \\frac{q(\\mathbf{x}_t \\vert \\mathbf{x}_0)}{q(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_0)} + \\log\\frac{q(\\mathbf{x}_1 \\vert \\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_0 \\vert \\mathbf{x}_1)} \\Big] \\\\ \u0026= \\mathbb{E}_q \\Big[ -\\log p_\\theta(\\mathbf{x}_T) + \\sum_{t=2}^T \\log \\frac{q(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t, \\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_{t-1} \\vert\\mathbf{x}_t)} + \\log\\frac{q(\\mathbf{x}_T \\vert \\mathbf{x}_0)}{q(\\mathbf{x}_1 \\vert \\mathbf{x}_0)} + \\log \\frac{q(\\mathbf{x}_1 \\vert \\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_0 \\vert \\mathbf{x}_1)} \\Big]\\\\ \u0026= \\mathbb{E}_q \\Big[ \\log\\frac{q(\\mathbf{x}_T \\vert \\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_T)} + \\sum_{t=2}^T \\log \\frac{q(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t, \\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_{t-1} \\vert\\mathbf{x}_t)} - \\log p_\\theta(\\mathbf{x}_0 \\vert \\mathbf{x}_1) \\Big] \\\\ \u0026= \\mathbb{E}_q [\\underbrace{D_\\text{KL}(q(\\mathbf{x}_T \\vert \\mathbf{x}_0) \\parallel p_\\theta(\\mathbf{x}_T))}_{L_T} + \\sum_{t=2}^T \\underbrace{D_\\text{KL}(q(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t, \\mathbf{x}_0) \\parallel p_\\theta(\\mathbf{x}_{t-1} \\vert\\mathbf{x}_t))}_{L_{t-1}} \\underbrace{- \\log p_\\theta(\\mathbf{x}_0 \\vert \\mathbf{x}_1)}_{L_0} ] \\end{aligned} $$that is, $\\mathcal{L}_{VLB}$ is turned into a sum of several entropies or KL divergences. Finally it is expressed as:\n$$ \\begin{aligned} \\mathcal{L}_{VLB} \u0026= L_T + L_{T-1} + \\dots + L_0 \\\\ \\text{where } L_T \u0026= D_\\text{KL}(q(\\mathbf{x}_T \\vert \\mathbf{x}_0) \\parallel p_\\theta(\\mathbf{x}_T)) \\\\ L_t \u0026= D_\\text{KL}(q(\\mathbf{x}_t \\vert \\mathbf{x}_{t+1}, \\mathbf{x}_0) \\parallel p_\\theta(\\mathbf{x}_t \\vert\\mathbf{x}_{t+1})) \\text{ for }1 \\leq t \\leq T-1 \\\\ L_0 \u0026= - \\log p_\\theta(\\mathbf{x}_0 \\vert \\mathbf{x}_1) \\end{aligned} $$The forward process $q$ has no learnable parameters, and $\\mathbf{x}_T$ is pure Gaussian noise, so $L_T$ can be treated as a constant. $L_t$ can be regarded as closing the distance between two Gaussian distributions; from solving the KL divergence of multivariate Gaussian distributions we have:\n$$ \\mathcal{L}_{VLB} = \\mathbb{E}_q\\Big[\\frac{1}{2{\\vert\\vert\\Sigma_\\theta(\\mathbf{x}_t, t)\\vert\\vert}^2_2}{\\vert\\vert\\tilde\\mu_t(\\mathbf{x}_t, \\mathbf{x}_0)-\\mu_\\theta(\\mathbf{x}_t, t)\\vert\\vert}^2\\Big] + C $$where $C$ is unrelated to the model parameters. Substituting $(1), (2), (3)$ into the formula above gives:\n$$ \\begin{aligned} L_t \u0026= \\mathbb{E}_{\\mathbf{x}_0, \\boldsymbol{\\epsilon}} \\Big[\\frac{1}{2 \\| \\boldsymbol{\\Sigma}_\\theta(\\mathbf{x}_t, t) \\|^2_2} \\| \\tilde{\\boldsymbol{\\mu}}_t(\\mathbf{x}_t, \\mathbf{x}_0) - \\boldsymbol{\\mu}_\\theta(\\mathbf{x}_t, t) \\|^2 \\Big] \\\\ \u0026= \\mathbb{E}_{\\mathbf{x}_0, \\boldsymbol{\\epsilon}} \\Big[\\frac{1}{2 \\|\\boldsymbol{\\Sigma}_\\theta \\|^2_2} \\| \\frac{1}{\\sqrt{\\alpha_t} \\Big( \\mathbf{x}_t - \\frac{1 - \\alpha_t}{\\sqrt{1 - \\bar{\\alpha}_t}} \\boldsymbol{\\epsilon}_t \\Big)} - \\frac{1}{\\sqrt{\\alpha_t}} \\Big( \\mathbf{x}_t - \\frac{1 - \\alpha_t}{\\sqrt{1 - \\bar{\\alpha}_t} \\boldsymbol{\\boldsymbol{\\epsilon}}_\\theta(\\mathbf{x}_t, t) \\Big)} \\|^2 \\Big] \\\\ \u0026= \\mathbb{E}_{\\mathbf{x}_0, \\boldsymbol{\\epsilon}} \\Big[\\frac{ (1 - \\alpha_t)^2 }{2 \\alpha_t (1 - \\bar{\\alpha}_t) \\| \\boldsymbol{\\Sigma}_\\theta \\|^2_2} \\|\\boldsymbol{\\epsilon}_t - \\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t)\\|^2 \\Big] \\\\ \u0026= \\mathbb{E}_{\\mathbf{x}_0, \\boldsymbol{\\epsilon}} \\Big[\\frac{ (1 - \\alpha_t)^2 }{2 \\alpha_t (1 - \\bar{\\alpha}_t) \\| \\boldsymbol{\\Sigma}_\\theta \\|^2_2} \\|\\boldsymbol{\\epsilon}_t - \\boldsymbol{\\epsilon}_\\theta(\\sqrt{\\bar{\\alpha}_t}\\mathbf{x}_0 + \\sqrt{1 - \\bar{\\alpha}_t}\\boldsymbol{\\epsilon}_t, t)\\|^2 \\Big] \\end{aligned} $$From the formula above we can see that the focus of model training is optimizing the MAE between the two Gaussian noises $\\boldsymbol\\epsilon$ and $\\boldsymbol\\epsilon_\\theta$.\nFor $L_0$, DDPM designs a discretized piecewise integral accumulation for $p_\\theta(\\mathbf{x}_0\\vert\\mathbf{x}_1)$, and also simplifies $\\mathcal{L}_{VLB}$ (ignoring the coefficients), giving:\n$$ \\begin{aligned} L_t^\\text{simple} \u0026= \\mathbb{E}_{t \\sim [1, T], \\mathbf{x}_0, \\boldsymbol{\\epsilon}_t} \\Big[\\|\\boldsymbol{\\epsilon}_t - \\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t)\\|^2 \\Big] \\\\ \u0026= \\mathbb{E}_{t \\sim [1, T], \\mathbf{x}_0, \\boldsymbol{\\epsilon}_t} \\Big[\\|\\boldsymbol{\\epsilon}_t - \\boldsymbol{\\epsilon}_\\theta(\\sqrt{\\bar{\\alpha}_t}\\mathbf{x}_0 + \\sqrt{1 - \\bar{\\alpha}_t}\\boldsymbol{\\epsilon}_t, t)\\|^2 \\Big] \\\\ \\Rightarrow \u0026 L_{simple} = L^{simple}_t + C \\end{aligned} $$As mentioned earlier, DDPM does not compute $\\Sigma_\\theta(\\mathbf{x}_t, t)$ but uses a non-trainable $\\beta_t$ or $\\tilde\\beta_t$. So the training procedure of DDPM is roughly as follows:\nObtain $\\mathbf{x}_0$, and randomly sample a $t$ from $1\\dots T$ Randomly sample a noise $\\epsilon_t \\sim \\mathcal{N}(\\mathbf{0}, \\mathbf{I})$ from the standard Gaussian distribution Minimize $\\|\\boldsymbol{\\epsilon}_t - \\boldsymbol{\\epsilon}_\\theta(\\sqrt{\\bar{\\alpha}_t}\\mathbf{x}_0 + \\sqrt{1 - \\bar{\\alpha}_t}\\boldsymbol{\\epsilon}_t, t) \\|$ The DDPM training flow chart is as follows:\nAppendix A. Reparameterization Usually the process of randomly sampling a sample from some distribution cannot be backpropagated through. Reparameterization uses an independent random variable to direct the randomness out.\nSuppose $\\mathbf{z}$ follows a Gaussian distribution with expectation $\\mu_\\theta$ and variance $\\sigma_\\theta^2$. Reparameterized sampling can be written in the following form:\n$$ \\mathbf{z} = \\mu_\\theta + \\sigma_\\theta \\odot \\epsilon, \\quad \\epsilon\\sim\\mathcal{N}(\\mathbf{0}, \\mathbf{I}) $$where $\\odot$ denotes element-wise multiplication. The formula above moves the randomness of $\\mathbf{z}$ onto $\\epsilon$ while still satisfying $\\mathbf{z}\\sim \\mathcal{N}(\\mathbf{z}; \\mu_\\theta, \\sigma^2_\\theta\\mathbf{I})$, and this process is differentiable.\nB. Jensen\u0026rsquo;s inequality Let $x$ be a random variable and $\\phi$ be a convex function; then the following holds:\n$$ \\phi(E(x)) \\leq E(\\phi(x)) $$If $x$ is a discrete random variable, this can be written as: $\\phi(\\frac{1}{n}\\sum x)\\leq\\frac{1}{n}\\sum\\phi(x)$\nC. KL divergence The KL divergence of the distributions $p(x)$ and $q(x)$ is denoted $D_{KL}(p\\|q)$, and:\n$$ D_{KL}(p\\|q) = E_{p(x)}[\\log\\frac{p(x)}{q(x)}] $$Jensen\u0026rsquo;s inequality can be used to show that the KL divergence is always greater than 0:\n$$ \\begin{aligned} D_{KL}(p\\|q) \u0026= E_{p(x)}[\\log\\frac{p(x)}{q(x)}] \\\\ \u0026= E_{p(x)}[-\\log\\frac{q(x)}{p(x)}] \\\\ \u0026\\ge-\\log\\Big( E_{p(x)}[\\frac{q(x)}{p(x)}] \\Big) \\\\ \u0026=-\\log\\Big(\\int p(x)\\frac{q(x)}{p(x)} dx\\Big) \\\\ \u0026=-\\log\\Big(\\int q(x)dx\\Big) \\\\ \u0026=-\\log(1) = 0 \\end{aligned} $$Reference Denoising Diffusion Probabilistic Models\nWhat are Diffusion Models\nUnderstanding Diffusion Model from the ground up\nExplaining Diffusion Model DDPM in one article\nDDPM of diffusion models\n","permalink":"https://simplestory.github.io/en/posts/20240817_60a1868ea6b6f7b05126264f9d212a9c/","summary":"\u003cp\u003eThis is an image generation model of the \u003cstrong\u003ediffusion model\u003c/strong\u003e type. To put it simply, by continuously adding Gaussian noise to the original image $\\mathbf{x}_0$ ($T$ rounds in total), $\\mathbf{x}_0$ can finally be turned into pure Gaussian noise $\\mathbf{x}_T$, and the model is responsible for restoring $\\mathbf{x}_T$ to $\\mathbf{x}_0$. This is illustrated below:\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://simplestory-blog-img.oss-cn-guangzhou.aliyuncs.com/in_posts/20240817/ddpm_instruction.png\" title=\"ddpm instruction\"\u003e\u003c/p\u003e\n\u003cp\u003eHere the process from $\\mathbf{x}_0$ to $\\mathbf{x}_T$ is called the forward process, and the process from $\\mathbf{x}_T$ to $\\mathbf{x}_0$ is the reverse or inference process.\u003c/p\u003e","title":"DDPM"},{"content":"Previous diffusion models (DM) generally operate over the full image, which makes the GPU memory required to run a DM model very large. The latent diffusion model (LDM), by contrast, maps the input to a latent space through a pretrained autoencoder encoder, performs the DM operations in that latent space, and finally maps the output back to the original image space through the autoencoder decoder. The authors also introduce the conditioning information into the DM through cross attention, thereby achieving controllable image generation.\nAutoencoder An autoencoder is a kind of neural network that learns in an unsupervised manner. It is trained by comparing the generated image with the original image, that is, it focuses on the model\u0026rsquo;s reconstruction ability; an AE model has no generative ability either. This network consists of two parts:\nEncoder: converts the original high-dimensional input data into a latent low-dimensional code. The input size is larger than the output size. Decoder: restores the latent low-dimensional code obtained by the encoder into the original input data. In LDM, the authors use two optional regularization methods, $\\text{KL-reg}$ and $\\text{VQ-reg}$, to avoid the large variance that the latent space is very likely to exhibit. The loss functions used are the perceptual loss and a Discriminator loss that involves adversarial training.\nKL-reg: the autoencoder model under this penalty is called AutoencoderKL, which can be regarded as a standard VAE (with a KL loss). This model adds a small KL penalty to the learned latent variable so that it approaches a standard normal distribution. VQ-reg: this autoencoder model can be regarded as VQ-GAN, but the vector quantized layer is placed in the decoder. Loss function The perceptual loss in LDM uses the learnable Learned Perceptual Image Patch Similarity (LPIPS) to measure the difference between two images. The Discriminator loss borrows the training process of the taming transformer.\nLPIPS When computing image similarity, traditional methods such as L2, SSIM and PSNR differ considerably from human judgment, especially when images are smooth, where traditional methods are very likely to fail. The authors use deep learning network features to measure image similarity and verified the effectiveness of this design through a large number of experiments. Here the authors introduce two new evaluation metrics:\n2AFC (two alternative forced choice): the experimenter is given one original image and two modified images, and must judge which one is closer to the original. A problem with this metric is that the experimenter may consciously choose which aspects of similarity to focus on. JND (just noticeable difference): the experimenter is shown two images in succession, each for 1 second with a 250-millisecond gap, and then asked to judge whether the two images are the same. Only asking the experimenter whether the two images are similar avoids, to some extent, the subjectivity of the experimenter\u0026rsquo;s choice when both images are partly similar. The structure of LPIPS is shown below:\nThe two patches $\\mathbf{x}$ and $\\mathbf{x}_0$ are fed separately into a feature extraction model $\\mathcal{F}$, which can be a network such as VGG or AlexNet (VGG-16 in LDM) The outputs of different layers (more precisely, different stages) of $\\mathcal{F}$ are extracted and normalized along the channel dimension to obtain $\\hat{y}^l, \\hat{y}^l \\in \\mathbb{R}^{H_l\\times W_l\\times C_l}$, where $l$ indicates that the output of the $l$-th layer is extracted $w^l\\in\\mathbb{R}^{C_l}$ is used to scale $\\hat{y}^l$ and $\\hat{y}^l$ per channel, after which their L2 distance is computed. Finally the average gives the result The formula is described as follows:\n$$ d(x,x_0)=\\sum_l\\frac{1}{H_lW_l}\\sum_{h,w}\\vert\\vert w_l\\odot(\\hat{y}^l_{hw} - \\hat{y}^l_{0hw})\\vert\\vert_2^2 $$When $w_l=1$ for $\\forall l$, the formula above can be seen as the cosine distance.\nDuring training, as shown in the figure above, the two distances are mainly used as the input of a small network $\\mathcal{G}$ to compute a score $\\hat{h}\\in(0,1)$, and then the model is trained with a cross-entropy loss; $h$ in the figure is the new metric introduced by the authors, which here is 2AFC.\nIn the LPIPS paper, the authors give three ways of training the model:\nlin: fix the pretrained weights of $\\mathcal{F}$ and train only the weights $w$ tune: load the pretrained weights of $\\mathcal{F}$ and train them together with $w$ scratch: initialize the model weights with a random Gaussian distribution and train from scratch Discriminator loss A PatchGAN discriminator is used to obtain a patch-level realism evaluation of the image, which is optimized with an adversarial loss function. Referring to VQ-GAN, its loss function can be roughly expressed as follows:\n$$ L_D = [\\log(D(x)) + \\log(1-D(\\hat{x}))] $$where $D$ is the discriminator, $x$ is a real sample and $\\hat{x}$ is a generated sample. For the details of this part, refer to VQ-GAN.\nConditoinal Denoiser The DM part is similar to DDPM, in that a UNet also predicts the noise. Both the Latent Diffusion Model and Stable Diffusion adopt a time-conditional UNet structure to predict the noise at each step, but the official files do not provide a detailed UNet structure figure. Referring to another paper, the rough structure of the time-conditional UNet is as follows:\nNote: this structure is based on version SD2.1, and the conditioning used is text conditioning. Compared with SD1.5, the main change is that the text encoder is replaced from CLIP ViT-L/14 with OpenCLIP-ViT/H; the rest is basically the same.\nFrom the figure above we can see that, compared with the original UNet, the main change is that the original convolution structures are replaced by the Cross Attention Down Block, Down Block, Cross Attention Mid Block, Up Block and Cross Attention Up Block modules, and that time embedding and condition embedding (text embedding in the figure) are added as extra inputs.\nExtra inputs First the time embedding. According to the forward process of diffusion models, we have a timestep, and the goal of the UNet is to predict the random noise added at each timestep during the forward process, so the timestep is necessary information for the UNet. Following the transformer approach, sinusoidal positional embedding is applied to the timestep, and then two linear layers give the time embedding.\nThe condition embedding requires a different network to extract features depending on the type of conditioning. For example, the figure shows the text type, for which the corresponding model used is OpenCLIP.\nExtra modules As for the Cross Attention Down Block and Cross Attention Up Block modules, the structure of the Cross Attention Down Block is as follows:\nThe structure of the Cross Attention Up Block is as follows:\nIn both modules above, the time embedding only interacts with the ResNet Block, while the condition embedding participates in the Transformer Model through cross attention.\nThe detailed structure of the ResNet Block is as follows. The time embedding is mapped by a linear layer to a suitable size so that it can be added element-wise to the output of the convolution layer. The final $1\\times 1$ convolution is optional and is meant to adjust the number of channels of the output feature map.\nThe detailed structure of the Transformer Model is as follows. This structure can be divided into three parts. The first part converts the input feature map into one-dimensional vectors and outputs them to the transformer. The second part is a regular transformer, in which the K and V values of the second Multi-Head Attention are produced by the condition embedding. The third part converts the one-dimensional vectors output by the transformer back into a feature map of the same size as the input.\nThe structures of the remaining modules (Up Block, Down Block and Cross Attention Mid Block) are as follows:\nAppendix The differences between the latent diffusion model and the stable diffusion model (the two models are basically the same in structure):\nCompared with the training data of Latent diffusion (LAION-400M), stable diffusion was trained on a much larger amount of data (LAION-2B-en), and the data was also pre-filtered, so the quality is somewhat higher The text encoder used in the latent diffusion model is a randomly initialized transformer (that is, trained from scratch), whereas the stable diffusion model uses a pretrained CLIP model The training sizes are also different. For example, the latent diffusion model was only trained at a resolution of $256\\times 256$, whereas the stable diffusion model is first pretrained at $256\\times 256$ and then fine-tuned at $512\\times 512$ For a more detailed answer, see here\nThe differences between the various versions of Stable diffusion. Since stable diffusion was released, the official team has put out many versions of the SD model, but the changes in principles and structure among these models are in fact not large; the differences are more likely in generation style.\nThe text encoder of SD 1.x uses OpenAI\u0026rsquo;s CLIP ViT-L/14 model, with 123.65M parameters, whereas SD 2.0 uses a larger text encoder: the CLIP ViT-H/14 model trained on the laion-2b dataset based on OpenCLIP, with 354.03M parameters In the text encoder, SD 1.x extracts the features of the last layer. Since the features of the last layer are immediately followed by CLIP\u0026rsquo;s contrastive learning task, the last layer\u0026rsquo;s features may have partly lost fine-grained semantic information. SD 2.0 extracts the features of the second-to-last layer of the text encoder The attention modules at different stages in SD 2.0 fix the attention head dim to 64, whereas SD 1.x uses a fixed number of attention heads for the attention modules at different stages; clearly the SD 2.0 setting is more common, but this change does not affect the model parameters As for the training data, the SD 1.x version actually ended up mainly using the subset of LAION-2B with an aesthetic score of 5 or above, whereas the SD 2.0 version uses the subset with a score of 4.5 or above, which amounts to enlarging the training dataset For more details, see here\nReference High-Resolution Image Synthesis with Latent Diffusion Models\nThe Unreasonable Effectiveness of Deep Features as a Perceptual Metric\nThe Big Data Myth: Using Diffusion Models for Dataset Generation to Train Deep Detection Models\nLatent Diffusion(Stable-Diffusion) Model (video)\nLatent Diffusion(Stable-Diffusion) Model (pdf)\nThe Illustrated Stable Diffusion\nFrom Autoencoder to Beta-VAE\nA Beginner\u0026rsquo;s Guide to Variational Methods: Mean-Field Approximation\n","permalink":"https://simplestory.github.io/en/posts/20240817_86b1d339bba4e0c154df43f21e99fe50/","summary":"\u003cp\u003ePrevious diffusion models (DM) generally operate over the full image, which makes the GPU memory required to run a DM model very large. The latent diffusion model (LDM), by contrast, maps the input to a latent space through a pretrained autoencoder encoder, performs the DM operations in that latent space, and finally maps the output back to the original image space through the autoencoder decoder. The authors also introduce the conditioning information into the DM through cross attention, thereby achieving controllable image generation.\u003c/p\u003e","title":"LDM"},{"content":"Ref: https://lilianweng.github.io/posts/2017-08-20-gan/\nSome background knowledge\nFor two distributions $P_1$ and $P_2$, their KL divergence (Kullback-Leibler divergence) and JS divergence (Jensen-Shannon divergence) can be used to describe the difference between the two distributions, specifically as follows:\n$$ \\begin{aligned} D_{KL}(P_1 \\vert\\vert P_2) \u0026= \\mathbb{E}_{x\\sim P_1}\\left[\\log\\frac{P_1}{P_2}\\right] = \\int_x P_1\\log\\frac{P_1}{P_2}dx \\\\ D_{JS}(P_1 \\vert\\vert P_2) \u0026= \\frac{1}{2}D_{KL}\\left(P_1\\vert\\vert\\frac{P_1+P_2}{2}\\right) + \\frac{1}{2}D_{KL}\\left(P_2\\vert\\vert\\frac{P_1+P_2}{P_2}\\right) \\end{aligned} $$A GAN usually contains two models: a discriminator $D$ and a generator $G$. The output of the discriminator is the probability that a sample comes from the real data, that is, its purpose is to tell real data apart from generated data, while the purpose of the generator is to make the generated data as close as possible to the real data distribution, that is, to make it hard for the discriminator to tell them apart.\nThe process above is realized by training $D$ and $G$ alternately. In practice the discriminator is usually trained first, then the generator, and this is repeated until the two models reach an equilibrium (Nash equilibrium).\nBefore getting to the main text, let me define some variables:\n$p_z$: the data distribution of the noise input $z$, usually a Gaussian distribution $p_g$: the data distribution generated by the generator $p_r$: the real data distribution Original GAN In the original GAN, for $D$ we want the discriminator\u0026rsquo;s score on real data to be as high as possible, that is, to maximize $\\mathbb{E}_{x\\sim p_r(x)}[\\log D(x)]$, while for fake data we want the discriminator\u0026rsquo;s score to be as small as possible (close to 0), that is, to maximize $\\mathbb{E}_{z\\sim p_z(z)}[\\log(1-D(G(z)))]$. For $G$, we want the generated data to be as close to the real data as possible (close to 1), which is reflected as minimizing $\\mathbb{E}_{z\\sim p_z(z)}[\\log(1-D(G(z)))]$.\nIn summary, the loss function of the original GAN is:\n$$ \\begin{aligned} \u0026\\min_G \\max_D L(D,G) \\\\ \u0026 \\begin{aligned} L(D,G) \u0026= \\mathbb{E}_{x\\sim p_r(x)}[\\log D(x)] + \\mathbb{E}_{z\\sim p_z(z)}[\\log(1-D(G(z)))] \\\\ \u0026= \\mathbb{E}_{x\\sim p_r(x)}[\\log D(x)] + \\mathbb{E}_{x\\sim p_g(x)}[\\log(1-D(x))] \\end{aligned} \\end{aligned} $$Taking the derivative of $L(D,G)$ with respect to $D$ gives the optimal discriminator $D$:\n$$ D^\\star(x) = \\frac{p_r(x)}{p_r(x)+p_g(x)} \\in [0, 1] $$When the generator is optimal, $p_r = p_g$, so the discriminator is $\\frac{1}{2}$.\nWhen both the discriminator and the generator are optimal, that is, $D^\\star(x)=\\frac{1}{2}$, substituting into $L(G,D)$ gives the value of the loss function at the global optimum:\n$$ \\begin{aligned} L(G^\\star,D^\\star) \u0026= \\mathbb{E}_{x\\sim p_r(x)}[\\log D(x)] + \\mathbb{E}_{x\\sim p_g(x)}[\\log(1-D(x))] \\\\ \u0026= \\int_x \\bigg(p_r(x)\\log(D^\\star(x)) + p_g(x)\\log(1-D^\\star(x))\\bigg)dx \\\\ \u0026= \\log\\frac{1}{2}\\int_x p_r(x)dx + \\log\\frac{1}{2}\\int_x p_g(x)dx \\\\ \u0026= -2\\log2 \\end{aligned} $$Assuming the discriminator is optimal, rewriting the JS divergence gives:\n$$ \\begin{aligned} D_{JS}(p_{r} \\| p_g) =\u0026 \\frac{1}{2} D_{KL}(p_{r} || \\frac{p_{r} + p_g}{2}) + \\frac{1}{2} D_{KL}(p_{g} || \\frac{p_{r} + p_g}{2}) \\\\ =\u0026 \\frac{1}{2} \\bigg( \\log2 + \\int_x p_{r}(x) \\log \\frac{p_{r}(x)}{p_{r} + p_g(x)} dx \\bigg) + \\\\\u0026 \\frac{1}{2} \\bigg( \\log2 + \\int_x p_g(x) \\log \\frac{p_g(x)}{p_{r} + p_g(x)} dx \\bigg) \\\\ =\u0026 \\frac{1}{2} \\bigg( \\log4 + L(G, D^*) \\bigg) \\end{aligned} $$that is, $L(G,D^\\star) = 2D_{JS}(p_r\\vert\\vert p_g) - 2\\log2$.\nWhen the discriminator can accurately separate real data from fake data, we have $\\forall x \\in p_r, D(x) = 1$ and $\\forall x \\in p_g, D(x) = 0$, and substituting into $L(G,D)$ gives 0, that is, the gradient is 0.\nFrom this we get the main reason why the original GAN is hard to train: when the discriminator converges very well, the generator\u0026rsquo;s gradient vanishes and the loss cannot decrease; when the discriminator does not converge well, the generator\u0026rsquo;s gradient provides no guidance. Only when the discriminator converges to a certain degree is the generator\u0026rsquo;s gradient effective and instructive, and that degree is very hard to control.\nLater the authors proposed an improved loss function for the generator: $\\mathbb{E}_{x\\sim p_g}[-\\log D(x)]$.\nRewriting the KL divergence gives:\n$$ \\begin{aligned} D_{KL}(p_g\\vert\\vert p_r) \u0026= \\mathbb{E}_{x\\sim p_g}\\bigg[\\log\\frac{p_g}{p_r}\\bigg] \\\\ \u0026= \\mathbb{E}_{x\\sim p_g}\\bigg[\\log\\frac{p_g/(p_r+p_g)}{p_r/(p_r+p_g)}\\bigg] \\\\ \u0026= \\mathbb{E}_{x\\sim p_g}\\bigg[\\log\\frac{1-D^\\star(x)}{D^\\star(x)}\\bigg] \\\\ \u0026= \\mathbb{E}_{x\\sim p_g}\\log[1-D^\\star(x)] - \\mathbb{E}_{x\\sim p_g}[\\log D^\\star(x)] \\end{aligned} $$Combining with $L(G, D^\\star)$ above gives:\n$$ \\begin{aligned} \\mathbb{E}_{x\\sim p_g}[-\\log D(x)] \u0026= D_{KL}(p_g\\vert\\vert p_r) - \\mathbb{E}_{x\\sim p_g}[\\log(1-D^\\star(x))] \\\\ \u0026= D_{KL}(p_g\\vert\\vert p_r) - 2D_{JS}(p_r\\vert\\vert p_g) + 2\\log2 + \\mathbb{E}_{x\\sim p_r}[\\log D^\\star(x)] \\end{aligned} $$Since the last two terms do not depend on the generator, minimizing $\\mathbb{E}_{x\\sim p_g}[-\\log D(x)]$ is ultimately equivalent to minimizing $D_{KL}(p_g\\vert\\vert p_r) - 2D_{JS}(p_r\\vert\\vert p_g)$.\nThis expression has to minimize the KL divergence between the generated distribution and the real distribution while at the same time maximizing the JS divergence between them, which is self-contradictory. Moreover, the JS divergence term makes model training unstable.\nFor the KL divergence term $D_{KL}(p_g\\vert\\vert p_r)$, we have:\nWhen $p_g(x) \\rightarrow 0$ and $p_r(x)\\rightarrow 1$, $p_g(x)\\log\\frac{p_g(x)}{p_r(x)}\\rightarrow 0$, that is, the contribution to $D_{KL}(p_g\\vert\\vert p_r)$ tends to zero When $p_g(x) \\rightarrow 1$ and $p_r(x)\\rightarrow 0$, $p_g(x)\\log\\frac{p_g(x)}{p_r(x)}\\rightarrow +\\infty$, that is, the contribution to $D_{KL}(p_g\\vert\\vert p_r)$ tends to positive infinity So the model penalizes the two errors above differently. The first error corresponds to the generator failing to generate real samples, and the penalty is tiny. The second error corresponds to the generator generating unrealistic samples, and the penalty is huge. With this behavior the model tends to generate some repetitive samples that are close to the real distribution, which makes the generated data lack diversity, that is, the mode collapse problem.\nWGAN Wasserstein GAN: https://zhuanlan.zhihu.com/p/25071913\nSupport: the subset of the non-zero part of a function; the support of a probability distribution is the set of all parts where the probability density is non-zero.\nManifold: a generalization of the concepts of curve and surface in high-dimensional space; we can understand this concept intuitively in lower dimensions. For example, we say a surface in 3D space is a 2D manifold, because its intrinsic dimension is only 2, and a point moving on this 2D manifold has only two degrees of freedom. Likewise, a curve in 3D or 2D space is a 1D manifold.\nMeasure: a generalization of the concepts of length, area and volume in high-dimensional space, which can be understood as hypervolume.\nWGAN proposes some solutions to the problem that GANs are hard to train, the most important of which is replacing the KL and JS divergences with the Wasserstein distance.\nCompared with the original GAN, the following changes are made:\nRemove the sigmoid from the last layer of the discriminator Do not take the log in the losses of the generator and discriminator After each update of the discriminator parameters, clip their absolute values to no more than a fixed constant Do not use momentum-based optimization algorithms (such as momentum and Adam); SGD, RMSProp and the like are recommended But in practice WGAN still has training difficulties, mainly because it forcibly enforces the Lipschitz constraint on the discriminator\u0026rsquo;s gradient through weight clipping. So WGAN-GP was proposed afterwards.\n","permalink":"https://simplestory.github.io/en/posts/20240811_787700a547309b9912534d7c3540b79b/","summary":"\u003cp\u003eRef: \u003ca href=\"https://lilianweng.github.io/posts/2017-08-20-gan/\"\u003ehttps://lilianweng.github.io/posts/2017-08-20-gan/\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSome background knowledge\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eFor two distributions $P_1$ and $P_2$, their KL divergence (Kullback-Leibler divergence) and JS divergence (Jensen-Shannon divergence) can be used to describe the difference between the two distributions, specifically as follows:\u003c/p\u003e\n$$\n\\begin{aligned}\nD_{KL}(P_1 \\vert\\vert P_2) \u0026= \\mathbb{E}_{x\\sim P_1}\\left[\\log\\frac{P_1}{P_2}\\right] = \\int_x P_1\\log\\frac{P_1}{P_2}dx \\\\\nD_{JS}(P_1 \\vert\\vert P_2) \u0026= \\frac{1}{2}D_{KL}\\left(P_1\\vert\\vert\\frac{P_1+P_2}{2}\\right) + \\frac{1}{2}D_{KL}\\left(P_2\\vert\\vert\\frac{P_1+P_2}{P_2}\\right)\n\\end{aligned}\n$$\u003cp\u003eA GAN usually contains two models: a discriminator $D$ and a generator $G$. The output of the discriminator is the probability that a sample comes from the real data, that is, its purpose is to tell real data apart from generated data, while the purpose of the generator is to make the generated data as close as possible to the real data distribution, that is, to make it hard for the discriminator to tell them apart.\u003c/p\u003e","title":"GAN"},{"content":"Note: VAE is not a model that only cares about reconstruction; its more important ability is image generation, so VAE is not an AE model — it has generative ability.\nUnlike the common Autoencoder, the variational autoencoder does not map the input to a fixed vector but to a distribution $p_\\theta$ (with parameters $\\theta$). The relationship between the input data $\\mathbf{x}$ and the latent variable $\\mathbf{z}$ is as follows:\n$p_\\theta(\\mathbf{z})$: prior probability $p_\\theta(\\mathbf{x}\\vert\\mathbf{z})$: likelihood $p_\\theta(\\mathbf{z}\\vert\\mathbf{x})$: posterior probability Assuming that after obtaining the optimal model parameters $\\theta^{\\star}$ we can generate samples that follow the real data distribution through the following steps:\nSample a $\\mathbf{z}^{(i)}$ from the prior $p_{\\theta^\\star}(\\mathbf{z})$ Generate $\\mathbf{x}^{(i)}$ according to the conditional probability $p_{\\theta^\\star}(\\mathbf{x}\\vert\\mathbf{z}=\\mathbf{z}^{(i)})$ The optimal parameters $\\theta^\\star$ can be obtained by maximizing the probability of recovering the real data distribution:\n$$ \\theta^\\star = argmax_\\theta\\prod_{i=1}^np_\\theta(\\mathbf{x}^{(i)}) $$Moving to log space gives:\n$$ \\theta^\\star = argmax_\\theta\\sum\\log p_\\theta(\\mathbf{x}^{(i)}) $$To bring in an encoding vector we have:\n$$ p_\\theta(\\mathbf{x}^{(i)}) = \\int p_\\theta(\\mathbf{x^{(i)}}\\vert\\mathbf{z})p_\\theta(\\mathbf{z})d\\mathbf{z} $$According to the formula above, we would need to iterate over all $\\mathbf{z}$ to compute it, so in practice $p_\\theta(\\mathbf{x}^{(i)})$ is very hard to obtain. The authors solve this problem with an approximate distribution, that is, for each input sample $\\mathbf{x}$, a distribution $q_\\phi(\\mathbf{z}\\vert\\mathbf{x})$ parameterized by $\\phi$ is introduced to directly approximate the posterior distribution $p_\\theta(\\mathbf{z}\\vert\\mathbf{x})$. This guarantees a one-to-one correspondence between the input sample, the latent variable and the reconstructed sample. So the final structure of VAE is fairly similar to that of an Autoencoder:\nThe latent variable is obtained from the approximate posterior $q_\\phi(\\mathbf{z}\\vert\\mathbf{x})$, that is, the encoder part. $q_\\phi(\\mathbf{z}\\vert\\mathbf{x})$ is also called the probabilistic encoder. The result is generated from the likelihood $p_\\theta(\\mathbf{x}\\vert\\mathbf{z})$, which corresponds to the decoder part. $p_\\theta(\\mathbf{x}\\vert\\mathbf{z})$ is also called the probabilistic decoder. We want the approximate distribution $q_\\phi(\\mathbf{z}\\vert\\mathbf{x})$ to be as similar as possible to $p_\\theta(\\mathbf{z}\\vert\\mathbf{x})$. A natural idea is to minimize the KL divergence between them, $D_{KL}(q_\\phi(\\mathbf{z}\\vert\\mathbf{x})\\vert\\vert p_\\theta(\\mathbf{z}\\vert\\mathbf{x}))$. Expanding and simplifying $D_{KL}$ gives:\n$$ \\begin{aligned} \u0026 D_\\text{KL}( q_\\phi(\\mathbf{z}\\vert\\mathbf{x}) \\vert\\vert p_\\theta(\\mathbf{z}\\vert\\mathbf{x}) ) \\\\ =\u0026 \\log p_\\theta(\\mathbf{x}) + D_\\text{KL}(q_\\phi(\\mathbf{z}\\vert\\mathbf{x}) \\vert\\vert p_\\theta(\\mathbf{z})) - \\mathbb{E}_{\\mathbf{z}\\sim q_\\phi(\\mathbf{z}\\vert\\mathbf{x})}\\log p_\\theta(\\mathbf{x}\\vert\\mathbf{z}) \\end{aligned} $$that is:\n$$ \\begin{aligned} \u0026 \\log p_\\theta(\\mathbf{x}) - D_\\text{KL}( q_\\phi(\\mathbf{z}\\vert\\mathbf{x}) \\vert\\vert p_\\theta(\\mathbf{z}\\vert\\mathbf{x}) ) \\\\ =\u0026 \\mathbb{E}_{\\mathbf{z}\\sim q_\\phi(\\mathbf{z}\\vert\\mathbf{x})}\\log p_\\theta(\\mathbf{x}\\vert\\mathbf{z}) - D_\\text{KL}(q_\\phi(\\mathbf{z}\\vert\\mathbf{x}) \\vert\\vert p_\\theta(\\mathbf{z})) \\end{aligned} $$Maximizing the formula above is our optimization objective, that is, maximizing the likelihood of generating the real data distribution while minimizing the KL divergence between the approximate posterior and the true posterior. Taking the negative gives the loss function:\n$$ \\begin{aligned} L_\\text{VAE}(\\theta, \\phi) \u0026= -\\log p_\\theta(\\mathbf{x}) + D_\\text{KL}( q_\\phi(\\mathbf{z}\\vert\\mathbf{x}) \\vert\\vert p_\\theta(\\mathbf{z}\\vert\\mathbf{x}) )\\\\ \u0026= - \\mathbb{E}_{\\mathbf{z} \\sim q_\\phi(\\mathbf{z}\\vert\\mathbf{x})} \\log p_\\theta(\\mathbf{x}\\vert\\mathbf{z}) + D_\\text{KL}( q_\\phi(\\mathbf{z}\\vert\\mathbf{x}) \\vert\\vert p_\\theta(\\mathbf{z}) ) \\\\ \\theta^{*}, \\phi^{*} \u0026= \\arg\\min_{\\theta, \\phi} L_\\text{VAE} \\end{aligned} \\tag{1} $$Since the KL divergence is non-negative, we have:\n$$ -L_\\text{VAE} = \\log p_\\theta(\\mathbf{x}) - D_\\text{KL}( q_\\phi(\\mathbf{z}\\vert\\mathbf{x}) \\| p_\\theta(\\mathbf{z}\\vert\\mathbf{x}) ) \\leq \\log p_\\theta(\\mathbf{x}) $$So by minimizing $L_\\text{VAE}$ we can maximize the lower bound on the probability of generating real data samples. $L_\\text{VAE}$ is also called the variational lower bound (VLB) or the evidence lower bound (ELBO).\nFrom equation (1) above we can see that computing the loss function requires sampling from the distribution $q_\\phi(\\mathbf{z}\\vert\\mathbf{x})$, and the random sampling process cannot be used to compute gradients. The VAE paper uses the reparameterization trick to move the randomness of the sampling process onto other variables:\n$$ \\begin{aligned} \\mathbf{z} \u0026\\sim q_\\phi(\\mathbf{z}\\vert\\mathbf{x}^{(i)}) = \\mathcal{N}(\\mathbf{z}; \\boldsymbol{\\mu}^{(i)}, \\boldsymbol{\\sigma}^{2(i)}\\boldsymbol{I}) \u0026 \\\\ \\mathbf{z} \u0026= \\boldsymbol{\\mu} + \\boldsymbol{\\sigma} \\odot \\boldsymbol{\\epsilon} \\text{, where } \\boldsymbol{\\epsilon} \\sim \\mathcal{N}(0, \\boldsymbol{I}) \u0026 \\scriptstyle{\\text{; Reparameterization trick.}} \\end{aligned} $$That is, a sample $\\mathbf{\\epsilon}$ is first drawn from the standard normal distribution, then multiplied by the standard deviation predicted by the VAE encoder and added to the mean predicted by the encoder; this way the gradients of the loss with respect to the parameters of the VAE encoder network can be computed.\n$L_\\text{VAE}$ can also be seen as the sum of a reconstruction loss and a regularization term, where the first term $\\mathbb{E}_{\\mathbf{z}\\sim q_\\phi(\\mathbf{z}\\vert\\mathbf{x})}$ corresponds to the reconstruction loss and $D_{KL}$ acts as a KL regularization term. Depending on the type of data, the $p_\\theta(\\mathbf{x}\\vert\\mathbf{z})$ distribution contained in the first term of the loss differs, and the resulting reconstruction loss differs as well. For example, when $p_\\theta(\\mathbf{x}\\vert\\mathbf{z})$ is a multivariate Gaussian with independent components, substituting the related expressions gives an L2 reconstruction loss, while when the distribution is a Bernoulli distribution the resulting reconstruction loss is the cross-entropy.\nHere is a small point to note. Since the KL divergence is asymmetric, why do we minimize $D_{KL}(q_\\phi\\vert\\vert p_\\theta)$ rather than $D_{KL}(p_\\theta\\vert\\vert q_\\phi)$? (See this article for details.)\n$D_{KL}(p_\\theta\\vert\\vert q_\\phi)$ is also called Forward KL, while $D_{KL}(q_\\phi\\vert\\vert p_\\theta)$ is called Reversed KL. Expanding Forward KL gives:\n$$ D_{KL}(p_\\theta\\vert\\vert q_\\phi) = \\sum_z p_\\theta(z)\\log\\frac{p_\\theta(z)}{q_\\phi(z)} $$For any $p_\\theta(z) \\gt 0$, we have $\\lim_{q_\\phi(z) \\to 0} \\log \\frac{p_\\theta(z)}{q_\\phi(z)} \\to \\infty$, which means that when $q_\\phi$ cannot cover $p_\\theta$, the value of Forward KL becomes very large. So the resulting $q_\\phi$ will cover the whole of $q_\\theta$, as shown below:\nExpanding Reversed KL gives:\n$$ D_{KL}(q_\\phi\\vert\\vert q_\\theta) = \\sum_z q_\\phi(z)\\log \\frac{q_\\phi(z)}{p_\\theta(z)} $$In the formula above, when $p_\\theta(z)$ is 0, $q_\\phi(z)$ must also be 0, otherwise the value of the KL divergence becomes abnormal. So the resulting $q_\\phi$ is confined within the range of $p_\\theta$, as shown below:\nReference Auto-Encoding Variational Bayes\nVAE among generative models\n","permalink":"https://simplestory.github.io/en/posts/20240811_a9423d60164f001a6715546c56c77154/","summary":"\u003cp\u003e\u003cstrong\u003eNote: VAE is not a model that only cares about reconstruction; its more important ability is image generation, so VAE is not an AE model — it has generative ability.\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eUnlike the common Autoencoder, the variational autoencoder does not map the input to a fixed vector but to a distribution $p_\\theta$ (with parameters $\\theta$). The relationship between the input data $\\mathbf{x}$ and the latent variable $\\mathbf{z}$ is as follows:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e$p_\\theta(\\mathbf{z})$: prior probability\u003c/li\u003e\n\u003cli\u003e$p_\\theta(\\mathbf{x}\\vert\\mathbf{z})$: likelihood\u003c/li\u003e\n\u003cli\u003e$p_\\theta(\\mathbf{z}\\vert\\mathbf{x})$: posterior probability\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eAssuming that after obtaining the optimal model parameters $\\theta^{\\star}$ we can generate samples that follow the real data distribution through the following steps:\u003c/p\u003e","title":"VAE"},{"content":"VQGAN is a generative model that performs well on a variety of image generation tasks (such as unconditional image generation, image completion, conditionally guided image generation and so on), the biggest highlight being image generation at the megapixel level.\nThe overall pipeline of the model is as follows:\nOverall it is similar to VQVAE, but the PixelCNN in VQVAE is replaced by a Transformer, and a PatchGAN discriminator is added during training to provide an adversarial loss.\nLoss function In VQVAE, the loss function can be expressed as follows:\n$$ \\begin{aligned} L_{VQ}(E,G,Z) \u0026= L_{rec} + \\vert\\vert\\text{sg}[E(x)] - z_q\\vert\\vert_2^2 + \\vert\\vert\\text{sg}[z_q] - E(x)\\vert\\vert_2^2 \\\\ \u0026= \\vert\\vert x - \\hat{x}\\vert\\vert_2^2 + \\vert\\vert\\text{sg}[E(x)] - z_q\\vert\\vert_2^2 + \\vert\\vert\\text{sg}[z_q] - E(x)\\vert\\vert_2^2 \\end{aligned} $$where $E,G,Z$ denote the encoder, decoder and codebook respectively. $\\text{sg}$ is the stop gradient operation. $L_{rec}$ is the reconstruction loss.\nIn VQGAN, the authors replace $L_{rec}$ with a perceptual loss and also add the adversarial loss from GANs. In practice, to achieve better results on a particular task, the L2 loss may be used together with the perceptual loss, or the L2 loss may even be replaced by an L1 loss, which needs to be determined experimentally for each specific task.\nPerceptual loss In tasks such as image style transfer and super-resolution, traditional pixel-loss functions (such as L1, L2 or cross entropy) all pursue equality between the model output and the target at the pixel level, but a very small deviation in pixel values makes essentially no difference in actual appearance while the value of the pixel-loss becomes very large. The perceptual loss mainly focuses on features such as style, color and texture of the image, which better reflects the model\u0026rsquo;s optimization objective.\nComputing this loss function requires an extra network to extract features; the concrete computation flow is roughly as follows:\n$f_W$ in the figure corresponds to the model we need to train. VGG-16 is a model pretrained on Imagenet, mainly used to extract features and not involved in weight updates. The perceptual loss is divided into two parts: content loss and style loss.\ncontent loss is the L2 loss at the feature level. Computing it at the feature level rather than on the original image is meant to avoid the output becoming excessively similar to the target while still keeping the ability to guide the update of the content in the output. The mathematical expression is as follows:\n$$ L^{\\phi, j}_{feat}(\\hat{y}, y) = \\frac{1}{C_jH_jW_j}\\vert\\vert \\phi_j(\\hat{y}) - \\phi_j(y)\\vert\\vert_2^2 $$where $\\phi_j(\\hat{y})$ and $\\phi_j(y)$ denote the output features of the model output $\\hat{y}$ and the target $y$ at the $j$-th layer of the VGG-16 network, with $\\phi_j(\\hat{y}), \\phi_j(y)\\in\\mathbb{R}^{H_j\\times W_j\\times C_j}$. As can be seen from the figure above, the feature layer extracted in the paper is the output of relu3_3.\nstyle loss is used to compute the differences between images in style, color, texture and so on. Suppose $\\phi_j(x)\\in\\mathbb{R}^{H_j\\times W_j\\times C_j}$ denotes the output feature of input $x$ at the $j$-th layer of the model $\\phi$. Here a Gram matrix $G_j^\\phi(x)$ of size $C_j\\times C_j$ is first defined:\n$$ G_j^\\phi(x)_{c,c^\\prime} = \\frac{1}{C_jH_jW_j}\\sum^{H_j}_{h=1}\\sum^{W_j}_{w=1}\\phi_j(x)_{h,w,c}\\phi_j(x)_{h,w,c^\\prime} $$In actual computation we can reshape $\\phi_j(x)$ to $\\psi\\in\\mathbb{R}^{C_j\\times H_jW_j}$, and then the Gram matrix computation simplifies as follows:\n$$ G^\\phi_j(x) = \\frac{\\psi\\psi^T}{C_jH_jW_j} $$The style loss is defined as the Frobenius norm distance on the Gram matrix:\n$$ L_{style}^{\\phi,j}(\\hat{y},y) = \\vert\\vert G_j^\\phi(\\hat{y}) - G_j^\\phi(y)\\vert\\vert_F^2 $$The size of the output of the Gram matrix computation depends only on the number of channels of the input (the output size is $C_j\\times C_j$), so as long as the number of channels is the same, the loss can be computed even if the sizes of the model output and the target differ.\nIn the feature extraction network, convolution layers at different depths extract different features (lower layers extract point and line features while higher layers extract abstract features), so in order to comprehensively evaluate the model\u0026rsquo;s style ability we need to extract the outputs of various network layers to compute the style loss. According to the figure above, the authors extract the output features of the four layers relu1_2, relu2_2, relu3_3 and relu4_3 to compute the style loss, and finally sum them as the value of the style loss.\nAs for the Frobenius norm, its computation formula is as follows ($A\\in\\mathbb{R}^{m\\times n}$):\n$$ \\vert\\vert A\\vert\\vert_F = \\sqrt{\\sum^m_{i=1}\\sum^n_{j=1}\\vert a_{ij}\\vert^2} = \\sqrt{\\text{trace}(A^\\star A)} = \\sqrt{\\sum_{i=1}^{\\min(m,n)}\\sigma_i^2} $$Three ways of computing the Frobenius norm are given above. Here $A^\\star$ denotes the conjugate transpose of $A$ (the conjugate transpose swaps rows and columns and then conjugates each element; for a real matrix the conjugate transpose is just the transpose), and $\\sigma_i$ are the singular values of $A$, using the trace function.\nAdversarial loss As for the adversarial loss part, the authors chose the Hinge loss suited to GANs. The ordinary Hinge loss is usually used in maximal margin classifiers such as support vector machines, improving model performance by widening as much as possible the score gap between positive and negative samples. Its mathematical expression is as follows:\n$$ L = \\max(0, 1-\\hat{y}y) $$where $\\hat{y}$ denotes the score the model outputs for a sample and $y$ denotes the ground truth label of the sample (1 or -1).\nWhen $\\hat{y}$ and $y$ have the same sign, that is, the predicted class matches the true class: if $\\vert\\hat{y}\\vert\\le 1$, it means the margin is not large enough, so the loss is not 0 and updating continues until $\\vert\\hat{y}\\vert\\gt 1$, where the loss becomes 0 When $\\hat{y}$ and $y$ have opposite signs, that is, the predicted class differs from the true class, the loss is not 0 In Geometric GAN, the Hinge loss is introduced and transformed into the following form:\n$$ \\begin{aligned} L(G,D) \u0026= L(G) + L(D) \\\\ \u0026= -E(D(G(z))) + E(\\max(0, 1-D(x))) + E(\\max(0, 1+D(G(z)))) \\end{aligned} $$The Hinge loss is mainly applied to the discriminator $D$. Only real samples with $D(x)\\lt 1$ and generated samples with $D(G(z))\\gt -1$ produce a loss, that is, only samples that are unreasonably classified contribute, which makes GAN training more stable.\nIn VQGAN, the adversarial loss is loosely expressed as follows:\n$$ L_{GAN}(\\{E,G,Z\\}, D) = [\\log(D(x)) + \\log(1-D(\\hat{x}))] $$Our goal is to obtain the optimal $E, G, Z$; the final summarized optimization objective is as follows:\n$$ Q^\\star = argmin_{E,G,Z}\\max_D\\mathbb{E}_{x\\sim p(x)}[L_{VQ}(E,G,Z) + \\lambda L_{GAN}(\\{E,G,Z\\}, D)] $$where $\\lambda$ is an adaptive weight, expressed as follows:\n$$ \\lambda = \\frac{\\nabla_{G_L}[L_{rec}]}{\\nabla_{G_L}[L_{GAN}]+\\delta} $$$L_{rec}$ is the perceptual loss, $\\nabla_{G_L}[\\cdot]$ is a function denoting the gradient of the input of that function with respect to the last layer $L$ of the model decoder, and $\\delta=10^{-6}$ is to prevent division by zero.\nThe newly added discriminator $D$ in VQGAN has the structure of a PatchGAN discriminator, which is a fully convolutional model structure. A usual GAN discriminator outputs only a single value, whereas a PatchGAN discriminator outputs an $n\\times n$ matrix in which each value represents the score of the corresponding small region (patch) of the image for the input model. This design lets the model focus more on the details of the image and no longer evaluates the realism of an image with a single value.\nRandom image generation The optimization objective above is already enough to train the various modules of the model, but in the random image generation task we need to sample $z_q$ and feed it to the decoder without an encoder. The VQVAE method predicts $z_q$ with PixelCNN. Considering that a Transformer has long-distance attention and that this sampling process can also be compared to the text sequence prediction problem, the authors use a Transformer-structure model to predict $z_q$ (specifically, the structure is GPT-2).\nThe sampling process is treated as autoregressive prediction, that is, the model is made to predict the distribution $p(s)=\\prod_i p(s_i\\vert s_{\\lt i})$, and the corresponding loss function maximizes the log-likelihood:\n$$ L_{Transformer} = \\mathbb{E}_{x\\sim p(x)}[-\\log p(s)] $$The Transformer module needs to be trained after the other modules have been trained; the concrete training process is as follows:\nThe input is passed through the encoder and the codebook to obtain $z_q\\in \\mathbb{R}^{h\\times w\\times n}$, which is unfolded into a 2D form $z_q^\\prime\\in \\mathbb{R}^{hw\\times n}$, that is, $hw$ codes of dimension $n$. The indices in the codebook corresponding to the combination of codes in $z_q^\\prime$ at this point are defined as unmodified_indices Some of the codes in $z_q^\\prime$ are replaced with randomly generated codes (still of dimension $n$), and the indices corresponding to the modified combination of codes are defined as modified_indices. Replacing with random codes amounts to adding noise to the features, which improves the generalization of the Transformer The modified_indices are fed into the Transformer model, and the model is trained with a cross-entropy loss to reconstruct the unmodified_indices With the help of the Transformer, VQGAN achieves image generation at the megapixel level.\nReference Taming Transformers for High-Resolution Image Synthesis\nPerceptual Losses for Real-Time Style Transfer and Super-Resolution\nImage-to-Image Translation with Conditional Adversarial Networks\nMegapixel image generation combining discretized coding and Transformers\n","permalink":"https://simplestory.github.io/en/posts/20240811_f09978b03d9ad93516b6003551892ddc/","summary":"\u003cp\u003eVQGAN is a generative model that performs well on a variety of image generation tasks (such as unconditional image generation, image completion, conditionally guided image generation and so on), the biggest highlight being image generation at the megapixel level.\u003c/p\u003e\n\u003cp\u003eThe overall pipeline of the model is as follows:\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://simplestory-blog-img.oss-cn-guangzhou.aliyuncs.com/in_posts/20240811/vq_gan.png\" title=\"vq_gan\"\u003e\u003c/p\u003e\n\u003cp\u003eOverall it is similar to VQVAE, but the PixelCNN in VQVAE is replaced by a Transformer, and a PatchGAN discriminator is added during training to provide an adversarial loss.\u003c/p\u003e","title":"VQ-GAN"},{"content":"Normally the pixel values of an image are discrete (between 0 and 255), so we can actually regard an image as a sequence of length $H\\times W\\times C$. When predicting the current pixel value, the preceding pixel values are taken as known information and participate in the prediction, and this proceeds recursively pixel by pixel, which is the autoregressive model:\n$$ p(x) = p(x_1)p(x_2\\vert x_1)\\dots p(x_n\\vert x_1,x_2,\\dots,x_{n-1}) $$But the process above generates one pixel at a time, so it runs slowly. VQ-VAE borrows the idea of the autoregressive model, and its most obvious characteristic is that the latent variable obtained through the encoder is a discrete variable, which compresses the image into a small feature space. The overall pipeline is roughly shown below:\nThe input $\\mathbf{x}$ of size $H\\times W\\times 3$ is passed through the encoder to obtain a preliminary intermediate variable $\\mathbf{z}=\\text{encoder}(\\mathbf{x})$ of reduced size $h\\times w\\times d$ The model maintains an embedding layer $\\mathbf{E}=[\\mathbf{e}_1, \\mathbf{e}_2, \\dots, \\mathbf{e}_K]$, that is, the codebook, where each $\\mathbf{e}_i$ has length $d$. For each $d$-dimensional vector in $\\mathbf{z}$, nearest neighbor search gives the corresponding $\\mathbf{e}_i$ value, that is, the mapping $\\mathbf{z}_q(\\mathbf{x}) =\\mathbf{e}_{argmin_j\\vert\\vert \\mathbf{z}(\\mathbf{x})-\\mathbf{e}_j\\vert\\vert_2}$. The resulting encoding vector is denoted $\\mathbf{z}_q$, and such an encoding vector is in fact a discrete vector $\\mathbf{z}_q$ is fed into the decoder to obtain the result $\\hat{\\mathbf{x}} = \\text{decoder}(\\mathbf{z}_q)$ The paper gives the loss function of VQ-VAE as follows:\n$$ L = -\\log p(\\mathbf{x}\\vert\\mathbf{z}_q(\\mathbf{x}))+\\vert\\vert\\text{sg}[\\mathbf{z}(\\mathbf{x})]-\\mathbf{z}_q(\\mathbf{x})\\vert\\vert_2^2+\\beta\\vert\\vert\\mathbf{z}(\\mathbf{x})-\\text{sg}[\\mathbf{z}_q(\\mathbf{x})]\\vert\\vert_2^2 $$The first term is used to optimize the encoder and the decoder, that is, the reconstruction loss; from the figure we can also see that the gradient of the loss with respect to $\\mathbf{z}$ is passed to the encoder. This term is usually the MSE loss between the reconstructed image and the input image, $\\vert\\vert \\mathbf{x}-\\text{decoder}(\\mathbf{z}_q)\\vert\\vert_2^2$.\nBecause our forward computation involves the $\\argmin$ operation, which has no gradient, the authors use the Straight-Through Estimator method to artificially design a gradient, namely $\\vert\\vert \\mathbf{x}-\\text{decoder}(\\mathbf{z} + \\text{sg}[\\mathbf{z}_q-\\mathbf{z}])\\vert\\vert_2^2$. The $\\text{sg}$ in the formula stands for stop gradient, that is, it is 0 during backpropagation and the original value in the forward computation, described mathematically as follows:\n$$ \\text{sg}(x) = \\begin{cases} x \\quad \\text{in forward process} \\\\ 0 \\quad \\text{backward process} \\end{cases} $$The last two terms of the loss function are related to the codebook. Usually we want $\\mathbf{z}_q$ and $\\mathbf{z}$ to be as close as possible, so the optimization objective is $\\vert\\vert\\mathbf{z}_q - \\mathbf{z}\\vert\\vert_2^2$. Since the gradient of this objective equals the gradient with respect to $\\mathbf{z}$ plus the gradient with respect to $\\mathbf{z}_q$, we can equivalently write it as follows:\n$$ \\vert\\vert\\text{sg}[\\mathbf{z}]-\\mathbf{z}_q\\vert\\vert_2^2 + \\vert\\vert\\mathbf{z}-\\text{sg}[\\mathbf{z}_q]\\vert\\vert_2^2 $$Equivalent here means the gradients are equivalent; the loss value actually computed in the forward pass is twice the original. In the formula above, the first term is the gradient of $\\mathbf{z}_q$, which amounts to fixing $\\mathbf{z}$ and moving $\\mathbf{z}_q$ towards $\\mathbf{z}$; the second term is the gradient of $\\mathbf{z}$, which amounts to fixing $\\mathbf{z}_q$ and moving $\\mathbf{z}$ towards $\\mathbf{z}_q$. Since $\\mathbf{z}_q$ is freer than $\\mathbf{z}$ (as $\\mathbf{z}$ is the output of the encoder), and to do our best to guarantee the reconstruction quality, we should try to make $\\mathbf{z}_q$ approach $\\mathbf{z}$. This step can use weights to adjust the importance of each loss term.\nIn summary, the final loss function of VQ-VAE is as follows:\n$$ L = \\vert\\vert \\mathbf{x}-\\text{decoder}(\\mathbf{z} + \\text{sg}[\\mathbf{z}_q-\\mathbf{z}])\\vert\\vert_2^2 + \\gamma\\vert\\vert\\text{sg}[\\mathbf{z}]-\\mathbf{z}_q\\vert\\vert_2^2 +\\beta \\vert\\vert\\mathbf{z}-\\text{sg}[\\mathbf{z}_q]\\vert\\vert_2^2 $$where $\\beta \\lt \\gamma$; the paper chooses $\\gamma = 1$ and $\\beta = 0.25$.\nFrom the description above we can see that VQ-VAE actually has little to do with VAE; it is merely an AE model whose goal is image reconstruction.\nIn the VQ-VAE paper, the authors use PixelCNN to predict $\\mathbf{z}_q$, which enables random image generation. During training the model trains VQ-VAE and PixelCNN jointly: the input data is passed through the encoder to obtain $\\mathbf{z}$, the codebook is queried to obtain $\\mathbf{z}_q$, and then PixelCNN produces a variable $\\mathbf{z}_p$ of the same size as $\\mathbf{z}_q$, which serves as the input of the decoder. At inference time, by contrast, an empty image is fed directly into PixelCNN to obtain $\\mathbf{z}_p^\\prime$, which is then fed directly into the decoder to generate an image.\nFor the PixelCNN model, see this article.\nReference Neural Discrete Representation Learning\nA concise introduction to VQ-VAE\n","permalink":"https://simplestory.github.io/en/posts/20240811_520db3fb8aba368d31c1f3e5e9fbd8ba/","summary":"\u003cp\u003eNormally the pixel values of an image are discrete (between 0 and 255), so we can actually regard an image as a sequence of length $H\\times W\\times C$. When predicting the current pixel value, the preceding pixel values are taken as known information and participate in the prediction, and this proceeds recursively pixel by pixel, which is the autoregressive model:\u003c/p\u003e\n$$\np(x) = p(x_1)p(x_2\\vert x_1)\\dots p(x_n\\vert x_1,x_2,\\dots,x_{n-1})\n$$\u003cp\u003eBut the process above generates one pixel at a time, so it runs slowly. VQ-VAE borrows the idea of the autoregressive model, and its most obvious characteristic is that the latent variable obtained through the encoder is a discrete variable, which compresses the image into a small feature space. The overall pipeline is roughly shown below:\u003c/p\u003e","title":"VQ-VAE"},{"content":"LSS is an early pioneering work that used BEV for autonomous driving perception, and many of the later BEV-related algorithms are variations built on the LSS framework.\nBEV perception BEV (Bird\u0026rsquo;s Eye View) perception, as the name suggests, solves the perception problem of autonomous driving from a top-down viewpoint. Object detection, segmentation, tracking and other problems are much more straightforward in this view, and for autonomous driving scenarios the height information of objects is actually not that important; most downstream tasks such as decision making and planning also work with this top-down view, and fusing information from multiple cameras becomes much easier. That is why autonomous driving solutions under BEV are simpler and more direct.\nFor pure-vision autonomous driving, the key to the BEV approach is how to transform or encode the data from the forward-facing camera view into a top-down view. There are currently four main methods:\nIPM (Inverse Perspective Mapping), which inversely maps the camera perspective space back to BEV space. IPM assumes that the ground is a perfect plane with no road height variation. This assumption differs greatly from real applications, so IPM is generally used for lane detection, where the assumption is usually reasonable. Lift-Splat, which roughly means using a Lift operation to lift 2D image features into a 3D frustum, and then using Splat to flatten it in the BEV direction. This framework was proposed by LSS and has had a great influence on later algorithms. MLP (Multi-Layer Perceptron), that is, using an MLP to learn the transformation. See VPN for more. This kind of method is very deployment-friendly, but it relies on a large amount of data to learn the mapping. Transformer, which uses a transformer for view transformation and is not limited by the receptive field the way a CNN is. The autonomous driving solution Tesla shared earlier uses a transformer. This kind of method is somewhat difficult to train and deploy, but may achieve better results. Lift-Splat-Shoot LSS is an end-to-end framework that extracts features from one or more camera images and lifts them into a frustum for each corresponding camera, then splats these frustums into a BEV grid. This is roughly the lift-splat pipeline; the following shoot can be understood as a downstream task, for example path prediction in the case of LSS.\nLift For pure vision, the depth corresponding to each pixel in the image is not certain, so LSS generates a representation at every possible depth of each pixel. Specifically, the Lift operation generates a discretized depth distribution $\\alpha$ for every point in the feature map ($D$ values in total), then takes the outer product with the feature $\\mathbf{c}$ of the corresponding point (of length $C$) and multiplies by the corresponding $\\alpha$ value. This yields the camera frustum. The process is roughly illustrated below:\nSplat This step projects the frustum generated above into a BEV grid of a given size. In the original LSS paper, the perception range along the $x$ and $y$ axes is $-50m\\sim 50m$ and along the $z$ axis is $-10m\\sim 10m$; the BEV cell size is $0.5m$ along the $x$ and $y$ axes and $20m$ along the $z$ axis, so the resulting BEV grid has a size of $200\\times 200\\times 1$.\nDuring projection, multiple points may fall into a single BEV cell, and LSS fuses them with sum pooling. Because the generated frustum contains many points, the model is not very efficient, so LSS optimizes the sum pooling by using a cumsum trick to improve efficiency.\nFor the details of LSS I recommend reading the code open-sourced by NVIDIA; it is not complicated, only 5 or 6 Python files. I briefly organized the algorithm flow, roughly as follows:\nThere are also some good code walkthroughs online:\nLSS (Lift, Splat, Shoot) paper + source code in-depth analysis Reference Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D\nA new paradigm for autonomous driving perception — summary and comparison of classic BEV perception papers\n","permalink":"https://simplestory.github.io/en/posts/20230624_593a3023b826e242be5651751b69d09e/","summary":"\u003cp\u003eLSS is an early pioneering work that used BEV for autonomous driving perception, and many of the later BEV-related algorithms are variations built on the LSS framework.\u003c/p\u003e\n\u003ch2 id=\"bev-perception\"\u003eBEV perception\u003c/h2\u003e\n\u003cp\u003eBEV (Bird\u0026rsquo;s Eye View) perception, as the name suggests, solves the perception problem of autonomous driving from a top-down viewpoint. Object detection, segmentation, tracking and other problems are much more straightforward in this view, and for autonomous driving scenarios the height information of objects is actually not that important; most downstream tasks such as decision making and planning also work with this top-down view, and fusing information from multiple cameras becomes much easier. That is why autonomous driving solutions under BEV are simpler and more direct.\u003c/p\u003e","title":"LSS"},{"content":"Monocular 3D detection is an important and challenging task in the autonomous driving field; the current difficulty is how to accurately predict the depth of objects. MonoRCNN uses geometric relations to decouple the depth estimation problem into the object\u0026rsquo;s real height and its projected height in the image.\nThe authors impose two conditions on the use of this model:\nThe model only predicts the heading angle of the object, that is, the yaw angle The camera intrinsics are known during both training and inference Architecture The model is similar to Faster R-CNN; the backbone is a combination of ResNet-50 and FPN, then RoIAlign is used to extract object features, followed by three branches: a 2D head, a 3D attribute head and a 3D distance head. The final predictions give the following parameters:\nclass label $cls$ and confidence $score$ 2D bounding box $\\mathbf{b} = (x_1, y_1, x_2, y_2)$, represented by the top-left and bottom-right coordinates projected coordinates of the center point of the 3D bounding box in the image, $\\mathbf{p} = (p_1, p_2)$ physical size of the 3D bounding box $\\mathbf{m} = (W, H, L)$, denoting the real width, height and length of the object respectively yaw angle of the 3D bounding box $\\mathbf{a} = (sin\\theta, cos\\theta)$ depth $Z$ of the center point of the 3D bounding box MonoRCNN predicts the coordinates of the 3D bounding box center in the image coordinate system $(p_1, p_2, Z)$, which are then converted to a point in camera coordinates with the camera intrinsic matrix $\\mathbf{P}$:\n$$ \\begin{bmatrix} p_1 \\cdot Z \\\\ p_2 \\cdot Z \\\\ Z \\end{bmatrix}_P = \\mathbf{P} \\cdot \\begin{bmatrix} x \\\\ y \\\\ z \\\\ 1 \\end{bmatrix}_C $$3D Distance Head As shown above, from simple geometric relations we obtain:\n$$ Z = \\frac{fH}{h} = fHh_{rec} $$where $h_{rec} = \\frac{1}{h}$. The authors predict $H$ and $h_{rec}$ by regression to obtain $Z$ indirectly. Since only the object height in the image needs to be obtained, this method can effectively deal with complex scenarios such as occlusion, reflection and truncated objects. At the same time, this decoupling also maintains self-consistency, so that depth can still be predicted accurately even when both predicted heights are inaccurate: for example, when the predicted $H$ is larger, the model adaptively decreases $h_{rec}$. For the details of this part, refer to the original paper.\nThe loss function of this branch is an uncertainty-aware regression loss:\n$$ \\begin{aligned} L_H \u0026= \\frac{L_1(\\hat{H}, H)}{\\sigma_H} + \\lambda_Hlog(\\sigma_H) \\\\ L_{h_{rec}} \u0026= \\frac{L_1(\\hat{h}_{rec}, h_{rec})}{\\sigma_{h_{rec}}} + \\lambda_{h_{rec}}log(\\sigma_{h_{rec}}) \\end{aligned} $$where $\\hat{H}$ and $\\hat{h}_{rec}$ are the ground truths, $H$ and $h_{rec}$ are the predictions, $\\lambda_H$ and $\\lambda_{h_{rec}}$ are positive coefficients used to balance the uncertainty terms, and $\\sigma_H$ and $\\sigma_{h_{rec}}$ are learnable variables for the uncertainty.\nThe authors show experimentally that $\\sigma_{h_{rec}}$ is more discriminative for objects at different distances than $\\sigma_H$, and that $fH\\sigma_{h_{rec}}$ can represent the uncertainty of the distance. So the authors use $\\frac{score}{fH\\sigma_{h_{rec}}}$ instead of $score$ to rank the predictions in order to improve model accuracy.\n3D Attribute Head This branch mainly predicts the physical size of the object, the yaw angle and the 2D keypoints. For the physical size and yaw angle, an L1 loss is used directly for regression:\n$$ \\begin{aligned} L_{size} \u0026= L_1(\\hat{\\mathbf{m}}, \\mathbf{m}) \\\\ L_{yaw} \u0026= L_1(\\hat{\\mathbf{a}}, \\mathbf{a}) \\end{aligned} $$For the 2D keypoints, define: $\\hat{\\mathbf{t}} = \\left ( \\frac{\\hat{p}_1 - x_1}{x_2 - x_1}, \\frac{\\hat{p}_2 - y_1}{y_2 - y_1} \\right )$, then:\n$$ L_{kpt} = L_1(\\hat{\\mathbf{t}}, \\mathbf{t}) $$Finally, the overall loss of the model is as follows:\n$$ \\begin{aligned} L =\u0026 \\lambda_{cls}L_{cls} + \\lambda_{bbox}L_{bbox} + \\lambda_{size}L_{size} \\\\ \u0026+ \\lambda_{yaw}L_{yaw} + \\lambda_{kpt}L_{kpt} + \\lambda_HL_H + \\lambda_{h_{rec}}L_{h_{rec}} \\end{aligned} $$where $\\lambda_{cls}=1$, $\\lambda_{bbox}=1$, $\\lambda_{size}=3$, $\\lambda_{yaw}=5$, $\\lambda_{kpt}=5$, $\\lambda_{H}=0.25$, $\\lambda_{h_{rec}}=1$\nReference Geometry-based Distance Decomposition for Monocular 3D Object Detection\n","permalink":"https://simplestory.github.io/en/posts/20230623_f25271ca7acc8b883a6ebeba1a0b263d/","summary":"\u003cp\u003eMonocular 3D detection is an important and challenging task in the autonomous driving field; the current difficulty is how to accurately predict the depth of objects. MonoRCNN uses geometric relations to decouple the depth estimation problem into the object\u0026rsquo;s real height and its projected height in the image.\u003c/p\u003e\n\u003cp\u003eThe authors impose two conditions on the use of this model:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eThe model only predicts the heading angle of the object, that is, the yaw angle\u003c/li\u003e\n\u003cli\u003eThe camera intrinsics are known during both training and inference\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"architecture\"\u003eArchitecture\u003c/h2\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://simplestory-blog-img.oss-cn-guangzhou.aliyuncs.com/in_posts/20230623/monorcnn_architecture.png\" title=\"monorcnn_architecture\"\u003e\u003c/p\u003e","title":"MonoRCNN"},{"content":"A lightweight stereo matching module is introduced into the 2D real-time object detection framework yolo to achieve efficient 3D object detection. The rough structure is shown below:\nAnchors For 3D object detection, the original yolo anchors designed for 2D objects need to be redesigned. The authors use 12 parameters to represent an anchor:\n$[x_{2d}, y_{2d}, w_{2d}, h_{2d}]$, representing the 2D object box $[c_x, c_y, z]$ represents the position of the 3D object center in the left image $[w_{3d}, h_{3d}, l_{3d}]$ represents the width, height and length of the 3D object box $[sin(2\\alpha), cos(2\\alpha)]$ to estimate the angle of the object Here the authors found that $[sin\\alpha, cos\\alpha]$ and $[sin(\\alpha+\\pi), cos(\\alpha+\\pi)]$ may produce identical rectangular boxes, so they use $[sin(2\\alpha), cos(2\\alpha)]$ to represent the angle of the object, and additionally add a classification branch to predict whether $\\vert\\alpha\\vert \\gt \\frac{\\pi}{2}$, which also indicates whether the object is facing the camera.\nA 3D statistical prior is applied to the anchors to improve the regression results. Iterating over the training set, for each anchor of a different size, the object boxes assigned to that anchor are collected based on IOU, and then the averages of $z$, $sin(2\\alpha)$ and $cos(2\\alpha)$ are computed. Because 3D boxes of different classes differ considerably, the authors collect the 3D prior information per class.\nConsidering the application scenario of autonomous driving, the authors project each anchor back into 3D space according to its average $z$ value, and then filter out the anchors that are far from the ground. This filtering is also done per class.\nMulti-Scale Stereo Feature Extraction For stereo matching, the main task is to build a cost volume. The current best method is the concatenation-based approach: the right feature map is shifted horizontally over the left feature map to compute the cost volume. To reduce computation time, the authors adopt a correlation-based method to build a lightweight cost volume: a normalized dot-product layer is used to compute over the left and right feature maps (see the inner-product layer and correlation layer mentioned in the two papers cited there). The output dimension of this method is $[B, max\\_disp, H, W]$; the number of output channels is small, which makes the model\u0026rsquo;s fusion stage lean towards monocular features, and subsequent downsampling of the result loses further information.\nThe authors use a ghost module to expand the number of channels. The ghost module uses depthwise convolution to produce additional features. The authors go further and concatenate the output of the ghost module with the input, tripling the number of channels. In part b of the structure figure above, the light purple part is the output of the ghost module.\nTo minimize information loss in the stereo matching stage while keeping computation time under control, the authors build a hierarchical multi-scale fusion mechanism:\nAt downsampling rates of 4x and 8x, the authors build cost volumes with a maximum disparity of 96 and 192 respectively, which are then fed into the ghost module, downsampled and concatenated with later, smaller-scale feature maps. At a downsampling rate of 16x, the number of channels is first downsampled with a $1\\times 1$ convolution, and then the authors use the concatenation-based method to create a small cost volume so as to retain more semantic information. The details are shown in the structure figure above.\nTraining Scheme and Loss Function First, for data augmentation, the authors apply photometric distortion to both stereo images simultaneously and perform random flipping during training. The random flipping here includes flipping the RGB images together, flipping the position and direction of the objects, and then swapping the left and right images.\nAs can be seen from the structure figure, the model has an extra auxiliary branch during training. Disparity supervision is important for improving the detection performance of the model. The authors upsample the final stereo feature to $[W/4, H/4]$, and supervise the predicted disparity with a coarse disparity ground truth obtained by the traditional block matching algorithm in OpenCV. At inference time this branch is not involved in the computation.\nAs for the loss function, focal loss is used for classification, smoothed-L1 loss for the object boxes, and stereo focal loss for the auxiliary disparity estimation (see AcfNet for details).\nFirst the expected disparity distribution is computed (with the variance directly set to $\\sigma = 0.5$):\n$$ P(d) = softmax(-\\frac{\\vert d-d^{gt}\\vert}{\\sigma}) = \\frac{exp(-c_d^{gt})}{\\sum_{d'=0}^{D-1}exp(-c_{d'}^{gt})} $$where $d$ denotes the disparity and $c_d$ denotes the confidence when the predicted disparity is $d$. Then the stereo focal loss can be defined as:\n$$ \\mathcal{L}_{SF} = \\frac{1}{\\vert \\mathcal{P}\\vert}\\sum_{p\\in\\mathcal{P}}\\left(\\sum_{d=0}^{D-1}(1-P_p(d))^{-\\alpha}\\cdot\\big(-P_p(d)\\cdot log\\hat{P}_p(d)\\big)\\right) $$where $D$ is the defined maximum disparity (max-disparity), $\\alpha$ is the weight of the focal loss (the larger $\\alpha$ is, the higher the weight of the correct disparity value), $\\mathcal{P}$ is the set of pixels involved in the computation, and $P_p(d)$ and $\\hat{P}_p(d)$ denote the expected and the actually predicted disparity distribution functions, respectively.\nThe final loss function is defined as the sum of the three loss functions above.\nReference YOLOStereo3D: A Step Back to 2D for Efficient Stereo 3D Detection\n","permalink":"https://simplestory.github.io/en/posts/20230623_e5106aece831780fe8decad391998e6a/","summary":"\u003cp\u003eA lightweight stereo matching module is introduced into the 2D real-time object detection framework yolo to achieve efficient 3D object detection. The rough structure is shown below:\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://simplestory-blog-img.oss-cn-guangzhou.aliyuncs.com/in_posts/20230623/yolostereo3d_architecture.png\" title=\"yolostereo3d_architecture\"\u003e\u003c/p\u003e\n\u003ch2 id=\"anchors\"\u003eAnchors\u003c/h2\u003e\n\u003cp\u003eFor 3D object detection, the original yolo anchors designed for 2D objects need to be redesigned. The authors use 12 parameters to represent an anchor:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e$[x_{2d}, y_{2d}, w_{2d}, h_{2d}]$, representing the 2D object box\u003c/li\u003e\n\u003cli\u003e$[c_x, c_y, z]$ represents the position of the 3D object center in the left image\u003c/li\u003e\n\u003cli\u003e$[w_{3d}, h_{3d}, l_{3d}]$ represents the width, height and length of the 3D object box\u003c/li\u003e\n\u003cli\u003e$[sin(2\\alpha), cos(2\\alpha)]$ to estimate the angle of the object\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eHere the authors found that $[sin\\alpha, cos\\alpha]$ and $[sin(\\alpha+\\pi), cos(\\alpha+\\pi)]$ may produce identical rectangular boxes, so they use $[sin(2\\alpha), cos(2\\alpha)]$ to represent the angle of the object, and additionally add a classification branch to predict whether $\\vert\\alpha\\vert \\gt \\frac{\\pi}{2}$, which also indicates whether the object is facing the camera.\u003c/p\u003e","title":"YOLOStereo3D"},{"content":"Point clouds are sparse in distribution, irregular in structure and unordered geometrically, so applying convolution operations directly on point clouds may make the generated features deviate greatly from their original representation.\nVoxelNet This model voxelizes the point cloud and then uses 3D convolution to obtain features. Specifically:\nThe point cloud space is divided equally into a voxel grid of size $v_D$, $v_H$, $v_W$, and at most $T$ points are randomly kept within each cell (extras are removed, fewer are left as they are). A VFE layer (voxel feature encoding layer) is applied to each voxel cell to extract voxel features, which are merged into a 4D tensor. 3D convolution is applied to the tensor obtained above to extract features further, and then an RPN structure is used to obtain the results As shown below:\nVFE VFE is the most important structure in VoxelNet. For a single-layer VFE, there are mainly the following steps:\nCompute the mean coordinate of all points in the voxel and the deviation of each point from the mean to obtain the input features of the points, which are merged as the input features of VFE $\\mathbf{V}_{in} = \\{\\hat{\\mathbf{p}}_i = [x_i, y_i, z_i, r_i, x_i-v_x, y_i-v_y, z_i-v_z]^T \\in \\mathbb{R}^7\\}_{i=1\\dots t}$ Each $\\hat{\\mathbf{p}}_i$ is passed through an FCN structure composed of fully connected layers, BN layers and ReLU layers to obtain the point feature $f_i \\in \\mathbb{R}^m$ Element-wise max pooling is applied to the features of all points in $V_{in}$ to obtain an aggregated feature $f \\in \\mathbb{R}^m$. The element-wise max pooling here can be understood as np.max(f, axis=0): an input feature of size (n_point_in_voxel, m) becomes a feature of size (1, m) after element-wise max pooling Concatenating $f$ and $f_i$ gives the final feature of each point, $f^{out}_i = [f^T_i, f^T]^T \\in \\mathbb{R}^{2m}$. Merging the $f^{out}_i$ of every point in the voxel gives the final output feature of VFE, $V_{out}=\\{f^{out}_i\\}_{i=1\\dots t}$ The concrete structure is shown below:\nThis per-point operation may not make full use of the computing device, so the paper also mentions an efficient implementation of VFE later on; see the original paper for details.\nRPN After the features are obtained through VFE, several structures composed of 3D convolution, BN layers and ReLU layers are used to extract features further. Finally the RPN structure below is used to obtain the final output:\nSECOND VoxelNet uses 3D convolution layers, and this convolution layer is computationally expensive, which makes the overall inference speed of the model slow. This paper proposes a sparse 3D convolution operation, and also proposes angle loss regression to address the insensitivity of VoxelNet to object orientation.\nThere are very detailed articles online about the sparse 3D convolution method and about SECOND; the key is to build the element index tables of the input and output. I recommend these two:\nAn analysis of the SECOND algorithm\nAn easy-to-understand explanation of the Sparse Convolution process\nData Augmentation SECOND devotes about half a page specifically to explaining some of the data augmentation methods they use.\nSample Ground Truths from the Database Because point cloud ground truth data is rather scarce, both the convergence speed of model training and the final performance are greatly limited. The authors augment the point cloud data with a data augmentation method similar to small object pasting.\nUse the existing data to build a database with point clouds and corresponding ground truth Introduce the objects in the database into the current point cloud data by concatenation Use collision detection when removing and concatenating objects to verify whether the current operation conforms to physical rules This method can effectively increase the number of ground truth objects in the point cloud, and can also improve the diversity of the data.\nObject Noise Similar to VoxelNet, noise is added to the objects in the point cloud independently and randomly. But VoxelNet uses the same parameters for all points, whereas the authors randomly pick a value from $\\Delta\\theta \\in [-\\pi/2, \\pi/2]$ for the random rotation, and randomly pick a value from a Gaussian distribution with mean 0 and standard deviation 1 for the random linear transformation.\nGlobal Rotation and Scaling The whole point cloud data and the corresponding annotation boxes are scaled and rotated, with a scaling range of $[0.95, 1.05]$ and a rotation range of $[-\\pi/4, \\pi/4]$.\nReference VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection\nSECOND: Sparsely Embedded Convolutional Detection\n","permalink":"https://simplestory.github.io/en/posts/20230305_f73b955eec390396d75e03ababcc4909/","summary":"\u003cp\u003ePoint clouds are sparse in distribution, irregular in structure and unordered geometrically, so applying convolution operations directly on point clouds may make the generated features deviate greatly from their original representation.\u003c/p\u003e\n\u003ch1 id=\"voxelnet\"\u003eVoxelNet\u003c/h1\u003e\n\u003cp\u003eThis model voxelizes the point cloud and then uses 3D convolution to obtain features. Specifically:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eThe point cloud space is divided equally into a voxel grid of size $v_D$, $v_H$, $v_W$, and at most $T$ points are randomly kept within each cell (extras are removed, fewer are left as they are).\u003c/li\u003e\n\u003cli\u003eA VFE layer (voxel feature encoding layer) is applied to each voxel cell to extract voxel features, which are merged into a 4D tensor.\u003c/li\u003e\n\u003cli\u003e3D convolution is applied to the tensor obtained above to extract features further, and then an RPN structure is used to obtain the results\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eAs shown below:\u003c/p\u003e","title":"VoxelNet and SECOND"},{"content":" This network performs 3D object detection using stereo vision, and belongs to the image-based result-lifting methods in 3D detection, which decompose 3D detection into two parts: 2D object detection and depth estimation.\nStereo RCNN extends Faster RCNN, as shown below:\nBackbone The authors use ResNet-101 combined with FPN, with shared weights. This backbone processes the left and right images at the same time, and the results are then merged as the input of Stereo RPN.\nStereo RPN To make the anchors generated for the left and right boxes correspond to each other, the authors use different GT boxes for objectness classification and box regression, as follows:\nIn the objectness classification stage, the bounding rectangle of the corresponding left and right GT boxes is taken as the union GT. An anchor whose IOU with the union GT is above 0.7 is taken as a positive sample, and below 0.3 as a negative sample. Since this IOU computation only involves the size of the anchor, it is the same for the left and right images. In the box regression stage, the respective GT boxes are used to regress the anchor coordinates. The parameters to regress are $[u, w, u', w', v, h]$, where $u$ and $v$ denote the horizontal and vertical coordinates of the center of the 2D box, and $h$ and $w$ denote its length and width. Because the input left and right images are rectified, the vertical coordinates of the left and right images are basically the same, that is, $v$ and $h$ are essentially the same. Finally NMS is used to take the top 2000 samples for training, and 300 at test time.\nStereo RCNN As can be seen from the network structure figure above, after Stereo RPN and the regular ROI Align there are two branches, namely Stereo Regression and Keypoint Prediction.\nStereo Regression After performing ROI Align on the left and right images separately, the results are concatenated and fed into fully connected layers. Here the ROIs are filtered: when the IOU of the left and right ROIs with the GT is both greater than 0.5 the sample is positive, and when it is less than 0.5 but at least 0.1 it is negative. The regression parameters are:\nobject class bounding box: the same as in stereo rpn object dimension: the average size is computed first, then the relative quantity is predicted viewpoint angle: $[sin\\alpha, cos\\alpha]$ Regarding the viewpoint angle, see the figure below:\n$\\theta$ is the orientation angle of the object in the camera coordinate system, and $\\beta$ is the azimuth angle of the object relative to the camera center. These three quantities projected onto the camera are basically the same, so to distinguish them better the authors choose to regress the view angle, that is, $\\alpha = \\theta + \\beta$, where $\\beta = arctan(-\\frac{x}{z})$, and to maintain continuity the final regression parameters are $[sin\\alpha, cos\\alpha]$.\nKeypoint Prediction Projecting the four bottom points of the 3D object box onto the image plane reveals that at most one 3D keypoint can be clearly projected inside the box (not on the left or right edge); the authors define this 3D keypoint as the perspective keypoint and only predict it on the left image. This keypoint can effectively constrain the 3D object box. For the concrete implementation details, refer to the original paper.\n3D Box Estimation Now we already have the 2D box, the size, the view angle and the perspective keypoint; with these we can obtain a preliminary 3D box: $(x, y, z, \\theta)$, that is, the center point and orientation angle of the 3D box. The method used minimizes the error of the projection of the 3D box onto the 2D box and the keypoints.\nThe input is $\\mathbf{z} =\\{u_l, v_t, u_r, v_b, u'_l, u'_r, u_p\\}$, denoting the coordinates of the top-left and bottom-right corners of the 2D box in the left image, the left and right coordinates in the $u$ direction of the 2D box in the right image, and the coordinates of the perspective keypoint. Equations are established by projecting the object points in the world coordinate system to the object points in image coordinates, for example the projection of the top-left corner:\n$$ \\begin{bmatrix} u_l \\\\ v_t \\\\ 1 \\end{bmatrix} = \\begin{bmatrix} x \\\\ y \\\\ z \\end{bmatrix} + \\begin{bmatrix} cos\\theta \u0026 0 \u0026 sin\\theta \\\\ 0 \u0026 1 \u0026 0 \\\\ -sin\\theta \u0026 0 \u0026 cos\\theta \\end{bmatrix} \\cdot \\begin{bmatrix} -\\frac{w}{2} \\\\ -\\frac{h}{2} \\\\ -\\frac{l}{2} \\end{bmatrix} $$This gives the equations for $u_l$ and $v_t$. The equations for the other parameters in $\\mathbf{z}$ can be derived in the same way (noting the changes of sign), and are then solved with the Gauss-Newton method.\nThe paper specifically notes that, for convenience of representation, the parameters in $\\mathbf{z}$ have already been normalized by the camera intrinsics\nDense 3D Box Alignment This part refines the accuracy of the 3D box obtained above. The authors define the region within the object range that lies below half of its height and between the left and right edges of the object as the valid region, and optimize over this ROI. The error function is defined as follows:\n$$ \\begin{aligned} E \u0026 = \\sum_{i = 0}^Ne_i \\\\ \u0026 = \\sum_{i = 0}^N\\left \\| I_l(u_i, v_i) - I_r(u_i - \\frac{b}{z + \\Delta z_i}, v_i) \\right \\| \\end{aligned} $$Combined with triangulation, the formula above can be understood as the authors converting the change in object depth into a change in image pixel coordinates, that is, disparity; the formula can in fact be seen as a way of computing a cost volume. Thus, by changing the pixel coordinate values the depth value changes, and the smallest overall pixel difference between the left and right images corresponds to the optimal $z$ value. A coarse-to-fine strategy can be used, for example first searching 50 steps at a precision of 0.5m, then searching 20 times at a precision of 0.05m.\nThis method can be applied to the post-processing of other stereo 3D detectors. Because the object ROI is a physical constraint, this method avoids the discontinuous, ill-posed problems of depth estimation, and it is robust to illumination, since every pixel contributes to the estimate.\nReference Stereo R-CNN based 3D Object Detection for Autonomous Driving\npaper_reading-Stereo R-CNN based 3D Object Detection for Autonomous Driving\nStereo RCNN official code\n","permalink":"https://simplestory.github.io/en/posts/20230127_2a9bce23520b48c68e00a87aa00aa611/","summary":"\u003cblockquote\u003e\n\u003cp\u003eThis network performs 3D object detection using stereo vision, and belongs to the image-based result-lifting methods in 3D detection, which decompose 3D detection into two parts: 2D object detection and depth estimation.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eStereo RCNN extends Faster RCNN, as shown below:\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://simplestory-blog-img.oss-cn-guangzhou.aliyuncs.com/in_posts/20230127/stereo_rcnn_arch.png\" title=\"stereo_rcnn_architecture\"\u003e\u003c/p\u003e\n\u003ch2 id=\"backbone\"\u003eBackbone\u003c/h2\u003e\n\u003cp\u003eThe authors use ResNet-101 combined with FPN, with shared weights. This backbone processes the left and right images at the same time, and the results are then merged as the input of Stereo RPN.\u003c/p\u003e","title":"Stereo RCNN"},{"content":" Deciding which gt to assign to which anchor is an important problem in object detection, namely the label assignment problem. OTA offers a new angle on this problem: treating label assignment as an optimal transport problem.\nLabel Assignment Previous label assignment strategies are mainly divided into two kinds: fixed and dynamic.\nFor fixed label assignment, Anchor-based detection models usually use an IoU threshold as the assignment criterion (such as Fast-RCNN, Faster-RCNN, YoLo3, RetinaNet and so on), while some Anchor-free detection models designate anchors within a certain range of the object center as positive samples (such as FCOS, Foveabox and so on). Fixed assignment methods usually adopt a fixed assignment strategy (for example a fixed IoU threshold or a fixed radius), which is the same for objects of different sizes and shapes, so the model may end up with a suboptimal assignment.\nFor dynamic label assignment, the prediction confidence of each anchor is generally used as a dynamic assignment criterion, and anchors with high confidence are assigned to the relevant objects; however, dynamic assignment methods can still suffer from ambiguous anchor assignment because they cannot exploit global information.\nOTA exploits global information and views label assignment as an optimal transport problem, with the main goal of finding the globally optimal matching of the objects in the image.\nOptimal Transport The standard optimal transport problem is as follows:\nThere are $m$ suppliers and $n$ demanders. The $i$th supplier can provide $s_i$ goods, and the $j$th demander needs $d_j$ goods. The cost of transporting one unit of goods from supplier $i$ to demander $j$ is defined as $c_{ij}$. The goal of the OT problem is to find a transport plan $\\pi^\\ast=\\{\\pi_{i,j}\\vert i=1,2,\\dots,m, j=1,2,\\dots,n\\}$ that minimizes the cost of transporting all goods, expressed as follows:\n$$ \\begin{aligned} \\min_{\\pi}\u0026 \\sum_{i=1}^m\\sum_{j=1}^n c_{ij}\\pi_{ij} \\\\ s.t. \u0026 \\sum_{i=1}^m\\pi_{ij} = d_j,\\quad \\sum_{j=1}^n\\pi_{ij} = s_i \\\\ \u0026 \\sum_{i=1}^ms_i = \\sum_{j=1}^nd_j \\\\ \u0026 \\pi_{ij} \\ge 0,\\quad i=1,2,\\dots,m,\\quad j=1,2,\\dots,n \\end{aligned} $$This problem can be solved in linear time, but considering the feature map dimensions and anchor scales involved in object detection, the authors use Sinkhorn Iteration to deal with the excessive computation time.\nSinkhorn Iteration converts the linear programming problem above into the following convex nonlinear form with an added entropy regularization term:\n$$ \\min_\\pi \\sum_{i=1}^m\\sum_{j=1}^n c_{ij}\\pi_{ij}+\\gamma E(\\pi_{ij}) $$where $E(\\pi_{ij}) = \\pi_{ij}(\\log\\pi_{ij}+1)$ and $\\gamma$ is a hyperparameter controlling the strength of the regularization. By the Lagrange multiplier method we have:\n$$ \\min_\\pi \\sum_{i=1}^m\\sum_{j=1}^n c_{ij}\\pi_{ij}+\\gamma E(\\pi_{ij})+\\alpha_j(\\sum_{i=1}^m\\pi_{ij}-d_j)+\\beta_i(\\sum_{j=1}^n\\pi_{ij}-s_i) $$where $\\alpha_j(j=1,2,\\dots,n)$ and $\\beta_i(i=1,2,\\dots,m)$ are the Lagrange multipliers. Taking the derivative of the formula above and setting it to 0 gives:\n$$ \\pi_{ij}^\\ast = \\exp(-\\frac{\\alpha_j}{\\gamma})\\exp(-\\frac{c_{ij}}{\\gamma})\\exp(-\\frac{\\beta_i}{\\gamma}) $$Let $u_j = \\exp(-\\frac{\\alpha_j}{\\gamma})$, $v_i = \\exp(-\\frac{\\beta_i}{\\gamma})$ and $M_{ij} = \\exp(-\\frac{c_{ij}}{\\gamma})$. Then the constraints can be rewritten as:\n$$ \\begin{aligned} \\sum_i\\pi_{ij} \u0026= u_j(\\sum_iM_{ij}v_i) = d_j \\\\ \\sum_j\\pi_{ij} \u0026= (\\sum_ju_jM_{ij})v_i = s_i \\end{aligned} $$The two equations above must hold simultaneously, which gives the following iterative update:\n$$ \\begin{aligned} u^{t+1}_j \u0026= \\frac{d_j}{\\sum_iM_{ij}v^t_i} \\\\ v^{t+1}_i \u0026= \\frac{s_i}{\\sum_jM_{ij}u^{t+1}_j} \\end{aligned} $$After $T$ iterations, the optimal plan $\\pi^\\ast$ is computed as follows:\n$$ \\pi^\\ast = diag(v)Mdiag(u) $$In the paper the authors set $\\gamma=0.1$ and $T=50$.\nOptimal Transport for Label Assignment Combining the label assignment problem with the OT problem: the authors treat gt as suppliers, with each gt supplying $k$ labels ($s_i=k,i=1,2,\\dots,m$), and anchors as demanders, with each anchor needing 1 label (that is, $d_j=1,j=1,2,\\dots,n$). The transport cost of one positive label from a gt to an anchor is defined as $c^{fg}$, taken as the sum of the $cls$ and $reg$ losses:\n$$ c^{fg}_{ij} = L_{cls}(P^{cls}_{ij}(\\theta),G^{cls}_i)+\\alpha L_{reg}(P^{box}_j(\\theta),G^{box}_i) $$where $\\theta$ denotes the model parameters, $P^{cls}_j$ and $P^{box}_j$ denote the predicted $cls$ score and box coordinates produced by anchor $a_j$, and $G^{cls}_i$ and $G^{box}_i$ denote the class and box coordinates of the ground truth annotation $gt_i$. $L_{cls}$ and $L_{reg}$ are the classification loss (cross-entropy loss or Focal loss can be used) and the box regression loss (IoU loss, GIoU loss or SmoothL1 loss can be used). $\\alpha$ is a balancing factor.\nBesides the matching of positive samples, a considerable portion of anchors are designated as negative samples during training, so the authors also add negative sample assignment by introducing a background supplier (which only provides negative labels). In the standard OT problem the total supply of the suppliers should equal the total demand of the demanders, so the background supply is computed as follows:\n$$ c^{bg}_j =L_{cls}(P^{cls}_j(\\theta), \\varnothing) $$where $\\varnothing$ denotes the background class. Appending $c^{bg}\\in R^{1\\times n}$ to the end of $c^{fg}\\in R^{m\\times n}$ gives the complete cost matrix $c\\in R^{(m+1)\\times n}$, and the supply vector $s$ is updated accordingly as follows:\n$$ s_i = \\begin{cases} k, \u0026 if\\quad i\\le m \\\\ n-m\\times k, \u0026 if\\quad i=m+1 \\end{cases} $$Given the cost matrix $c$, the supply vector $s\\in R^{m+1}$ and the demand vector $d\\in R^n$, solving this OT problem gives the optimal assignment plan $\\pi^\\ast\\in R^{(m+1)\\times n}$.\nSome optimization designs Center prior. That is, anchors are selected for assignment within a designated region around the object center. In theory OTA can designate any anchor in the global scope as a positive sample, and the center prior lets OTA focus more on the potential positive sample regions, which stabilizes the early training of the model. The concrete operation used by the authors is: for each gt, select the top $r^2$ anchors in each FPN layer according to the center distance between the anchor and the gt as positive sample candidates; for the remaining anchors that are not selected, a constant is added at the corresponding position of the cost matrix to reduce the probability of the anchor being selected as a positive sample. Dynamic $k$ estimation. Generally each gt should have a different suitable number of positive anchors/points, because each gt has a different size, scale and degree of occlusion. Based on the IoU values between the predicted boxes and the gt, the authors select the top q IoU values for each gt and take their sum as the suitable number of positive anchors/points for that gt (because the suitable number of positive anchors/points for a gt should be positively correlated with the number of anchors/points that can regress that gt well). Putting the above together, the pseudo-code of OTA is as follows:\nReference OTA: Optimal Transport Assignment for Object Detection\nOTA code\n","permalink":"https://simplestory.github.io/en/posts/20221006_1087bec7ef12911010976d203f36fd7f/","summary":"\u003cblockquote\u003e\n\u003cp\u003eDeciding which gt to assign to which anchor is an important problem in object detection, namely the label assignment problem. OTA offers a new angle on this problem: treating label assignment as an optimal transport problem.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch3 id=\"label-assignment\"\u003eLabel Assignment\u003c/h3\u003e\n\u003cp\u003ePrevious label assignment strategies are mainly divided into two kinds: fixed and dynamic.\u003c/p\u003e\n\u003cp\u003eFor fixed label assignment, Anchor-based detection models usually use an IoU threshold as the assignment criterion (such as Fast-RCNN, Faster-RCNN, YoLo3, RetinaNet and so on), while some Anchor-free detection models designate anchors within a certain range of the object center as positive samples (such as FCOS, Foveabox and so on). Fixed assignment methods usually adopt a fixed assignment strategy (for example a fixed IoU threshold or a fixed radius), which is the same for objects of different sizes and shapes, so the model may end up with a suboptimal assignment.\u003c/p\u003e","title":"OTA"},{"content":" PointNet, PointNet++ and Frustum PointNet. The commonly used 3D point cloud object detection.\nPointNet PointNet applies deep learning directly to point cloud data. Point cloud data is a subset of the points in Euclidean space, and has three properties: disorder, the spatial relationship between points, and invariance.\nDisorder. Point cloud data is a set and is insensitive to the order of the data, which requires the model to be invariant to different permutations of the data. There are several approaches. The first is to sort the point cloud by some rule, but for high-dimensional space it is very hard to find a stable sorting rule. Another is to treat it as a sequence and train it with an RNN, but this approach struggles to handle input elements of extremely large length (thousands to tens of thousands, as in a point cloud). Finally the authors use a symmetric function to aggregate the information of each point: $f(\\lbrace x_1,x_2,\\cdots,x_n\\rbrace)\\approx g(h(x_1),h(h(x_2),\\cdots,h(x_n))$, where the left side of the equation is the goal and the right side is the designed symmetric function. From this we can see that each point in the point cloud is processed separately with $h$ and then passed into the symmetric function $g$. In the paper the authors choose $h$ to be an MLP and $g$ to be max pooling. The spatial relationship between points. An object usually consists of point clouds in a specific space, and there are spatial relationships between points; PointNet exploits these relationships by concatenating global features with local features. Invariance. The object represented by point cloud data should be invariant to some spatial transformations (rigid transformations). PointNet aligns the point cloud data before extracting features to guarantee invariance. The alignment is achieved by training a small network (T-Net) to obtain a transformation matrix and multiplying it with the input point cloud data. This operation can be seen as adjusting the spatial points; intuitively it rotates them into an orientation that is favorable for classification or segmentation (a rigid transformation). In the actual implementation a related loss function is also added to make the transformation matrix approximately orthogonal, so as to reduce the loss of point cloud information. (Regularization: $L_{reg}=\\lVert I-AA^T\\rVert_F^2$, that is, an L2 loss) The rough structure of PointNet is as follows:\nThe figure contains both the classification and the segmentation part, where segmentation combines the global pooled feature with the point cloud features after the second alignment. In the source code the classification part uses T-Net while the segmentation part can do without it, with little improvement to the results, mainly because the PointNet structure itself cannot learn the local relationships between points, even with T-Net added.\nPointNet++ PointNet++ improves on some of the problems of PointNet.\nFirst let me introduce the basic network layers\nSampling layer. Usually the raw point cloud data is enormous, and extracting local features for every point would be extremely expensive, so PoinNet++ samples the point cloud first. The sampling method used is Farthest Point Sampling (FPS), which covers the whole sampling space better than random sampling. Grouping layer. To obtain the local features of a point, we need to obtain the local region of that point. In images, the local region of a pixel is defined as the pixels within a certain Manhattan distance around it. For point cloud data, the local region of a point is defined as the points within a sphere of a given radius around it. The grouping layer finds the points contained in the local region formed around each sampled point, so that features can be extracted afterwards. Feature extraction layer. Here PointNet is borrowed to extract features from each local point cloud. The layers above form the basic module of PointNet++. These basic modules are similar to those of a CNN; cascading several such modules gives PointNet++, which can extract shallow to deep features from point cloud data. The improvements over PointNet are as follows:\nPointNet only expresses the features of each point and has a poor ability to extract local structural information. So PointNet++ uses a sampling layer and a grouping layer to integrate local neighborhoods. PointNet\u0026rsquo;s global feature is obtained directly by max pooling, which causes a large information loss for both classification and segmentation tasks. For this reason PointNet++ downsamples level by level through multiple set abstractions (as shown below), obtaining local-global features of different scales and levels. For segmentation tasks, PointNet directly concatenates the global feature and the local feature, which produces weak discriminative features, whereas PointNet++ designs an encoder-decoder structure for classification and segmentation tasks; for the segmentation part, it first downsamples and then upsamples, using skip connections to concatenate the local-global features of the corresponding layers. Finally the rough structure of the model is as follows:\nSince the distribution of a point cloud in space is irregular and non-uniform, although PointNet++ can learn the local features of a point cloud, the non-uniform distribution of the point cloud in each local region may prevent the model from extracting features well. For this reason the authors adopt two methods:\nMulti-Scale Grouping (MSG). That is, local features of different scales are extracted and concatenated together. This operation increases the computation considerably. As shown in figure a below Multi-Resolution Grouping (MRG). In this method the features extracted for each local region are represented by two concatenated vectors, as shown in figure b below. The first part is obtained by passing the features extracted by the previous layer through the feature extraction layer again, and the second part is obtained by directly extracting features from all the points in the raw point cloud data corresponding to this local region. Principle of the FPS algorithm:\nThe input point cloud has $N$ points; randomly choose one point $P_0$ as the starting point, giving the sample set $S=\\lbrace P_0\\rbrace$; Compute the distance from every point to $P_0$, forming an $N$-dimensional array $L$, choose the point with the largest distance as the sampling point $P_1$ and add it to the sample set $S=\\lbrace P_0,P_1\\rbrace$; Compute the distance from every point to $P_1$; for each point $N_i$, if its distance to $P_1$ is smaller than $L[i]$, update the distance array $L[i]=d(N_i,P_1)$, that is, the array $L$ stores the nearest distance from each point to the sample set $S$; Select the point corresponding to the maximum value in $L$ and add it to the sample set $S=\\lbrace P_0,P_1,P_2\\rbrace$; Repeat steps 2-4 until the length of the sample set reaches $n$, that is, $n$ target points have been sampled This involves the choice of the initial point and the distance metric, generally as follows:\nChoice of the initial point: Pick a random point, so that the result differs every time Pick the point farthest from the centroid of the point cloud, so that the result is the same every time and the initial point is usually located at a local extremum, with a certain descriptive ability Distance metric: Euclidean distance, mainly for point clouds, giving uniform sampling in 3D space Geodesic distance, that is, the distance of the shortest path between two vertices along the mesh surface, mainly for triangle meshes, giving uniform sampling over the triangle mesh surface Finally let me mention the decoder of the segmentation task. After the processing of the encoder above, what we have is the feature representation of a small number of points; to segment the point cloud, PointNet++ uses a form of inverse interpolation to upsample. Specifically:\nSuppose the point cloud before interpolation is $P_1$ with dimension $N_1\\times C$, and the point cloud after it is $P_2$ with dimension $N_2\\times C_2$, where $N_2\\gt N_1$ (since this operation is upsampling). The upsampling computation follows the formula below:\n$$ \\begin{aligned} \u0026 f^{(j)}(x)=\\frac{\\sum_{i=1}^kw_i(x)f_i^{(j)}}{\\sum_{i=1}^kw_i(x)} \\\\ where\\qquad \u0026w_i(x)=\\frac{1}{d(x,x_i)^p}, \\ j=1,2,\\cdots,C \\end{aligned} $$That is, for each point $x$ in $P_2$, find the $k$ points $x_1,x_2,\\cdots,x_k$ in the point cloud space $P_1$ before sampling that are closest to it, and the weighted sum of the features of these $k$ points gives the feature of $x$, where the weight is inversely related to the distance, that is, the farther away, the lower the contribution. The distance metric function here is $d$. The parameters chosen in the paper are $p=2$ and $k=3$.\nThe operation above only gives global-level features; to obtain accurate segmentation results we also need local-level features, so PointNet++ adds skip connections. The operation is fairly simple: take the point cloud features of the corresponding encoder layer, concatenate them with the current layer, and then integrate them through a unit pointnet (a series of MLPs). For details, refer to the PointNet++ network structure figure above.\nFrustum PointNet Detection by combining RGB image information with point cloud information. The main pipeline is: first obtain the object boxes in the 2D image through an object detection model, map the object boxes into the 3D point cloud to obtain frustum proposals, and then use a point cloud based model to perform instance segmentation and 3D bounding box regression on the frustum proposal regions. The rough structure is as follows:\nThe first part in the figure is the 2D object detection network, which gives the object boxes and their classes; the object boxes are then mapped to 3D frustum proposals. The second part is the instance segmentation part: the points in the frustum proposal obtained in the previous part are sampled to $N$ points with dimension $N\\times C$, and together with the previously obtained one-hot class vector they are used as the input of the instance segmentation network, which outputs a mask. In the last part, the $M$ points filtered by the mask from the previous step are used as input, first corrected and aligned by T-Net, and then another network regresses and predicts the 3D bounding box.\nWhen mapping the 2D detection boxes into the 3D point cloud to generate frustum proposals, since each frustum proposal has a different orientation, the paper rotates it to be orthogonal to the camera, which increases the rotation invariance of the subsequent algorithm. As shown from a to b below.\nWhen only the frustum proposal is used, the resulting point cloud contains a large number of background points. So the instance segmentation in the second part separates the foreground from the background, and the segmented foreground points are also centered to introduce translation invariance, as shown in figure c above.\nAlthough the instance segmentation part obtains the center point of the object, the foreground point cloud may not fully contain the object, so the center point still needs to be corrected. The last part uses T-Net to learn the residual of the object center point. Finally Box Estimation Net outputs the various parameters of the final 3D bounding box, with an output dimension of $3+4\\times NS+2\\times NH$, where 3 represents the residual regression for the center point (XYZ), attached after the T-Net regression; $NS$ represents the number of anchors of different sizes, each anchor having four dimensions, namely the confidence of that anchor and the residual regression of length, width and height; $NH$ represents the anchors of different orientations, each anchor having two parameters, namely the confidence and the orientation angle $\\theta$.\nThe detailed structure of the parts of the model after the 2D detection network is roughly as follows: (two versions are included, corresponding to using PointNet and PointNet++ to extract point cloud features)\nIn practice the size and angle of a 3D bounding box jointly determine the box information, so the paper also proposes a corner loss for the bounding box:\n$$ L_{corner} = \\sum_{i=1}^{NS}\\sum_{j=1}^{NH}\\delta_{ij}min\\{\\sum_{k=1}^8\\vert\\vert P_k^{ij}-P_k^\\ast\\vert\\vert, \\sum_{i=1}^8\\vert\\vert P_k^{ij}-P_k^{\\ast\\ast}\\vert\\vert\\} $$Among the $NS\\times NH$ anchors, only the bounding box with the correct size and head orientation incurs the loss (in the remaining cases $\\delta=0$). For the $k$-th corner $P_k^{ij}$ of the box with the $i$-th size and the $j$-th orientation, it is moved to the center of the ground truth box and the L1 loss is computed with each corresponding corner. Considering the 180-degree flip problem, the loss with the corner after a 180-degree rotation, $P^{\\ast\\ast}$, is also computed and the minimum is taken. The overall loss is as follows:\n$$ L_{multi−task} =L_{seg} + \\lambda(L_{c1−reg} + L_{c2−reg} + L_{h−cls} + L_{h−reg} + L_{s−cls} + L_{s−reg} + \\gamma L_{corner}) $$Reference PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation\npointnet code\nPointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space\npointnet++ code\nFrustum PointNets for 3D Object Detection from RGB-D Data\nfrustum-pointnet code\nAn interpretation of the PointNet series papers\n(3D object detection) Frustum PointNets\n","permalink":"https://simplestory.github.io/en/posts/20210807_1c4ff5cd7d69cb58295eab75872ba2fd/","summary":"\u003cblockquote\u003e\n\u003cp\u003ePointNet, PointNet++ and Frustum PointNet. The commonly used 3D point cloud object detection.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"pointnet\"\u003ePointNet\u003c/h2\u003e\n\u003cp\u003ePointNet applies deep learning directly to point cloud data. Point cloud data is a subset of the points in Euclidean space, and has three properties: disorder, the spatial relationship between points, and invariance.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eDisorder. Point cloud data is a set and is insensitive to the order of the data, which requires the model to be invariant to different permutations of the data. There are several approaches. The first is to sort the point cloud by some rule, but for high-dimensional space it is very hard to find a stable sorting rule. Another is to treat it as a sequence and train it with an RNN, but this approach struggles to handle input elements of extremely large length (thousands to tens of thousands, as in a point cloud). Finally the authors use a symmetric function to aggregate the information of each point: $f(\\lbrace x_1,x_2,\\cdots,x_n\\rbrace)\\approx g(h(x_1),h(h(x_2),\\cdots,h(x_n))$, where the left side of the equation is the goal and the right side is the designed symmetric function. From this we can see that each point in the point cloud is processed separately with $h$ and then passed into the symmetric function $g$. In the paper the authors choose $h$ to be an MLP and $g$ to be max pooling.\u003c/li\u003e\n\u003cli\u003eThe spatial relationship between points. An object usually consists of point clouds in a specific space, and there are spatial relationships between points; PointNet exploits these relationships by concatenating global features with local features.\u003c/li\u003e\n\u003cli\u003eInvariance. The object represented by point cloud data should be invariant to some spatial transformations (rigid transformations). PointNet aligns the point cloud data before extracting features to guarantee invariance. The alignment is achieved by training a small network (T-Net) to obtain a transformation matrix and multiplying it with the input point cloud data. This operation can be seen as adjusting the spatial points; intuitively it rotates them into an orientation that is favorable for classification or segmentation (a rigid transformation). In the actual implementation a related loss function is also added to make the transformation matrix approximately orthogonal, so as to reduce the loss of point cloud information. (Regularization: $L_{reg}=\\lVert I-AA^T\\rVert_F^2$, that is, an L2 loss)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThe rough structure of PointNet is as follows:\u003c/p\u003e","title":"The PointNet Series"},{"content":" 3D point cloud detection using three views (bird\u0026rsquo;s eye view, front view, and the original RGB image).\nBird\u0026rsquo;s view The point cloud data is divided equally into $M$ slices; each slice is gridded at a granularity of $0.1m$, and the highest point cloud height within each cell is extracted as the height map of the bird\u0026rsquo;s eye view. The intensity map uses the reflectance value of the point with the greatest height in the cell, and the density map counts the number of point cloud points $N$ in each cell and normalizes the feature as $\\min (1,\\frac{\\log (N+1)}{\\log 64})$, where 64 should be the number of laser beams. The final feature has $M+2$ channels.\nFront\u0026rsquo;s view Considering that the laser point cloud is very sparse, the point cloud is projected onto a cylindrical surface to form the front view. Given a 3D point $p=(x,y,z)$, its corresponding front view coordinates are $p_{fv}=(r,c)$, with:\n$$ \\begin{aligned} r \u0026= \\lfloor\\arctan \\frac{z}{\\sqrt{x^2+y^2}}/\\Delta\\phi\\rfloor \\\\ c \u0026= \\lfloor\\arctan \\frac{y}{x}/\\Delta\\theta\\rfloor \\end{aligned} $$where $\\Delta\\phi$ and $\\Delta\\theta$ denote the resolution in the vertical and horizontal directions, respectively.\n3D prior box Inspired by RPN, the authors also designed a network structure to generate 3D prior boxes. This structure is only used on the bird\u0026rsquo;s eye view, and a prior box is represented by the parameters $(x_{bv},y_{bv},l_{bv},w_{bv})$. Here $l_{bv}$ and $w_{bv}$ can be obtained by clustering the training set, and since objects in the bird\u0026rsquo;s eye view also have a rotation angle, the prior box needs a rotation angle as well; the authors set it to $\\lbrace 0^\\circ,90^\\circ\\rbrace$.\nThe assignment of positive and negative samples to prior boxes mainly refers to the IOU between the prior box and the ground truth box: an IOU greater than 0.7 is taken as a positive sample, below 0.5 as a negative sample, and in between as an ignored sample; to reduce computation, the authors delete all ignored samples. Among the remaining valid prior boxes, NMS with an IOU threshold of 0.7 is used to further reduce the number of boxes; during training only the top 2000 boxes from NMS are kept, while at test time the top 300 are kept.\nThis structure uses cross-entropy loss for the class and $smooth\\ l1$ loss for the position and size. The regression variables for position and size are $t=(\\Delta x,\\Delta y,\\Delta z,\\Delta l,\\Delta w,\\Delta h)$, where $(\\Delta x,\\Delta y,\\Delta z)$ is the center offset, normalized by the corresponding prior box size, and $(\\Delta l,\\Delta w,\\Delta h)$ is computed as: $\\Delta S = \\log \\frac{S_{GT}}{S_{anchor}}$, $S\\in\\lbrace l,w,h\\rbrace$.\nNetwork During the final training, the model corrects the position and size of the box by regressing the corner offsets of the 8 corners of the 3D box, $(\\Delta x_0,\\Delta x_1,\\dots,\\Delta y_0,\\Delta y_1,\\dots,\\Delta z_0,\\Delta z_1,\\dots)$. These offsets are normalized by the diagonal length of the corresponding prior box.\nThe positive/negative prediction result is determined by the IOU between the prior box and the predicted box: greater than 0.5 is regarded as positive, otherwise negative. As for the loss function, following the RPN structure above, the class loss is cross-entropy loss and the regression loss is $smooth\\ l1$ loss. At inference time, NMS with an IOU threshold of 0.05 is applied at the end to remove redundant predicted boxes.\nThe model structure is as follows:\nIn the figure, the features of each view are converted by ROI pooling into feature vectors of the same size, then concatenated and finally fused further through Deep fusion.\nStructural regularization The network structure also uses two methods: drop-path and auxiliary losses.\nIn each iteration, a global drop path or a local drop path is chosen at random. For a global drop path, one of the three views is chosen at random, only the path on that view is kept and the paths of the other views are suppressed; for a local drop path, the path input to each connection node is randomly dropped with a probability of 50%.\nThe auxiliary losses structure is as follows:\nThis structure shares weights with the corresponding network layer and participates in the training loss backpropagation, and is removed at inference time.\nReference Multi-View 3D Object Detection Network for Autonomous Driving\n","permalink":"https://simplestory.github.io/en/posts/20210806_8e5f2ccbcaba22801139e42b929ed820/","summary":"\u003cblockquote\u003e\n\u003cp\u003e3D point cloud detection using three views (bird\u0026rsquo;s eye view, front view, and the original RGB image).\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch3 id=\"birds-view\"\u003eBird\u0026rsquo;s view\u003c/h3\u003e\n\u003cp\u003eThe point cloud data is divided equally into $M$ slices; each slice is gridded at a granularity of $0.1m$, and the highest point cloud height within each cell is extracted as the height map of the bird\u0026rsquo;s eye view. The intensity map uses the reflectance value of the point with the greatest height in the cell, and the density map counts the number of point cloud points $N$ in each cell and normalizes the feature as $\\min (1,\\frac{\\log (N+1)}{\\log 64})$, where 64 should be the number of laser beams. The final feature has $M+2$ channels.\u003c/p\u003e","title":"Multi-View 3D Object Detection Network"},{"content":" The multi-object tracking frameworks commonly used in industry at present.\nSORT This algorithm is based on an object detection network (Faster Rcnn) and makes use of the Kalman filter and the Hungarian algorithm (or the KM algorithm), greatly improving the speed of multi-object tracking. The rough pipeline is as follows:\nThe object detection algorithm obtains the object boxes Detections of the current frame, the Kalman filter obtains the Tracks of the current frame, and IOU matching is performed between the Detections and the Tracks. The final results fall into three categories:\nUnmatched Tracks: that is, Tracks that are partly unmatched. If the mismatch persists for $T_{lost}$ times, the object is deleted. Unmatched Detections: that is, Detections that are partly unmatched. A new Track needs to be assigned to such a Detection. matched Tracks: this part means a match has been found, and is used to update the Kalman filter. The Kalman filter can predict the state of the object boxes of the next frame from the state of the Tracks, and the update uses the observations (the matched Tracks) and the estimates to update the states of all Tracks.\nDeepSORT Compared with SORT, DeepSORT improves the performance of SORT by integrating appearance information and using cascaded matching. This allows the model to better handle objects that are occluded for a long time, greatly reducing the measured ID switch metric. Appearance information refers to the features corresponding to an object; in the paper, a deep association metric trained on a large person re-identification dataset is used to extract appearance features (borrowing models from the ReID field). The overall pipeline is roughly as follows:\nEach track maintains a variable $h$ recording the difference between the current frame and the frame of the last successful match; when a track matches a detection, $h$ is set to 0. When $h$ exceeds a preset threshold $A_{max}$, the track is considered terminated (that is, a track that has not been matched for a long time is considered finished). For detections that were not matched successfully, they may be new tracks or false detections. The paper sets an Unconfirmed state to distinguish them: when the track is matched successfully for $n\\_init$ consecutive frames (set to 3 in the paper), its state is changed to Confirmed; otherwise the track is deleted. The transitions between states are as follows:\nFor matching, the algorithm uses both the appearance model and the motion model to compute the similarity and obtain the cost matrix, and uses a gating matrix to limit overly large values in the cost matrix.\nThe motion model uses the squared Mahalanobis distance between the detection and the track to characterize the degree of motion matching:\n$$ \\begin{aligned} d^{(1)}(i,j) \u0026= (d_j-y_i)^TS_i^{-1}(d_j-y_i) \\\\ b^{(1)}_{i,j} \u0026= I[d^{(1)}(i,j) \\le t^{(1)}] \\end{aligned} $$where $d_j$ denotes the $j$th detection, $y_i$ denotes the $i$th track, and $S_i^{-1}$ represents the covariance of $d$ and $y$. $b_{i,j}^{(1)}$ is an indicator that uses the 0.95 quantile of the chi-square distribution ($t^{(1)}=9.4877$) as the threshold; below the threshold the match is considered successful.\nMahalanobis distance:\nThe Mahalanobis distance represents the covariance distance of data, and is an effective way to compute the similarity between two unknown sample sets. Unlike the Euclidean distance, it takes into account the correlations between the various characteristics of the data.\nFor a multivariate vector $x=(x_1,x_2,\\dots,x_p)^T$ with mean $\\mu=(\\mu_1,\\mu_2,\\dots,\\mu_p)^T$ and covariance matrix $\\Sigma$, its Mahalanobis distance is:\n$$ D_M(x) = \\sqrt{(x-\\mu)^T\\Sigma^{-1}(x-\\mu)} $$The covariance matrix is square, with a dimension matching that of the samples.\nThe appearance model uses cosine distance to measure the distance between appearance features (a feature vector is extracted by the model, and cosine distance is used for comparison):\n$$ \\begin{aligned} d^{(2)}(i,j) \u0026= min\\{1-r_j^Tr_k^{(i)}\\vert r_k^{(i)}\\in R_i\\} \\\\ b^{(2)}_{i,j} \u0026= I[d^{(2)}(i,j) \\le t^{(2)}] \\end{aligned} $$where $r_j^Tr_k^{(i)}$ computes the cosine similarity (cosine distance = 1 - cosine similarity). Matching with motion information alone in SORT leads to fairly serious ID switching; introducing the appearance model and cascaded matching can alleviate this problem. $b_{i,j}^{(2)}$ is an indicator: if the cosine distance is below the threshold the match is considered successful, and this threshold is a hyperparameter (set to 0.2 in the paper). Since a track that is too long causes the appearance to change, continuing to use this minimum cosine distance as a metric carries risk, so the paper only computes the minimum cosine distance over the most recent $L_k=100$ detections of a track.\nThe overall matching cost is obtained by weighting the motion model and the appearance model:\n$$ c_{i,j} = \\lambda d^{(1)}(i,j)+(1-\\lambda)d^{(2)}(i,j) \\tag{5} $$$\\lambda$ is a hyperparameter; considering that the camera may move, the authors set it to 0.\nThe gating matrix is computed as follows:\n$$ b_{i,j} = \\prod_{m=1}^2b^{(m)}_{i,j} \\tag{6} $$$b_{i,j}$ is also an indicator; only when $b_{i,j}=1$ can a preliminary match be made.\nAnother method proposed by DeepSORT to deal with objects occluded for a long time is cascaded matching. When a track has been occluded for a long time, the Kalman filter keeps predicting without being able to update. In this situation, when two tracks compete for the same detection, the track that has been occluded longer often yields a smaller Mahalanobis distance and thus is more likely to match the detection, which interferes with the continuity of tracks. So cascaded matching is used to ensure that each matching round considers tracks with the same occlusion time, and tracks with less occlusion time are considered first. The algorithm is roughly as follows:\nIn the final stage of matching, an IOU-based matching is also performed on the Unconfirmed and unmatched tracks with age=1. This can alleviate large changes caused by appearance jumps or partial occlusion.\nThe appearance feature part borrows a network model from the person re-identification field; this network needs to be trained offline in advance, independently of the object detection and tracker modules, and its function is to extract the feature inside the corresponding bounding box and obtain a fixed-dimensional embedding as the representative of that bbox, used when computing similarities. The paper uses a wide residual network, whose structure is as follows:\nReference SIMPLE ONLINE AND REALTIME TRACKING\nSIMPLE ONLINE AND REALTIME TRACKING WITH A DEEP ASSOCIATION METRIC\nDeep SORT multi-object tracking algorithm code analysis\n","permalink":"https://simplestory.github.io/en/posts/20210805_59094eb979620e0e11f92ffaa71cf228/","summary":"\u003cblockquote\u003e\n\u003cp\u003eThe multi-object tracking frameworks commonly used in industry at present.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"sort\"\u003eSORT\u003c/h2\u003e\n\u003cp\u003eThis algorithm is based on an object detection network (Faster Rcnn) and makes use of the Kalman filter and the Hungarian algorithm (or the KM algorithm), greatly improving the speed of multi-object tracking. The rough pipeline is as follows:\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://simplestory-blog-img.oss-cn-guangzhou.aliyuncs.com/in_posts/20210805/SORT.png\" title=\"SORT\"\u003e\u003c/p\u003e\n\u003cp\u003eThe object detection algorithm obtains the object boxes Detections of the current frame, the Kalman filter obtains the Tracks of the current frame, and IOU matching is performed between the Detections and the Tracks. The final results fall into three categories:\u003c/p\u003e","title":"SORT and DeepSORT"},{"content":" Object detection has produced many algorithms because of the complexity of combining the localization and classification branches. These methods improve algorithm performance from different angles, but are hard to measure in a unified way. This paper uses multiple attention mechanisms to unify different object detection heads: attention over the feature level, over the spatial position and across channels, corresponding respectively to scale awareness, spatial awareness and task awareness. This method can noticeably improve model performance without increasing the computational cost.\nGiven a set of feature maps extracted from $L$ different layers of a feature pyramid (that is, the backbone), $F_{in} = \\lbrace F_i\\rbrace^L_{i=1}$, the feature maps in the set are upsampled or downsampled to an intermediate size to obtain a feature tensor $F\\in R^{L\\times H\\times W\\times C}$, where $L$ denotes the number of extracted layers and $H$, $W$, $C$ denote the height, width and number of channels. Let $S=H\\times W$ and reshape the features to $F\\in R^{L\\times S\\times C}$. These three dimensions correspond to three effects:\nDifferences in object scale correspond to features at different scales; Different sets of transformations for different object shapes correspond to different spatial positions of the features; The variety of object representations and tasks corresponds to different channel features Learning an attention function directly over all dimensions would be very expensive, so the paper converts the attention function into a sequence of attentions over the 3 corresponding dimensions, each of which only attends to its current dimension:\n$$ W(F) = \\pi_C(\\pi_S(\\pi_L(F)\\cdot F)\\cdot F)\\cdot F $$Scale-aware Attention First, attention over scale is introduced to dynamically fuse features from different scales based on semantic importance:\n$$ \\pi_L(F)\\cdot F = \\sigma\\left(f\\left(\\frac{1}{SC}\\sum_{S,C}F\\right)\\right)\\cdot F $$$f(\\cdot)$ is a linear function, approximated here with a $1\\times 1$ convolution layer. $\\sigma(\\cdot)$ is the hard-sigmoid function, as follows:\n$$ \\sigma(x) = \\max (0,\\min (1,\\frac{x+1}{2})) $$Spatial-aware Attention Next, spatial-aware attention is used to improve the model\u0026rsquo;s discriminative ability at different spatial positions. Because the $S$ dimension is large, the paper decouples it: deformable convolution is first used to sparsify the attention learning, and then features are integrated across scales:\n$$ \\pi_S(F)\\cdot F = \\frac{1}{L}\\sum^L_{l=1}\\sum^K_{k=1}w_{l,k}\\cdot F(l;p_k+\\Delta p_k;c)\\cdot \\Delta m_k $$where $K$ is the number of sparse sampling positions and the remaining parameters are similar to those in deformable convolution.\nTask-aware Attention To promote joint learning and the generalization of object representations, the paper also designs a task-aware attention. It can dynamically switch feature channels on and off to assist different tasks:\n$$ \\pi_C(F)\\cdot F = \\max(\\alpha^1(F)\\cdot F_c+\\beta^1(F),\\alpha^2(F)\\cdot F_c+\\beta^2(F)) $$where $[\\alpha^1,\\alpha^2,\\beta^1,\\beta^2]^T = \\theta(\\cdot)$ are hyperparameters that control the activation threshold, and the implementation of $\\theta(\\cdot)$ is similar to dynamic ReLU.\nStructure The attention mechanisms above are used in sequence and can also be stacked several times. The structure figure of DynamicHead is roughly as follows:\nAny type of backbone can be used to extract the feature pyramid and further scale it to a unified size to build the feature tensor, which is then used as the input of DynamicHead and passed through several DyHead modules containing scale-aware, spatial-aware and task-aware attention, stacked in series. The output of DyHead can be used for different tasks, such as classification, center/box regression and so on.\none stage: Single-stage detectors obtain object positions by densely sampling the feature maps, which simplifies the design of the object detector. The integration of DynamicHead with a single-stage detector is shown below:\ntwo stage: Two-stage detectors extract intermediate representations from the feature pyramid through the RPN layer and the ROI pooling layer. To couple this characteristic, the paper uses scale-aware and spatial-aware attention before ROI pooling, and replaces the original fully connected layers with task-aware attention, as follows:\nReference Dynamic Head: Unifying Object Detection Heads with Attentions\nmicrosoft/DynamicHead\n","permalink":"https://simplestory.github.io/en/posts/20210804_697dd2d087a31ba4aab4ff575bde80be/","summary":"\u003cblockquote\u003e\n\u003cp\u003eObject detection has produced many algorithms because of the complexity of combining the localization and classification branches. These methods improve algorithm performance from different angles, but are hard to measure in a unified way. This paper uses multiple attention mechanisms to unify different object detection heads: attention over the feature level, over the spatial position and across channels, corresponding respectively to scale awareness, spatial awareness and task awareness. This method can noticeably improve model performance without increasing the computational cost.\u003c/p\u003e\n\u003c/blockquote\u003e","title":"Dynamic Head"},{"content":" Make it dynamic.\nDynamic Convolution Ordinary static convolution applies the same kernel to all input images, whereas dynamic convolution adapts to different images and processes them with more suitable convolution parameters, which increases the expressive power of the model without increasing network depth or width.\nDynamic convolution dynamically aggregates multiple parallel convolution kernels according to attention. The attention dynamically adjusts the weight of each kernel according to the input, thereby producing an adaptive dynamic convolution. Because the attention is a function of the input, dynamic convolution is no longer a linear function. Superimposing kernels in a nonlinear way through attention gives stronger representational power. The rough structure is as follows:\nAs shown above, compared with ordinary static convolution, dynamic convolution introduces two extra computations: the attention module and the superposition of the kernels. But the computation of these two parts is very small, and superposing multiple kernels is also very efficient computationally. The small extra computation together with the significant gain in expressive power makes dynamic convolution very suitable for lightweight neural networks.\nRestricting the attention values simplifies the learning of the attention model and also narrows the value space of the superposed kernel. The paper restricts the attention values to between 0 and 1 with all attentions summing to 1. As shown below, if 3 kernels are used, restricting the attention to between 0 and 1 confines the superposed kernel to the two triangular pyramids, and having the attentions sum to 1 further confines the superposed kernel to the triangle with these three kernels as vertices. For these two constraints the paper uses softmax.\nIn addition, keeping the attention values close to a uniform distribution helps multiple kernels learn simultaneously in the early stage of training. Because softmax outputs sparser attention, the paper uses temperature annealing to improve model accuracy (attention close to a uniform distribution can be achieved by using a larger temperature).\nDynamic ReLU Usually the parameters of ReLU and of some of its improved versions (Leaky ReLU, PReLU) are fixed, so the paper proposes dynamic ReLU. It can adjust the parameters of ReLU according to the input features.\nThe original ReLU is $y = \\max \\lbrace x,0\\rbrace$; for the $c$-th feature of the input vector $x$, the activation value is computed as $y_c = \\max \\lbrace x_c, 0\\rbrace$. ReLU can be written uniformly as the piecewise linear function $y_c = \\max\\limits_k \\lbrace a^k_cx_c+b^k_c\\rbrace$. Based on the formula above, DY-ReLU can be defined as the maximum of several ($K$) linear functions. That is, for all inputs $x=\\lbrace x_c\\rbrace$ the adaptive $a^k_c$, $b^k_c$ satisfy:\n$$ \\begin{aligned} y_c \u0026= f_{\\theta(x)}(x_c) \\\\ \u0026= \\max\\limits_{1\\le k\\le K}\\{a^k_c(x)x_c+b^k_c(x)\\} \\end{aligned} $$The parameters $(a^k_c,b^k_c)$ are the output of the hyper function $\\theta(x)$:\n$$ \\theta(x) = [a^1_1,\\dots,a^1_C,\\dots,a^K_1,\\dots,a^K_C,b^1_1,\\dots,b^1_C,\\dots,b^K_1\\dots,b^K_C]^T $$where $C$ is the number of channels, and the activation parameters $(a^k_c, b^k_c)$ are not only related to $x_c$ but also to $x_{j\\ne c}$, that is, they are also related to the inputs of other channels.\nThe paper implements the hyper function with a lightweight network similar to the SE module. For an input $x$ of size $C\\times H\\times W$, global average pooling is first used to compress the spatial information, then two fully connected layers are applied (with ReLU in between), and finally a normalization layer is applied, which uses $2\\sigma(x)-1$ to constrain the result between -1 and 1 ($\\sigma$ is the Sigmoid function). The final output consists of $2KC$ elements, corresponding to the residuals of $a_{1:C}^{1:K}$ and $b_{1:C}^{1:K}$, that is, $\\Delta a_{1:C}^{1:K}$ and $\\Delta b_{1:C}^{1:K}$. The final output is the sum of the initial values and the residuals:\n$$ \\begin{aligned} a^k_c(x) \u0026= \\alpha^k+\\lambda_a\\Delta a^k_c(x) \\\\ b^k_c(x) \u0026= \\beta^k+\\lambda_b\\Delta b^k_c(x) \\end{aligned} $$where $\\alpha^k$ and $\\beta^k$ are the initial values of $a^k_c$ and $b^k_c$. $\\lambda_a$ and $\\lambda_b$ are scalars used to control the magnitude of the residuals. $\\alpha^k$, $\\beta^k$, $\\lambda_a$ and $\\lambda_b$ are hyperparameters independent of the channel; by default $\\lambda_a=1.0$ and $\\lambda_b=0.5$, so the default hyperparameters are $\\alpha^{1:K}$ and $\\beta^{1:K}$.\nThe paper also gives three implementations of DY-ReLU, roughly as follows:\nDY-ReLU-A Spatial and channel-shared. The final output size is $2K$; the computation is relatively simple and the expressive power is also weaker.\nDY-ReLU-B Spatial-shared and channel-wise. The final output size is $2KC$, that is, $2K$ per channel.\nDY-ReLU-C Spatial and channel-wise. Every element of every channel has an independent activation function, so the final output has $2KHWC$ parameters, and using a fully connected layer directly would bring a lot of extra computation. The paper improves on this as shown in the figure above: the spatial position is decoupled into another attention branch, and finally the channel parameters $a_{1:C}^{1:K}$ and $b_{1:C}^{1:K}$ are multiplied by the spatial position attention $\\pi_{1:HW}$. The spatial position attention is computed simply with $1\\times 1$ convolution and a normalization method, using a constrained softmax function:\n$$ \\pi_{h,w} = \\min \\{\\frac{\\gamma\\exp (z_{h,w}/\\tau)}{\\sum_{h,w}\\exp (z_{h,w}/\\tau)}, 1\\} $$where $\\gamma$ is a scalar used to amplify the softmax values and prevent vanishing gradients; the paper sets it to $\\frac{HW}{3}$. $z_{h,w}$ is the output of the $1\\times 1$ convolution. $\\tau$ is the temperature, and a large temperature (such as $\\tau$ = 10) can be used to prevent the spatial position attention from being too sparse in the early stage of training.\nReference Dynamic Convolution: Attention over Convolution Kernels\nDynamic ReLU\n","permalink":"https://simplestory.github.io/en/posts/20210803_2d8c6e5a2f995c3808422afc8967c85e/","summary":"\u003cblockquote\u003e\n\u003cp\u003eMake it dynamic.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"dynamic-convolution\"\u003eDynamic Convolution\u003c/h2\u003e\n\u003cp\u003eOrdinary static convolution applies the same kernel to all input images, whereas dynamic convolution adapts to different images and processes them with more suitable convolution parameters, which increases the expressive power of the model without increasing network depth or width.\u003c/p\u003e\n\u003cp\u003eDynamic convolution dynamically aggregates multiple parallel convolution kernels according to attention. The attention dynamically adjusts the weight of each kernel according to the input, thereby producing an adaptive dynamic convolution. Because the attention is a function of the input, dynamic convolution is no longer a linear function. Superimposing kernels in a nonlinear way through attention gives stronger representational power. The rough structure is as follows:\u003c/p\u003e","title":"Dynamic Convolution and Dynamic ReLU"},{"content":" Mathematics is the queen of the sciences. — Gauss\nProbabilistic undirected graphical model Before understanding the probabilistic undirected graphical model, let us look at the definition of the probabilistic graphical model (PGM): it is a probability distribution represented by a graph. Suppose there is a joint probability distribution $P(Y)$, where $Y\\in \\mathcal{Y}$ is a group of random variables, and the probability distribution $P(Y)$ is represented by an undirected graph $G=(V, E)$; that is, in graph $G$, a node $v\\in V$ represents a random variable $Y_v$, $Y_v=(Y_v)_{v\\in V}$; an edge $e\\in E$ represents the probabilistic dependency between random variables.\nGiven a joint probability distribution $P(Y)$ and the undirected graph $G$ representing it. Three properties are explained below.\nPairwise Markov property\n$$ P(Y_u,Y_v\\mid Y_O) = P(Y_u\\mid Y_O)P(Y_v\\mid Y_O) $$where $u$ and $v$ are any two nodes in the undirected graph not connected by an edge, with corresponding random variables $Y_u$ and $Y_v$, and $O$ is all the other nodes in the undirected graph besides $u$ and $v$, with the corresponding group of random variables $Y_O$. That is, given the group of random variables $Y_O$, the random variables $Y_u$ and $Y_v$ are conditionally independent.\nLocal Markov property\n$$ P(Y_u,Y_O\\mid Y_W) = P(Y_v\\mid Y_W)P(Y_O\\mid Y_W) $$where $v\\in V$ is any node in the undirected graph $G$ with corresponding random variable $Y_v$, $W$ are all the nodes connected to $v$ by edges with the corresponding group of random variables $Y_W$, and $O$ are all the other nodes besides $v$ and $W$, with the corresponding group of random variables $Y_O$. That is, given the group of random variables $Y_W$, the random variable $Y_v$ is independent of the group of random variables $Y_O$.\nGlobal Markov property\n$$ P(Y_A,Y_B\\mid Y_C) = P(Y_A\\mid Y_C)P(Y_B\\mid Y_C) $$where the node sets $A$ and $B$ are any node sets in the undirected graph $G$ separated by the node set $C$, with corresponding groups of random variables $Y_A$, $Y_B$ and $Y_C$. That is, given the group of random variables $Y_C$, the groups of random variables $Y_A$ and $Y_B$ are conditionally independent.\nThe three definitions above are equivalent.\nCombining the above, here is the definition of a probabilistic undirected graphical model. Suppose there is a joint probability distribution $P(Y)$ represented by an undirected graph $G=(V,E)$; in graph $G$, nodes represent random variables and edges represent the dependencies between random variables. If the joint probability distribution $P(Y)$ satisfies the pairwise, local or global Markov property, this joint probability distribution is called a probabilistic undirected graphical model, or a Markov random field (MRF).\nFactorization The core of a probabilistic graphical model is computing the joint probability distribution\nA subset of nodes in an undirected graph $G$ in which every two nodes are connected by an edge is called a clique. If $C$ is a clique of the undirected graph $G$ and no other node of $G$ can be added to it to make it a larger clique, then this $C$ is called a maximal clique.\nThe operation of expressing the joint probability distribution of a probabilistic undirected graphical model as the product of functions of the random variables on its maximal cliques is called the factorization of the probabilistic undirected graphical model.\nGiven a probabilistic undirected graphical model, let its undirected graph be $G$, let $C$ be a maximal clique of $G$, and let $Y_c$ denote the random variable corresponding to $C$. By the Hammersley-Clifford theorem we have that, if and only if the joint probability density of the random variables is strictly positive, the probabilistic undirected graphical model is equivalent to a Gibbs random field (GRF). And the joint probability distribution $P(Y)$ of the GRF can be written as the product of functions $\\Psi_C(Y_C)$ over all maximal cliques $C$ of the graph, namely:\n$$ \\begin{aligned} P(Y) \u0026 = \\frac{1}{Z}\\prod_C\\Psi_C(Y_C) \\\\ Z \u0026 = \\sum_Y\\prod_C\\Psi_C(Y_C) \\end{aligned} $$where $Z$ is the normalization factor, used to ensure that $P(Y)$ forms a probability distribution, that is, sums to 1. The function $\\Psi_C(Y_C)$ is called the potential function. Here the potential function is required to be strictly positive; for convenience it is usually defined as an exponential function:\n$$ \\Psi_C(Y_C) = exp\\{-E(Y_C)\\} $$Conditional random field Definition: Let $X$ and $Y$ be random variables and $P(X\\mid Y)$ be the conditional probability distribution of $Y$ given $X$. If the random variable $Y$ constitutes a Markov random field represented by an undirected graph $G=(V,E)$, that is,\n$$ P(Y_v\\mid X,Y_w,w\\ne v) = P(Y_v\\mid X,Y_w,w\\sim v) $$holds for any node $v$, then $P(Y\\mid X)$ is called a conditional random field. Here $w\\sim v$ denotes all nodes $w$ in graph $G=(V,E)$ connected to node $v$ by edges, $w\\ne v$ denotes all nodes other than node $v$, and $Y_v$, $Y_u$ and $Y_w$ are the random variables corresponding to nodes $v$, $u$ and $w$.\n$X$ and $Y$ can have different graph structures, but in general they are assumed to have the same graph structure, which gives the linear chain conditional random field. Let $X=(X_1,X_2,\\cdots,X_n)$ and $Y=(Y_1,Y_2,\\cdots,Y_n)$ both be random variable sequences represented by linear chains; if, given the random variable sequence $X$, the conditional probability distribution $P(Y\\mid X)$ of the random variable sequence $Y$ constitutes a conditional random field, that is, it satisfies the Markov property\n$$ P(Y_i\\mid X,Y_1,\\cdots,Y_{i-1},Y_{i+1},\\cdots,Y_n) = P(Y_i\\mid X,Y_{i-1},Y_{i+1}) \\\\ i=1,2,\\cdots,n \\ (only one side is considered when i=1 and n) $$then $P(Y\\mid X)$ is called a linear chain conditional random field, which is also a log linear model.\nParameterized form of the linear chain conditional random field For the linear chain conditional random field $P(Y\\mid X)$, given that the random variable $X$ takes the value $x$, the conditional probability of the random variable $Y$ taking the value $y$ has the following parameterized form:\n$$ P(y\\mid x) = \\frac{1}{Z(x)}exp\\left(\\sum_{i,k}\\lambda_kt_k(y_{i-1},y_i,x,i)+\\sum_{i,l}\\mu_ls_l(y_i,x,i)\\right) $$where\n$$ Z(x) = \\sum_yexp\\left(\\sum_{i,k}\\lambda_kt_k(y_{i-1},y_i,x,i)+\\sum_{i,l}\\mu_ls_l(y_i,x,i)\\right) $$In the formulas, $t_k$ and $s_l$ are feature functions, $\\lambda_k$ and $\\mu_l$ are the corresponding weights, and $Z(x)$ is the normalization factor; the summation runs over all possible output sequences. $t_k$ is a feature function defined on edges, called a transition feature, which depends on the current and the previous position; $s_l$ is a feature function defined on node $l$, called a state feature, which depends on the current position. Both feature functions depend on the position and are local feature functions, and they generally take the value 1 or 0, being 1 when the condition is met and 0 otherwise.\nSimplified form of the linear chain conditional random field Suppose there are $K_1$ transition features and $K_2$ state features, with $K=K_1+K_2$; then\n$$ \\begin{aligned} \u0026 f_k(y_{i-1},y_i,x,i) = \\begin{cases} t_k(y_{i-1},y_i,x,i), \u0026\\ k=1,2,\\cdots, K_1 \\\\ s_l(y_i,x,i), \u0026\\ k=K_1+l;\\ l=1,2,\\cdots, K_2 \\end{cases} \\\\ \u0026 \\Rightarrow f_k(y,x) = \\sum_{i=1}^nf_k(y_{i-1},y_i,x,i), \\qquad k=1,2,\\cdots,K \\end{aligned} $$Likewise, $w_k$ is used to express the weights uniformly:\n$$ w_k = \\begin{cases} \\lambda_k, \u0026\\ k=1,2,\\cdots,K_1 \\\\ \\mu_l, \u0026\\ k=K_1+l;\\ l=1,2,\\cdots,K \\end{cases} $$In summary we obtain:\n$$ P(y\\mid x) = \\frac{1}{Z(x)}exp\\sum_{k=1}^{K}w_kf_k(y,x) \\\\ Z(x) = \\sum_yexp\\sum_{k=1}^Kw_kf_k(y,x) $$If $w=(w_1,w_2,\\cdots,w_K)^T$ denotes the weight vector and $F(y,x)=(f_1(y,x),f_2(y,x),\\cdots,f_K(y,x))^T$ denotes the global feature vector, then the formula above can be expressed further as follows:\n$$ P_w(y\\mid x) = \\frac{exp(w\\cdot F(y,x))}{Z_w(x)} \\\\ Z_w(x) = \\sum_yexp(w\\cdot F(y,x)) $$Matrix form of the linear chain conditional random field For each position $i=1,2,\\cdots,n+1$ of the input sequence $x$, define an $m$-order matrix ($m$ being the number of values $y_i$ can take):\n$$ M_i(x) = [M_i(y_{i-1},y_i\\mid x)] \\\\ M_i(y_{i-1},y_i\\mid x) = exp(W_i(y_{i-1},y_i\\mid x)) \\\\ W_i(y_{i-1},y_i\\mid x) = \\sum_{k=1}^Kw_kf_k(y_{i-1},y_i,x,i) $$Combining the simplified form above with the $m$-order matrix here gives\n$$ \\begin{aligned} \u0026 exp\\sum_{k=1}^Kw_k\\sum_{i=1}^nf_k(y_{i-1},y_i,x,i) \\\\ = \u0026 exp\\sum_{k=1}^K\\sum_{i=1}^nw_kf_k(y_{i-1},y_i,x,i) \\\\ = \u0026 exp\\sum_{i=1}^n\\sum_{k=1}^Kw_kf_k(y_{i-1},y_i,x,i) \\\\ = \u0026 exp\\sum_{i=1}^nW_i(y_{i-1},y_i\\mid x) \\\\ = \u0026 \\prod_{i=1}^nexp(W_i(y_{i-1},y_i\\mid x)) \\\\ = \u0026 \\prod_{i=1}^nM_i(y_{i-1},y_i\\mid x) \\\\ = \u0026 \\prod_{i=1}^{n+1}M_i(y_{i-1},y_i\\mid x) \\qquad (adding the start state and the stop state) \\end{aligned} $$In the last step of the derivation above, $y_0=start$ and $y_{n+1}=stop$ denote the start state and the stop state respectively. This gives:\n$$ P_w(y\\mid x) = \\frac{1}{Z(x)}\\prod_{i=1}^{n+1}M_i(y_{i-1},y_i\\mid x) \\\\ Z_w(x) = (M_1(x)M_2(x)\\cdots M_{n+1}(x))_{start,stop} $$The normalization factor $Z_w(x)$ is the sum of the unnormalized probabilities $\\prod_{i=1}^{n+1}M_i(y_{i-1},y_i\\mid x)$ of all paths $y_1y_2\\cdots y_n$ through the states from start to stop.\nSome applications CRF is suited to situations where there are dependencies between contextually related samples. It has a great many applications in the NLP field; for the CV field, limited as my knowledge is, the applications I have seen so far are all in the image segmentation field. Segmentation is pixel-level, so every pixel in the image is a node $x$ and the corresponding label is $y$; the relationships between pixels can be represented as edges. Since the surrounding pixels have a much greater influence on the label than other pixels, we can infer the label of the target pixel from the labels of the surrounding pixels, and this whole setup forms a conditional random field.\nDenseCRF DenseCRF is a fully connected conditional random field; what differs from an ordinary conditional random field is that the pairwise potential function describes the relationship between every pixel and all other pixels.\nFrom the earlier definition we know that the potential function of a CRF is $\\Phi_c(X_c)=e^{-E(X_c)}$ (other functions can be chosen, but $\\Phi_c(X_c) \u003e 0$ must always hold), where $E(X_c)$ is the energy function; for the segmentation field the energy function is generally expressed as follows:\n$$ E(X) = \\sum_i\\phi_u(x_i)+\\sum_{i","permalink":"https://simplestory.github.io/en/posts/20210516_5df2f4dc23f490c5d68321fa51cabc59/","summary":"\u003cblockquote\u003e\n\u003cp\u003eMathematics is the queen of the sciences. — Gauss\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"probabilistic-undirected-graphical-model\"\u003eProbabilistic undirected graphical model\u003c/h2\u003e\n\u003cp\u003eBefore understanding the probabilistic undirected graphical model, let us look at the definition of the probabilistic graphical model (PGM): it is a probability distribution represented by a graph. Suppose there is a joint probability distribution $P(Y)$, where $Y\\in \\mathcal{Y}$ is a group of random variables, and the probability distribution $P(Y)$ is represented by an undirected graph $G=(V, E)$; that is, in graph $G$, a node $v\\in V$ represents a random variable $Y_v$, $Y_v=(Y_v)_{v\\in V}$; an edge $e\\in E$ represents the probabilistic dependency between random variables.\u003c/p\u003e","title":"CRF and Its Application in Segmentation"},{"content":" Image segmentation is pixel-level classification of an image, and mainly includes semantic segmentation and instance segmentation. Semantic segmentation only requires segmentation at the pixel level, while instance segmentation additionally requires separating different individuals on top of semantic segmentation. This post briefly introduces several basic semantic segmentation networks.\nFCN FCN (Full Convolution Network) is a classic image segmentation network whose main structure is an encoder-decoder, where the encoder extracts image feature information and the decoder interprets the extracted features. FCN replaces the fully connected layers in the network with convolution layers, so that 2D feature maps are obtained; these are then upsampled through deconvolution (transposed convolution), and finally a softmax layer produces the mask for each class. It can also handle input images of different sizes. The rough structure is shown below.\nThe structure figure above contains 3 FCN models, namely FCN-8s, FCN-16s and FCN-32s.\nFCN-32s segments the input image directly by 32x downsampling followed by 32x upsampling. FCN-16s performs 2x upsampling, adds the result element-wise (Eltwise) to the 16x downsampled feature map of the previous layer, and then performs 16x upsampling for segmentation. FCN-8s performs 2x upsampling after the previous Eltwise operation, applies Eltwise with the 8x downsampled feature map, and then performs 8x upsampling for segmentation.\nThe paper\u0026rsquo;s comparison shows that FCN-8s performs better than the other two models, that is, fusing feature maps from multiple layers helps improve segmentation accuracy.\nSegNet From the FCN structure above we can see that in the decoder stage the model uses only a single deconvolution layer for upsampling, which is very coarse. SegNet also has an encoder-decoder structure, but the two parts are symmetric: the encoder follows the Vgg-16 network for feature extraction, while the decoder uses unpooling to upsample the feature maps and then convolves the upsampled images, with the aim of refining the geometric shape of objects and making up for the detail lost when the pooling layers in the encoder shrink the objects. The upsampling also makes use of pooling indices, that is, the weight indices saved during the pooling operations in the encoder, so that during upsampling the data is placed directly at the index positions and then passed through convolution layers for learning. The structure is as follows.\nDeconvNet This network is similar to SegNet in that it also uses a symmetric encoder-decoder structure and also uses pooling indices for upsampling, but unlike SegNet the model uses fully connected layers between the encoder and the decoder. To obtain instance-level segmentation results, the trained model can be applied to the proposals of an image and the results finally combined into the final segmentation, which also solves the size problem of the original FCN network and gives more detailed object information. The rough structure of the model is as follows.\nUNet The structure of UNet is also a symmetric encoder-decoder. The structure is shown first:\nFrom the structure figure we can see that each upsampling step (this model uses deconvolution for upsampling) is followed by channel concatenation with the corresponding part of the encoder, and before concatenation the feature map of the corresponding encoder part must be cropped; the cropping keeps the central part of the image and cuts away the border so that the image size matches the corresponding decoder part. In the end a feature map with 2 channels is obtained, and to further obtain a probability map a softmax is applied to the feature map.\nSummary For deep learning based semantic segmentation, a general framework is roughly as follows:\nFront end: feature extraction with a fully convolutional network Downsampling + upsampling: Convlution + Deconvlution/Resize Multi-scale feature fusion: element-wise addition of features, feature concatenation Obtaining a pixel-level segement map: determining the class of each pixel Back end: using CRF (conditional random field) / MRF (Markov random field) to refine the output of the front end into an optimized segmentation map Reference Fully Convolutional Networks for Semantic Segmentation\nSegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation\nLearning Deconvolution Network for Semantic Segmentation\nU-Net: Convolutional Networks for Biomedical Image Segmentation\n","permalink":"https://simplestory.github.io/en/posts/20210430_b30ab79aff271e301dde9ca18c2e8085/","summary":"\u003cblockquote\u003e\n\u003cp\u003eImage segmentation is pixel-level classification of an image, and mainly includes semantic segmentation and instance segmentation. Semantic segmentation only requires segmentation at the pixel level, while instance segmentation additionally requires separating different individuals on top of semantic segmentation. This post briefly introduces several basic semantic segmentation networks.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch3 id=\"fcn\"\u003eFCN\u003c/h3\u003e\n\u003cp\u003eFCN (Full Convolution Network) is a classic image segmentation network whose main structure is an encoder-decoder, where the encoder extracts image feature information and the decoder interprets the extracted features. FCN replaces the fully connected layers in the network with convolution layers, so that 2D feature maps are obtained; these are then upsampled through deconvolution (transposed convolution), and finally a softmax layer produces the mask for each class. It can also handle input images of different sizes. The rough structure is shown below.\u003c/p\u003e","title":"Basic Networks for Image Segmentation"},{"content":" Object detection is an important branch of the computer vision field, and its essential content is: What objects are where? At the same time, as a basic task it supports some more complex problems in the field (such as instance segmentation, image text recognition, object tracking and so on). From the application point of view, object detection can be divided into two major categories: general object detection and detection applications. General object detection mainly explores methods for detecting different types of objects within a unified algorithmic framework, similar to human visual perception; detection applications focus on actual scenarios, such as pedestrian detection, face detection and text detection.\nMap The object detection field has developed for about 20 years since 1999. The development path of the related algorithm models during this period is shown in the figure below (taken from the original paper):\nThis post follows that path and records the key contributions of each algorithm along the course of development.\nTraditional Detection Methods In the earlier stage of development, the algorithms in this field were mainly traditional algorithms, that is, you had to design the relevant features yourself for machine learning.\nViola-Jones Detectors In 2001, P. Viola and M. Jones published a classic object detection algorithm (the Viola-Jones Detector). This algorithm uses Haar-like features as input for detection on top of Adaboost, and also uses a sliding window to determine whether some region contains an object. The introduction of the sliding window adds a huge computational burden, so the authors adopted several methods to speed up the algorithm:\nIntegral image: Haar features consider adjacent rectangular regions at a specific position, adding up the pixels of each rectangular region and then subtracting them (example). To compute Haar-like features, all pixels in the rectangular region need to be summed, and the rectangular regions that an image can form vary in size; if every rectangular region required traversing all its pixels and then summing, that would be very slow. The integral image means that for any point in the image, the integral image value at that point equals the sum of all pixels to the upper left of that point, with the formula: $$ I(x,y) = \\sum_{x'\u003c x}\\sum_{y' \u003c y}f(x',y') $$The integral image has the following relation:\n$$ I(x,y) = f(x,y)+I(x,y-1)+I(x-1,y)-I(x-1,y-1) $$Here $I$ denotes the integral image and $f$ denotes the original image. With the integral image, the sum of pixels of any rectangular region in the image can be computed:\n$$ S_{abcd} = I(d)-I(b)-I(c)+I(a) $$ Feature selection: the number of Haar features extracted from an image is fairly large; the authors assume that only a small subset of these features is useful and use Adaboost for feature selection, reducing the computational burden by reducing the number of features. Detection cascades: to further reduce the computation, the authors do not use a traditional Adaboost classifier but cascade multiple small Adaboosts, and once some level of the cascade detects that a region is not an object, the detection of that region stops immediately. HOG Detectors In 2005, N. Dalal and B. Triggs proposed the HOG (Histogram of Oriented Gradients) feature extraction algorithm, which forms features by computing and collecting histograms of gradient orientations over local regions of the image. This is done because the appearance and shape of local objects in an image can be well described by the distribution of gradient or edge direction densities. Specifically, the grayscale image is divided into blocks, the gradient or edge orientation histograms of the pixels in each block are computed, and finally these histograms are combined to form the features.\nThe figure above shows the concrete steps a typical HOG detector performs. Gamma correction is used here to adjust the contrast of the image, reduce the influence of local shadows and illumination changes, and also suppress noise interference. Because of local illumination changes and changes in foreground-background contrast, the range of gradient magnitudes varies enormously. This requires normalizing the gradient magnitudes, and normalization can further compress the effects of illumination, shadows and edges. The authors perform the normalization by combining the features of several cells.\nDeformable Part-based Model (DPM) In 2008, P. Felzenszwalb proposed the Deformable Part Model, a component-based detection algorithm that is an extension of the HOG detector. A typical DPM model consists of a root filter and several part filters, where the root filter covers the whole object and the part filters are high-resolution templates covering parts of the object. For an object, the model formally defines it as a tuple: $(F_0,P_0,P_1,...,P_n,b)$, where $F_0$ is the root filter, $P_i$ is the model of the $i$-th part, and b represents the bias value. Each part model is represented by a triple $(F_I, v_i, d_i)$: $F_i$ is the model of the $i$-th part, $v_i$ is a two-dimensional vector specifying the anchor position of the $i$-th filter (the undeformed standard position) relative to the root coordinates, and $d_i$ is a four-dimensional vector specifying the parameters of a quadratic function that represents the deformation cost of each possible position of the part relative to the anchor position.\nThe figure above visualizes the root model, the part models and the part deformation costs. Taking a pedestrian as an example here, the left image is the root model, which roughly represents an upright pedestrian; the middle images are the part models, each of which is the part inside the rectangular box at twice the resolution of the root model, which gives better results; the right images are the deformation costs of the part models, where brighter regions indicate larger deformation cost, and the deformation cost at the standard position of the part model is 0. To reduce the complexity of the model, both the root model and the part models are axially symmetric.\nDPM uses the traditional sliding window detection approach, searching at various scales by building an image pyramid. The figure below shows the pedestrian detection process at a certain scale, that is, the matching process of the pedestrian model. The response score of a certain position with the root or part model is the inner product of that model with the features in the sub-window region anchored at that position (that is, with the top-left corner coordinates). The model can also be regarded as a filter operator, and the response score is how similar the features are to the model being matched — the more similar, the higher the score. The left side shows the detection process of the root model; in the filtered image, brighter regions represent higher response scores. The right side shows the detection process of each part model. First the feature image is matched with the model to obtain the filtered image. Then response transformation is performed. The response transformation takes the anchor position as the reference position, combines the degree to which the part model matches the features and the deformation cost of the part model relative to the standard position, and obtains the optimal part model position and response score.\nDeep Learning Detection Methods Since the AlexNet neural network achieved SOTA on Imagenet in 2012, neural networks began to receive attention. Here I introduce several key algorithms.\nRCNN In 2014, R. Girshick et al. published RCNN, the first to introduce neural network models into the object detection field. RCNN obtains a large number of candidate boxes from the original image through selective search, extracts image features for each candidate box using a convolutional neural network, then applies a linear SVM classifier to those image features, and finally obtains the class and position/size of the objects in the image.\nOverall the model can be divided into three parts:\nGenerating class-independent candidate regions A convolutional network used to extract a fixed-length feature vector from each candidate region A set of linear SVM classifiers SPPNet RCNN needs to resize each candidate region to a fixed size and then extract convolutional features for each region. These two steps bring some performance bottlenecks. To solve these bottlenecks, K. He et al. proposed the SPPNet model in 2014. The authors add a pooling layer before the last fully connected layer of the feature extraction network to produce a fixed output for arbitrary input, thereby solving the problem in RCNN that candidate region inputs must be of fixed size. The model also borrows the idea of the spatial pyramid, that is, dividing an image into blocks of different scales, extracting features for each block and finally concatenating them.\nThe figure below shows the difference between SPPNet and RCNN when handling input images of different sizes. RCNN obtains images of a fixed size by cropping or resizing and then feeds them into the convolutional neural network for computation. SPPNet, by contrast, first extracts features through a convolutional network and then obtains a fixed-size feature vector through a pooling layer as the input of the fully connected layers. The most important point is that features only need to be extracted once per image.\nThe main purpose of the spatial pyramid pooling layer is to produce a fixed-size output for input of any size. The idea is to first divide a feature map of any size into 16, 4 and 1 blocks, then perform max pooling on each block, and concatenate the pooled features to obtain an output of fixed dimension, satisfying the needs of the fully connected layers.\nFast RCNN Like RCNN, SPPNet also needs to train a CNN to extract features and then train SVMs to classify those features. This requires huge storage space, and training the parts separately is also complicated. In 2015 R. Girshick introduced the Fast RCNN model, which unifies the detector and the box regressor within one framework for unified training. It takes the whole image and a set of candidate boxes as input; the model first extracts features from the whole image with a set of convolution layers and max pooling layers, then, on the extracted feature map, uses RoI pooling to extract a fixed-length feature vector for each candidate box, and then feeds it into fully connected layers that finally split into two branches for object recognition and bounding box regression. Although Fast RCNN is a big improvement over RCNN and SPPNet, the model still uses selective search to generate candidate boxes, which is the main speed bottleneck of the model.\nFaster RCNN RCNN, SPPNet and Fast RCNN all use selective search to obtain object candidate boxes, and this operation slows down the whole model. In 2015 S. Ren et al. proposed Faster RCNN, the first end-to-end deep learning object detection model running at close to real time. Its main contribution is the RPN layer. The RPN layer can generate candidate regions quickly, and it also introduces anchor boxes to select candidate regions. The rough structure is as follows, where the red box denotes the RPN layer (figure taken from the internet)\nFaster RCNN abandons the traditional approach of generating candidate boxes with sliding windows and selective search and directly uses RPN to generate detection boxes. This is a huge advantage of Faster RCNN, as it can greatly speed up the generation of detection boxes; it is also from here that object detection introduced Anchors as object candidate boxes.\nThe generation of Anchors is mainly determined by two parameters: the aspect ratio and the scaling scale. For example, the figure on the right shows the nine anchor boxes Faster RCNN generates under three aspect ratios (ratios=[0.5, 1.0, 2.0]) and three scales (scales=[8, 16, 32]).\nThe anchor boxes are distributed over every pixel of the feature map, that is, the center of an anchor box is at the position of the feature map pixel. The resulting anchor boxes cover the whole feature map, and the anchor boxes under the same pixel have different scales, so in fact the anchor boxes introduce the multi-scale method commonly used in detection.\nThe rough computation is as follows:\n$$ \\begin{cases} w\\times h = s \\\\ \\frac{w}{h} = ratio \\end{cases} \\Rightarrow \\begin{cases} h = \\sqrt{\\frac{s}{ratio}} \\\\ w = ratio\\cdot h = \\sqrt{s\\cdot ratio} \\end{cases} $$and further\n$$ \\begin{cases} h = \\sqrt{\\frac{s}{ratio}\\cdot scale} \\\\ w = ratio\\cdot h = \\sqrt{s\\cdot ratio \\cdot scale} \\end{cases} $$Generating anchor boxes on a feature map looks roughly like the figure below (taken from the internet):\nAs for the RPN layer, the figure below shows the concrete structure of the RPN network. As can be seen, the RPN network is actually split into 2 lines: the upper line classifies the anchors with softmax to obtain positive and negative classes, and the lower line computes the bounding box regression offsets for the anchors to obtain accurate proposals. The final Proposal layer is responsible for combining the positive anchors and the corresponding bounding box regression offsets to obtain proposals, while removing proposals that are too small or beyond the boundary. In fact, once the whole network reaches the Proposal Layer, it has accomplished the equivalent of object localization.\nRough RPN flow:\ngenerate anchors -\u0026gt; softmax classifier extracts positive anchors -\u0026gt; bbox reg regresses positive anchors -\u0026gt; Proposal Layer generates proposals\nFeature Pyramid Networks (FPN) In 2017, T.-Y. Lin et al. proposed FPN on the basis of Faster RCNN. Previous neural network object detection models only extracted the features output by the topmost layer for detection, but the deeper layers of a convolutional network are good for class recognition while being bad for object localization. FPN contains a top-down network structure with lateral connections, which can be used to build high-level semantics at various scales. Basically all the various networks proposed afterwards include this structure.\nIn the top-down structure of FPN, the detailed operations between network layers are as follows:\nYou Only Look Once (YOLO) In 2015, R. Joseph proposed YOLO, the first single-stage model in the deep learning object detection field. This model divides the image into $S\\times S$ grids: if the center of an object falls inside a grid cell, that grid cell is responsible for predicting the object. Each grid cell predicts $B$ bounding boxes along with the confidence of these boxes, where confidence means whether there is an object in the box and the confidence in that judgment: $Pr(Object)*IOU^{truth}_{pred}$. If there is no object in the box then $Pr(Object)$ is 0, otherwise the $IOU$ between the predicted box and the ground truth box is used as the confidence value.\nThe authors unify the object detection pipeline into a single neural network. This neural network uses the information of the whole image to predict the bounding boxes of objects while recognizing their classes, achieving real-time end-to-end object detection. Each bounding box predicts five variables: x, y, w, h, conf, where the first four are coordinate variables, with x and y being the center of the box and w and h denoting the width and length. Note here that the center coordinates are relative to the grid cell size while the width and height are relative to the whole image, and conf denotes the IOU between the predicted box and the positive box.\nLater the authors also absorbed the characteristics of other network models and improved YOLO, releasing YOLO2 and YOLO3.\nSingle Shot MultiBox Detector (SSD) Like YOLO, SSD is a single-stage detection model proposed by W. Liu et al. in 2015. The main characteristic of SSD is that it uses a multi-scale detection structure. Similar to FPN, SSD extracts feature maps from different convolution layers for detection. Since the receptive field sizes of different layers differ, this approach can take objects of different sizes into account. SSD also uses convolution layers instead of fully connected layers for the final classification and regression operations.\nRetinaNet Although single-stage models run faster than two-stage models, their accuracy is generally lower. In 2017, T.-Y. Lin et al. proposed RetinaNet. The authors believe the cause of the low accuracy of single-stage models is the extreme imbalance of positive and negative samples, and for this reason they designed the focal loss function to rebalance positive and negative samples, which is also the main contribution of RetinaNet.\nThe final loss function is:\n$$ loss = -\\alpha(1-p_t)^\\gamma * log(p_t) $$where both $\\alpha$ and $\\gamma$ are hyperparameters. In the authors\u0026rsquo; own experiments on their model, $\\alpha=0.25$ and $\\gamma=2$ worked best. This function is only used when computing the classification loss.\nCornerNet Although this last method is not in the authors\u0026rsquo; roadmap, I personally find the anchor-free idea quite interesting, so I added it here. In single-stage object detection networks the Anchor-box approach is basically always used, but this approach requires a large number of anchors, which leads to an imbalance between positive and negative samples and reduces training efficiency; at the same time the anchor settings introduce many hyperparameters (such as number, size and aspect ratio), increasing the difficulty of network design. In 2018, Hei Law published CornerNet, which obtains the position of an object by detecting the corners of the object box, thus eliminating the need for the anchor step. The rough structure is as follows:\nThe feature extraction network here is mainly Hourglass. The Hourglass structure was proposed by Newell et al. in 2016; it mainly repeats bottom-up and top-down passes while combining the intermediate results, and in human pose estimation it can make good use of the spatial relationships of different body parts. It looks roughly like the figure below:\nCorner pooling is used to localize the vertices of an object. To better suit corner detection, in object detection tasks the corners of an object often lie outside the object, so corner detection cannot rely on local features; instead, all the features along the row and the column containing that point should be scanned. The figure below takes the top-left corner as an example.\nNote here that the pooling over each row and column is directional; as shown below, max pooling is performed along the row or column in the direction of the black arrow:\nDatasets Pascal VOC In early computer vision (2005 to 2012), VOC was a very important open dataset, and the parts used most for object detection tasks are VOC07 and VOC12. VOC07 has roughly 5k training images containing 12k+ annotated objects; VOC12 has 11k training images containing 27k+ annotated objects. Both annotate the 20 classes of objects commonly seen in daily life.\nILSVRC This dataset (2010 to 2017) contains 200 classes, and the total number of images is much larger than VOC; for example ILSVRC-14 contains 517k images and 534k annotated objects.\nMS-COCO The COCO dataset, developed from 2015 to the present, is the most challenging dataset in the current object detection field. It has fewer classes than ILSVRC but far more annotated objects than ILSVRC. The MS-COCO-17 dataset has 164k images in total, containing 897k annotated objects of 80 classes.\nOpen Images The dataset released by the Open Images detection challenge, which started in 2018, is larger in scale than the COCO dataset. For the object detection task, Open Images contains 1910k images and 15440k annotated objects of 600 classes.\nMetrics Computing precision and recall:\nActually positive Actually negative Predicted positive TP FP Predicted negative FN TN Precision: among the samples predicted as positive, the proportion that are truly positive.\n$$ P = \\frac{TP}{FP+TP} $$Recall: among the truly positive samples, the proportion that are predicted as positive.\n$$ R = \\frac{TP}{FN+TP} $$In the early days, the commonly used object detection evaluation metric was FPPI (false positives per image), that is, the average number of objects correctly retrieved per image:\n$$ FPPI = \\frac{FP}{N} $$VOC2007 introduced the AP value. The AP value is the average of the precision at different recall rates. This definition is for a single class; for multiple classes, mAP (the average of the APs of all classes) is used for evaluation.\nIn object detection, whether a box is correct can be determined by computing the overlap (IOU) between that box and the ground truth box and comparing it with a set overlap threshold.\n$$ IOU(A,B) = \\frac{A\\cap B}{A\\cup B} $$By setting different IOU thresholds, the strictness of the evaluation criterion can be changed. After 2014, the MS-COCO dataset\u0026rsquo;s evaluation criterion defines the AP value as the average AP over IOU thresholds from 0.5 to 0.95.\nThe figure below lists the accuracy of various detection algorithms:\nChallenge Lightweight object detection: the overall performance of object detection algorithms still lags somewhat behind human vision, and the gap in speed is even more obvious. So lightweight model design is an important research direction;\nDetection meets AutoML: today\u0026rsquo;s detection models are increasingly complex and also depend on people\u0026rsquo;s experience. Network architecture search can greatly reduce the extent to which people are involved in model structure design; models in this direction already exist, such as Efficientdet, and it will be a development direction in the future as well;\nWeakly supervised detection: from the development of datasets we can see that detection models have enormous data requirements, while annotating datasets consumes a lot of manpower and time, and annotations by different people have some bias. So how to achieve weakly supervised object detection is an important problem;\nDetection in videos: how to perform object detection in videos is also an important problem in the detection field. The usual approach is to detect every frame of the video, but this ignores the correlation information between video frames. Mining the spatial and temporal correlations in videos is very important for object detection in videos\nReference This post is mainly based on the following articles; for more detailed content see the original paper: Object Detection in 20 Years: A Survey\nRapid object detection using a boosted cascade of simple features\nHistograms of oriented gradients for human detection\nA discriminatively trained, multiscale, deformable part model\nRich feature hierarchies for accurate object detection and semantic segmentation\nSpatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition\nFast R-CNN\nFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks\nFeature Pyramid Networks for Object Detection\nYou Only Look Once: Unified, Real-Time Object Detection\nSSD: Single Shot MultiBox Detector\nFocal Loss for Dense Object Detection\nCornerNet: Detecting Objects as Paired Keypoints\nCornerNet: a new idea for object detection algorithms\nThe PASCAL Visual Object Classes (VOC) Challenge\nImageNet Large Scale Visual Recognition Challenge\nMicrosoft coco: Common objects in context\nThe Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale\nHow to better understand Kaiming He\u0026rsquo;s \u0026ldquo;Focal Loss\u0026rdquo;?\nSSD object detection\nIn-depth understanding of YOLO v1\nUnderstanding Faster RCNN in one article\nSPP-Net\nDPM (Deformable Parts Model)\nHOG feature algorithm\nViola Jones Face Detector\n","permalink":"https://simplestory.github.io/en/posts/20201120_0595a0de31914bca4c526fb50d30b2dd/","summary":"\u003cblockquote\u003e\n\u003cp\u003eObject detection is an important branch of the computer vision field, and its essential content is: What objects are where?\nAt the same time, as a basic task it supports some more complex problems in the field (such as instance segmentation, image text recognition, object tracking and so on). From the application point of view, object detection can be divided into two major categories: general object detection and detection applications. General object detection mainly explores methods for detecting different types of objects within a unified algorithmic framework, similar to human visual perception; detection applications focus on actual scenarios, such as pedestrian detection, face detection and text detection.\u003c/p\u003e\n\u003c/blockquote\u003e","title":"Object Detection"},{"content":" Usual object detection algorithms are all based on Anchors; although this generates many useless anchor boxes and brings a time-consuming NMS post-processing step, the accuracy of the algorithm is higher, because the anchor boxes basically cover all object regions. There is another approach in the detection field based on Anchor free, that is, without anchor boxes; as far as I know these are mainly Cornernet, Centernet (Keypoint Triplets for Object Detection) and Centernet (Objects as Points). This type of algorithm does not need to generate anchor boxes, and therefore does not need the time-consuming NMS post-processing. Here I mainly record the processing method of Centernet (Objects as Points).\nModel structure Centernet mainly borrows the operation of keypoint detection to detect the center position of an object, and extends it to the object detection field, that is, obtaining the box size through regression. This algorithm can be conveniently extended to other fields, such as the object detection, 3D box prediction and pose mentioned in the paper. For the backbone the authors mainly use hourglass, resnet and DLA models; for object detection, three branches are added after the backbone, mainly: a Heat maps branch to obtain the object center point, Center offset to correct the object center point, and Box size to regress the object box size. The rough structure is as follows (where cls denotes the number of classes to detect):\nData flow Having described the model structure, next comes the data processing. The authors annotate the object center points with a heat map, and the heat map is produced by a Gaussian kernel.\nFirst the $H\\times W\\times 3$ image is downsampled (the downsampling factor $R$ is set to $4$ here) to give an image of $H/R\\times W/R\\times cls$, where $cls$ is the number of classes. Then the center point of the ground truth box on the scaled image is computed: $\\hat{p}(x,y)=\\lfloor\\frac{p}{R}\\rfloor$, and the center point is mapped through the Gaussian kernel $exp(-\\frac{(x-\\hat{p}_x)^2+(y-\\hat{p}_y)^2}{2\\sigma_p^2})$, finally giving the heat map $\\hat{Y}\\in [0,1]^{\\frac{W}{R}\\times\\frac{H}{R}\\times 3}$.\nThe generated heat map does not have 1 at the center point position and 0 elsewhere. It is 1 at the object point and gradually decreases as it spreads outward. In the Gaussian kernel mapping, the corresponding Gaussian function is drawn centered at $\\hat{p}$ and the region with $\\hat{p}$ as the center and $radius$ as the radius is cropped out; meanwhile $\\sigma_p$ in the Gaussian kernel is the standard deviation adapted to the object size, which in the authors\u0026rsquo; implementation is set to $\\frac{2\\times radius}{6}$. The design of the cropped region radius $radius$ and of the Gaussian kernel parameter $\\sigma_p$ here both refer to another paper, Cornernet. In Cornernet, the standard deviation of the Gaussian kernel is designed to be $\\frac{1}{3}$ of the radius of the allowed deviation region (that is, $radius$). Considering that a box generated by a point near the object point (the object point in Cornernet being the corner of the object box) will have a fairly large IoU with the object box, directly assigning 0 to that point is inappropriate, so an allowed deviation region is formed by setting an IoU threshold (set to 0.7 in Centernet), within which the values of the points around the object point gradually decrease following the Gaussian function. The boxes generated by these points have three limiting cases of intersection with the object box; the region radius is computed for each of the three cases, and the smallest of the three radii is taken as the value of $radius$.\nComputing radius The generated box circumscribes the object box\nFrom the figure above the IoU can be computed:\n$$ IoU = \\frac{w\\cdot h}{(w+2r)\\cdot(h+2r)} \\ge iou\\_overlap $$so the maximum radius in this case satisfies:\n$$ \\begin{aligned} \u0026 \\frac{w\\cdot h}{(w+2r)\\cdot(h+2r)} = k = iou\\_overlap \\\\ \\Rightarrow \u0026 \\ 4k\\cdot r^2 + 2k(w+h)\\cdot r + (k-1)wh = 0 \\end{aligned} $$This is a quadratic equation in one variable; provided it has a solution, the positive solution for $r$ can be computed with the formula. ($y = \\frac{-b\\pm\\sqrt{b^2-4ac}}{2a}$, where $b^2-4ac\\ge 0$).\nThe generated box is inscribed in the object box\nFrom the figure above the IoU can be computed:\n$$ IoU = \\frac{(w-2r)\\cdot(h-2r)}{w\\cdot h} \\ge iou\\_overlap $$so the maximum radius in this case satisfies:\n$$ \\begin{aligned} \u0026 \\frac{(w-2r)\\cdot(h-2r)}{w\\cdot h} = k =iou\\_overlap \\\\ \\Rightarrow \u0026 \\ 4\\cdot r^2 - 2(w+h)\\cdot h + (1-k)wh = 0 \\end{aligned} $$The positive solution for $r$ can likewise be obtained\nThe generated box is half circumscribed and half inscribed with respect to the object box\nThe IoU can be computed as follows:\n$$ IoU = \\frac{(w-r)\\cdot(h-r)}{2\\cdot h\\cdot w - (w-r)\\cdot(h-r)} \\ge iou\\_overlap $$so the maximum radius in this case satisfies:\n$$ \\begin{aligned} \u0026 \\frac{(w-r)\\cdot(h-r)}{2\\cdot h\\cdot w - (w-r)\\cdot(h-r)} = k = iou\\_overlap \\\\ \\Rightarrow \u0026 \\ (1+k)\\cdot r^2 - (w+h)(1+k)\\cdot r + (1-k)wh = 0 \\\\ \\Rightarrow \u0026 \\ r^2 -(w+h)\\cdot r + \\frac{1-k}{1+k}wh = 0 \\end{aligned} $$Similarly the positive solution for $r$ can be found\nTaking all three cases together, the smallest of the three is taken as the value of $radius$.\nLoss function This model has good scalability; by adding different branches it can be applied to different fields, among which two branches are fixed (Heat maps and Center offset), with corresponding loss functions being the center point loss and the center point offset loss.\nCenter point loss Here the authors also design the loss function referring to Cornernet\u0026rsquo;s, in the form:\n$$ L_k = -\\frac{1}{N} \\begin{cases} (1-\\hat{Y}_{xyc})^\\alpha log(\\hat{Y}_{xyc}) \\ \u0026 \\text{if } Y_{xyc}=1 \\\\ (1-Y_{xyc})^\\beta(\\hat{Y}_{xyc})^\\alpha log(1-\\hat{Y}_{xyc}) \\ \u0026 \\text{otherwise} \\end{cases} $$where $N$ denotes the number of object center points in the image, $\\alpha$ and $\\beta$ are hyperparameters, and the authors chose $\\alpha=2$ and $\\beta=4$. $Y_{xyc}$ denotes the value at the corresponding position on the ground truth heat map, while $\\hat{Y}_{xyc}$ denotes the value at the corresponding position on the heat map predicted by the model.\nOverall this is an improved $focal \\ loss$ function. When $Y_{xyc}=1$, the corresponding weight can be changed through $(1-\\hat{Y}_{xyc})^\\alpha$: if the model prediction is already close to $1$, the sample is easy for the model and the corresponding loss weight decreases; otherwise the sample is hard and the weight increases.\nWhen $Y_{xyc}\\neq 1$, there are two cases. One is that the predicted point is close to the object point, that is, $1-Y_{xyc}$ is small; here the model prediction should be 0, and if the model prediction is close to 1 the corresponding sample loss weight increases — but since the point is close to the object point, a large prediction at that point is acceptable, so $(1-Y_{xyc})^\\beta$ is used to reduce the weight as a balance. The other is that the predicted point is far from the object point, that is, $1-Y_{xyc}$ is large; under the effect of $(1-Y_{xyc})^\\beta$ the corresponding sample loss weight increases — but since the point is far from the object point, a small prediction at that point is acceptable, so $\\hat{Y}_{xyc}^\\alpha$ is used to reduce the weight as a balance.\nCenter point offset loss Because the center point coordinates are rounded when downsampling the ground truth annotations, a deviation is introduced, so an L1 loss is used to guide the model to make up for this difference:\n$$ \\begin{aligned} L_{off} \u0026= \\frac{1}{N}\\sum_p\\lvert \\hat{O}_{\\hat{p}} - (\\frac{p}{R}-\\hat{p})\\rvert \\\\ \u0026= \\frac{1}{N}\\sum_p\\lvert \\hat{O}_{\\hat{p}} - (\\frac{p}{R}-\\lfloor \\frac{p}{R}\\rfloor)\\rvert) \\end{aligned} $$where $N$ denotes the number of object center points in the image and $\\hat{O}\\in \\mathcal{R}^{\\frac{W}{R}\\times\\frac{H}{R}\\times 2}$ denotes the center point offsets predicted by the model. Note that this loss function only acts on the corresponding object points; all other positions are ignored.\nOther losses The model can be extended, and the corresponding loss functions differ somewhat; here only the loss function for object detection is explained.\nLet $(x_1^{(k)}, y_1^{(k)}, x_2^{(k)}, y_2^{(k)})$ denote the object box coordinates of object $k$ in class $c_k$, whose center point coordinates are $p_k=(\\frac{x_1^{(k)}+x_2^{(k)}}{2}, \\frac{y_1^{(k)}+y_2^{(k)}}{2})$; for object detection the model mainly regresses the size of each object box $s_k=(x_2^{(k)}-x_1^{(k)}, y_2^{(k)}-y_1^{(k)})$. To reduce the computational burden, the authors use a unified object box size for the objects of each class, that is, all objects of a class have only one size. The L1 loss is used as follows:\n$$ L_{size} = \\frac{1}{N}\\sum_{k=1}^N\\lvert \\hat{S}_{p_k} - s_k\\rvert $$where $N$ denotes the number of object center points in the image and $\\hat{S}\\in \\mathcal{R}^{\\frac{W}{R}\\times \\frac{H}{R}\\times 2}$ denotes the object box sizes predicted by the model.\nThe authors do not normalize the sizes but scale the magnitude of the loss with coefficients:\n$$ L_{det} = L_k + \\lambda_{size}L_{size} + \\lambda_{off}L_{off} $$The formula above is for object detection; the two hyperparameter values chosen by the authors are $\\lambda_{size}=0.1$ and $\\lambda_{off}=1$.\nInference stage On the heat map output of the model, the $topK$ peaks are extracted for each class. A point is extracted when it is the maximum among its surrounding 8 points; this operation can be done with a $3\\times 3$ max pooling. This is in fact equivalent to performing NMS. Then the corresponding predicted boxes are obtained from the extracted peak points, the predicted center point offsets and the predicted sizes:\n$$ \\begin{aligned} \u0026(\\hat{x}_i+\\delta\\hat{x}_i-\\hat{w}_i/2,\\ \\hat{y}_i+\\delta\\hat{y}_i-\\hat{h}_i/2\\\\ \u0026\\hat{x}_i+\\delta\\hat{x}_i+\\hat{w}_i/2,\\ \\hat{y}_i+\\delta\\hat{y}_i+\\hat{h}_i/2) \\end{aligned} $$Reference Objects as Points\nCornerNet: Detecting Objects as Paired Keypoints\nThrow away anchor! The real CenterNet — an interpretation of the Objects as Points paper\n","permalink":"https://simplestory.github.io/en/posts/20201025_93f382c06f3476ad1f59351e0d99f842/","summary":"\u003cblockquote\u003e\n\u003cp\u003eUsual object detection algorithms are all based on Anchors; although this generates many useless anchor boxes and brings a time-consuming NMS post-processing step, the accuracy of the algorithm is higher, because the anchor boxes basically cover all object regions. There is another approach in the detection field based on Anchor free, that is, without anchor boxes; as far as I know these are mainly Cornernet, Centernet (Keypoint Triplets for Object Detection) and Centernet (Objects as Points). This type of algorithm does not need to generate anchor boxes, and therefore does not need the time-consuming NMS post-processing. Here I mainly record the processing method of Centernet (Objects as Points).\u003c/p\u003e\n\u003c/blockquote\u003e","title":"Anchor free--Centernet"},{"content":" Having studied so many deep learning algorithms, the loss function is always an unavoidable point, but fortunately the loss functions used in general are fairly fixed, with occasional changes made as things go. This post mainly lists some commonly used loss functions.\nL1-loss The L1 loss function, also known as Mean Absolute Error (MAE). Its mathematical expression is as follows:\n$$ MAE = \\frac{1}{n}\\sum^n_{i=1}\\vert f(x_i)-y_i\\vert $$where $n$ denotes the number of samples in the training batch, $f(x_i)$ denotes the value predicted by the model at sample $x_i$, and $y_i$ is the ground truth value corresponding to sample $x_i$.\nL2-loss The L2 loss function, also known as Mean Square Error (MSE). Its mathematical expression is as follows:\n$$ MSE = \\frac{1}{n}\\sum^n_{i=1}(f(x_i)-y_i)^2 $$where $n$ denotes the number of samples in the training batch, $f(x_i)$ denotes the value predicted by the model at sample $x_i$, and $y_i$ is the ground truth value corresponding to sample $x_i$.\nSmooth L1 loss The L1 loss function is not sensitive to data outliers, while the L2 loss is overly sensitive. This loss function was proposed to make training smooth. Specifically:\n$$ loss(x_i) = \\begin{cases} 0.5 * (f(x_i)-y_i)^2, \\ \u0026\\vert f(x_i)-y_i\\vert \u003c1 \\\\ \\vert f(x_i)-y_i\\vert - 0.5, \\ \u0026\\text{otherwise} \\end{cases} $$Only the loss computation for sample $x_i$ is shown here, where $f(x_i)$ denotes the value predicted by the model at sample $x_i$ and $y_i$ is the ground truth value corresponding to sample $x_i$.\nCross Entropy loss The cross entropy loss function (CE) is mainly used in classification algorithms.\nFor 0-1 classification, we have\n$$ loss(x_i) = -y_i * log(p_i)-(1-y_i) * log(1-p_i) $$where $p_i$ denotes the probability that the model predicts sample $x_i$ as positive, and $y_i$ is the ground truth class of sample $x_i$, being 1 for positive samples and 0 for negative samples.\nFor multi-class classification, we have\n$$ loss(x) = -\\sum^c_{j=1}y_i * log(p_i) \\\\ y_i = \\begin{cases} 1, \\ \u0026\\text{the label of x is i} \\\\ 0, \\ \u0026\\text{otherwise} \\end{cases} $$where $p_i$ is the probability that the model predicts the sample as class $i$; the model generally applies softmax first and then computes the cross entropy loss.\nFocal loss This loss function is meant to deal with the imbalance between positive and negative samples, mainly addressing the skewed ratio of positive to negative samples among the anchors in single-stage object detection algorithms.\nLet\n$$ p_t = \\begin{cases} p, \\ \u0026 y=1 \\\\ 1-p, \\ \u0026 y=0 \\end{cases} $$then the cross entropy loss function can be written uniformly as:\n$$ loss = -log(p_t) $$To adjust the ratio of positive to negative samples, a natural idea is to weight with a coefficient, as follows:\n$$ loss = -\\alpha_t * log(p_t) \\\\ \\alpha_t = \\begin{cases} \\alpha, \\ \u0026 y=1 \\\\ 1-\\alpha, \\ \u0026 y=0 \\end{cases} $$But this does not distinguish easy samples from hard samples, and normally the number of easy samples is far larger than the number of hard samples. Switching finally to an adaptive coefficient gives:\n$$ loss = -(1-p_t)^\\gamma * log(p_t) $$The authors found experimentally that adjusting the adaptive coefficient gives a slight improvement in model performance, so the final loss function is as follows:\n$$ loss = -\\alpha(1-p_t)^\\gamma * log(p_t) $$where both $\\alpha$ and $\\gamma$ are hyperparameters. In the authors\u0026rsquo; own experiments on the model, $\\alpha=0.25$ and $\\gamma=2$ worked best. This function is only used when computing the classification loss; see the loss function of the Retina model for details.\nReference Focal Loss for Dense Object Detection\nHow to better understand Kaiming He\u0026rsquo;s \u0026ldquo;Focal Loss\u0026rdquo;?\nObject detection: RetinaNet: Focal Loss for Object Detection\n","permalink":"https://simplestory.github.io/en/posts/20201018_25c3043509dadf35f8ecc40adf5a369c/","summary":"\u003cblockquote\u003e\n\u003cp\u003eHaving studied so many deep learning algorithms, the loss function is always an unavoidable point, but fortunately the loss functions used in general are fairly fixed, with occasional changes made as things go. This post mainly lists some commonly used loss functions.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch3 id=\"l1-loss\"\u003eL1-loss\u003c/h3\u003e\n\u003cp\u003eThe L1 loss function, also known as Mean Absolute Error (MAE). Its mathematical expression is as follows:\u003c/p\u003e\n$$\nMAE = \\frac{1}{n}\\sum^n_{i=1}\\vert f(x_i)-y_i\\vert\n$$\u003cp\u003ewhere $n$ denotes the number of samples in the training batch, $f(x_i)$ denotes the value predicted by the model at sample $x_i$, and $y_i$ is the ground truth value corresponding to sample $x_i$.\u003c/p\u003e","title":"Some Common Loss Functions"},{"content":" At the end of 2019 and the beginning of 2020, Google Brain published two papers proposing a fast and accurate model for image classification and for object detection respectively, naming them EfficientNet and EfficientDet. The authors also adjusted the model parameters to obtain a series of efficient classification and detection models.\nEfficientNet From previous work and their own experiments, the authors summarized two observations:\nScaling any single dimension of a model (depth, width, resolution) improves the model\u0026rsquo;s accuracy, but for very large scaling factors the accuracy gain diminishes In scaling a convolutional network, scaling all dimensions of the model in a balanced way yields better accuracy and speed compound scaling Based on the two observations above, the authors no longer use the traditional single-dimension model scaling but scale all dimensions of the model (depth, width and resolution).\nFirst, for a convolution model we can write:\n$$ \\mathcal{N} = \\underset{i=1...s}{\\bigodot}\\mathcal{F}_i^{L_i}(X_{\\langle H_i, W_i, C_i\\rangle}) $$where $\\mathcal{F}_i^{L_i}$ means that in network block $i$ the network layer $F_i$ is repeated $L_i$ times, $\\langle H_i,W_i,C_i\\rangle$ denotes the input size of the $i$-th layer, and $\\mathcal{N}$ is the whole convolutional network model.\nFrom the formula above we get the abstracted model scaling optimization problem:\n$$ \\begin{aligned} \\underset{d,w,r}{max} \u0026\\ Accuracy(\\mathcal{N}(d,w,r)) \\\\ s.t. \u0026\\ \\mathcal{N} = \\underset{i=1...s}{\\bigodot}\\hat{\\mathcal{F}}_i^{d\\cdot \\hat{L}_i}(X_{\\langle r\\cdot \\hat{H}_i, r\\cdot \\hat{W}_i, w\\cdot \\hat{C}_i\\rangle}) \\\\ \u0026\\ \\text{Memory}(\\mathcal{N}) \\le target\\_memory \\\\ \u0026\\ \\text{FLOPS}(\\mathcal{N}) \\le target\\_flops \\end{aligned} \\tag{1} $$where $L_i,H_i,W_i,C_i$ denote the model depth, input length, input width and model width after scaling respectively. $w,d,r$ are the scaling coefficients corresponding to the model width, depth and resolution.\nThe authors use a scaling ratio coefficient $\\phi$ to uniformly scale all dimensions of the model:\n$$ \\begin{aligned} \\text{depth:} \u0026 \\ d = \\alpha^\\phi \\\\ \\text{width:} \u0026 \\ w = \\beta^\\phi \\\\ \\text{resolution:} \u0026 \\ r = \\gamma^\\phi \\\\ s.t. \u0026 \\ \\alpha \\cdot \\beta^2 \\cdot \\gamma^2 \\approx 2 \\\\ \u0026 \\ \\alpha \\ge 1,\\beta \\ge 1,\\gamma \\ge 1 \\\\ \\end{aligned} \\tag{2} $$where $\\alpha,\\beta,\\gamma$ are constants that can be determined by a small grid search. As for the constraint $\\alpha\\cdot\\beta^2\\cdot\\gamma^2\\approx 2$ given by the authors, it means that the scaled model will have roughly $2^\\phi$ times the floating point operations of the original model, because doubling the model depth doubles the model\u0026rsquo;s computation, while doubling the model width or resolution increases the computation fourfold, so the base factor by which the model\u0026rsquo;s computation grows is $\\alpha\\cdot\\beta^2\\cdot\\gamma^2$.\nArchitecture To get better results, the model baseline is also a very important component; the authors used network search to find the EfficientNet-B0 network within a constrained search space. The search targets $ACC(m)\\times[FLOPS(m)/T]^w$, where $ACC(m)$ denotes the model accuracy, $FLOPS(m)$ denotes the model\u0026rsquo;s computation, $T$ is the computation target of the model and $w=-0.07$ is a hyperparameter controlling the balance between accuracy and computation; the reason for using computation rather than inference latency as the evaluation target is that the authors want to decouple the model from the device, since the inference latency differs across devices while the computation is the same. The final model baseline is as follows:\nFrom the figure we can see that its main components are MBConv and the SE block. MBConv is the inverted bottleneck used in Mobilenetv2, while the SE block is similar to an attention mechanism — a lightweight module that optimizes the model.\nStarting from the resulting baseline, the following two scaling steps give the final series of classification models:\nFirst fix $\\phi=1$ and, assuming more available resources, perform a small grid search over $\\alpha,\\beta,\\gamma$ based on equations 1 and 2. The best values under the constraint $\\alpha\\cdot\\beta^2\\cdot\\gamma^2\\approx 2$ turn out to be $\\alpha=1.2,\\beta=1.1,\\gamma=1.15$, which are used as the parameters of the EfficientNet-B0 model. Fix $\\alpha,\\beta,\\gamma$ and scale with different $\\phi$ using equation 3 to obtain the B1 to B7 models. At this point the eight models EfficientNet-B0 to B7 have been constructed.\nEfficientDet To obtain an accurate and fast object detection model, the authors propose the EfficientDet model based on a single-stage detection algorithm, mainly solving the following two problems:\nHow to perform efficient multi-scale feature fusion — the authors\u0026rsquo; answer is BiFPN How to scale the model effectively — here the authors apply the same compound scaling method as in EfficientNet BiFPN The purpose of multi-scale feature fusion is to aggregate features from different resolutions so that the model generalizes better. FPN is an effective means of multi-scale feature processing; it adds a top-down path to obtain information from other branches. But since there is only one path, its expressive power is limited, so PANet was later proposed, adding a bottom-up path on top of FPN. After that, researchers used network search to find NAS-FPN, whose expressive power is higher than PANet, but its computation is enormous and it is not suitable for practical scenarios. Finally the authors propose BiFPN, whose structure is as follows:\nBiFPN is improved on top of PANet. First, the authors remove the nodes in PANet that have only a single input, in order to simplify the model, because nodes with only a single input do not need feature fusion and contribute relatively little in a structure whose purpose is feature fusion, so removing them has little impact. Second, an extra connection is added between the input and output nodes of the structure, so as to fuse more features without adding too much computation. Finally, unlike PANet which has only one top-down and one bottom-up path, the authors reuse BiFPN repeatedly to allow the fusion of the high-level features extracted by the model.\nWhen fusing features, the usual approach is to unify the size of each feature map and then add them up, which means feature maps of different resolutions have the same weight when fused, with no difference. However, the authors observe that different resolutions contribute differently to the final output of the model, so treating feature inputs of different resolutions with equal importance is inappropriate. Based on this idea, the authors consider three weighting methods:\nUnbounded fusion: $O = \\sum_i w_i\\cdot I_i$, where $w_i$ is a learnable network weight, which can be a scalar (weighting each feature), a vector (weighting each channel) or a tensor (weighting each pixel). Using scalars for weighting gives a considerable accuracy gain with minimal computation, but scalar weights can be very large or very small, which makes model training unstable. For this reason the authors adopt weight normalization to bound the range of each weight. Softmax-based fusion: $O = \\sum_i\\frac{e^{w_i}}{\\sum_je^{w_j}}\\cdot I_i$, that is, using softmax for weighting, so the range of the weights is fixed between 0 and 1, and the values represent their respective importance. But the extra softmax computation is rather slow; to reduce the cost the authors propose the weighting method below. Fast normalized fusion: $O = \\sum_i\\frac{w_i}{\\epsilon+\\sum_jw_j}\\cdot I_i$, where $w_i \\ge 0$ is guaranteed by applying the activation function $Relu$ beforehand. This method avoids the time-consuming softmax operation. Finally, BiFPN consists of bidirectional cross-scale connections and weighted feature fusion as shown in the figure above. To speed things up, the authors also replace the convolutions in the feature fusion part with separable convolutions, and add batch normalization to BiFPN with an activation function computed after every convolution.\nNote: Note here that the input nodes used for the input-to-output connections of BiFPN and for the input-to-intermediate-node connections are different nodes with the same values. For details see this article.\nThe rough structure of EfficientDet is finally as follows:\ncompound scaling Compound scaling uses a method similar to EfficientNet, except that for object detection models there are more dimensions to scale; here the authors use a heuristic-based scaling method, again with $\\phi$ as the scaling coefficient.\nFor the backbone, the authors use the EfficientNet-B0 to B6 models.\nBiFPN scaling. The BiFPN depth $D_{bifpn}$ is increased linearly, while the BiFPN width $W_{bifpn}$ is increased exponentially. The authors then performed a grid search over ${1.2,1.25,1.3,1.35,1.4,1.45}$ to obtain the best scaling base of $1.35$, and finally scale as follows:\n$$ \\begin{aligned} W_{bifpn} \u0026 = 64 \\cdot 1.35^\\phi \\\\ D_{bifpn} \u0026 = 3 + \\phi \\end{aligned} $$For the prediction head, its width is fixed to the width of BiFPN, that is, $W_{pred}=W_{bifpn}$, but its depth is increased linearly:\n$$ D_{box} = D_{class} = 3+\\lfloor \\phi/3\\rfloor $$As for the input image resolution, because feature levels 3 to 7 of the model are used by BiFPN, the input dimension must be a multiple of $2^7=128$. It is increased linearly as follows:\n$$ R_{input} = 512+\\phi\\cdot 128 $$Through this series of operations we obtain eight object detection models, EfficientDet-D0 to D7; EfficientDet-D6 and D7 have the same parameters except for the resolution parameter. Specifically:\nReference This post is mainly based on the following articles:\nEfficientNet: Rethinking Model Scaling for Convolutional Neural Networks\nEfficientDet: Scalable and Efficient Object Detection\nThe unglamorous pytorch implementation of efficientdet that topped the SoTA charts\n","permalink":"https://simplestory.github.io/en/posts/20201003_fc3691951ecd38cc130a442bcc0566ed/","summary":"\u003cblockquote\u003e\n\u003cp\u003eAt the end of 2019 and the beginning of 2020, Google Brain published two papers proposing a fast and accurate model for image classification and for object detection respectively, naming them EfficientNet and EfficientDet. The authors also adjusted the model parameters to obtain a series of efficient classification and detection models.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"efficientnet\"\u003eEfficientNet\u003c/h2\u003e\n\u003cp\u003eFrom previous work and their own experiments, the authors summarized two observations:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eScaling any single dimension of a model (depth, width, resolution) improves the model\u0026rsquo;s accuracy, but for very large scaling factors the accuracy gain diminishes\u003c/li\u003e\n\u003cli\u003eIn scaling a convolutional network, scaling all dimensions of the model in a balanced way yields better accuracy and speed\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"compound-scaling\"\u003ecompound scaling\u003c/h3\u003e\n\u003cp\u003eBased on the two observations above, the authors no longer use the traditional single-dimension model scaling but scale all dimensions of the model (depth, width and resolution).\u003c/p\u003e","title":"The Efficient Model Family"},{"content":" IoU is an important metric in object detection; it measures how good a predicted box is by computing the ratio of the intersection to the union of the predicted box and the ground truth box. But the usual bounding box adjustment function generally adopts the L2 norm, that is, the loss is computed in an MSE manner, and some studies have shown that this is not the optimal optimization function, hence the IOU loss functions.\nComputation of IoU:\n$$ IoU = \\frac{\\vert B \\cap B^{gt}\\vert}{\\vert B \\cup B^{gt}\\vert} $$where $B$ denotes the predicted box and $B^{gt}$ denotes the ground truth box; overall $IoU$ is the ratio of the intersection of the predicted and ground truth boxes to their union. It can effectively reflect the detection quality of the predicted box, and it is also scale invariant, that is, insensitive to scale.\nIoU Loss From the $IoU$ formula we can obtain a loss function simply and crudely:\n$$ \\mathcal{L}_{IoU} = 1-IoU = 1-\\frac{\\vert B \\cap B^{gt}\\vert}{\\vert B \\cup B^{gt}\\vert} $$This loss function has an unavoidable flaw: when the two boxes do not intersect, the loss is constantly 1, so a non-intersecting box that is close and one that is far away give the same output; the direction of the gradient is lost and optimization is impossible.\nNote: $Iou \\ Loss$ has more than the implementation mentioned above; for example, there is the cross-entropy form of $UnitBox$ and the $Smooth-L1$ form of $IoUNet$, but they all share the problem above.\nGIoU Loss (generalized IoU loss) To solve the problem above, $GIoU$ appeared. It adds a penalty term on top of $IoU \\ loss$. The larger the distance between the predicted box and the ground truth box, the larger the value of the penalty term. The concrete formula is as follows:\n$$ \\mathcal{L}_{GIoU} = 1-IoU+\\frac{\\vert C-B\\cup B^{gt}\\vert}{\\vert C\\vert} $$where $C$ is the smallest enclosing box containing both the predicted box and the ground truth box. During training, $GIoU \\ loss$ tends to first enlarge the predicted box so that it overlaps the ground truth box, and then, guided by the $IoU$ term in the formula, maximizes the overlapping region. But when the ground truth box completely contains the predicted box, the penalty term in the formula above becomes 0, that is, $GIoU loss$ degrades to $IoU \\ loss$, and because this loss relies too heavily on the $IoU$ term, more iterations are needed to converge, especially in the horizontal and vertical directions.\nDIoU Loss (Distance-IoU loss) Just like $GIoU$, $DIoU$ also adds a penalty term on top of $IoU \\ loss$, but $DIou$ takes into account the distance between the predicted and ground truth boxes, the overlap ratio and the scale, specifically as follows:\n$$ \\begin{aligned} \\mathcal{L}_{DIoU} \u0026 = 1-IoU+\\mathcal{R}(\\mathcal{B}, \\mathcal{B}^{gt}) \\\\ \\mathcal{R}_{DIoU} \u0026 = \\frac{\\rho^2(b, b^{gt})}{c^2} \\end{aligned} $$where $b$ and $b^{gt}$ denote the center points of the predicted box and the ground truth box respectively, $\\rho$ is the distance formula between the two points — here the Euclidean distance between the two center points — and $c$ is the diagonal distance of the smallest box containing the predicted box and the ground truth box, as shown in detail below.\nThis loss function directly minimizes the distance between the two boxes, so it converges faster than $GIoU$. $DIoU$ can also be applied to $NMS$, making the resulting boxes more reasonable.\nCIoU (Complete IoU Loss) Taking the aspect ratio of the boxes into account on top of $DIoU$ gives $CIoU$, specifically as follows:\n$$ \\begin{aligned} \\mathcal{L}_{CIoU} \u0026 = 1-IoU+\\mathcal{R}(\\mathcal{B}, \\mathcal{B}^{gt}) \\\\ \\mathcal{R}_{CIoU} \u0026 = \\frac{\\rho^2(b, b^{gt})}{c^2}+\\alpha \\mathcal{v} \\\\ \\mathcal{v} \u0026 = \\frac{4}{\\pi^2}\\left(arctan\\frac{\\mathcal{w}^{gt}}{\\mathcal{h}^{gt}}-arctan\\frac{\\mathcal{w}}{\\mathcal{h}}\\right)^2 \\\\ \\alpha \u0026 = \\frac{\\mathcal{v}}{(1-IoU)+\\mathcal{v}} \\end{aligned} $$Note:\nThe gradient of $CIoU$ contains the gradient of $\\mathcal{v}$, which is roughly as follows:\n$$ \\begin{aligned} \\frac{\\partial\\mathcal{v}}{\\partial\\mathcal{w}} \u0026 = \\frac{8}{\\pi^2}\\left(arctan\\frac{\\mathcal{w}^{gt}}{\\mathcal{h}^{gt}}-arctan\\frac{\\mathcal{w}}{\\mathcal{h}}\\right)\\times\\frac{\\mathcal{h}}{\\mathcal{w}^2+\\mathcal{h}^2} \\\\ \\frac{\\partial\\mathcal{v}}{\\partial\\mathcal{h}} \u0026 = -\\frac{8}{\\pi^2}\\left(arctan\\frac{\\mathcal{w}^{gt}}{\\mathcal{h}^{gt}}-arctan\\frac{\\mathcal{w}}{\\mathcal{h}}\\right)\\times\\frac{w}{\\mathcal{w}^2+\\mathcal{h}^2} \\end{aligned} $$When the aspect ratio is in $[0,1]$, the value of $\\mathcal{w}^2+\\mathcal{h}^2$ is usually very small, so $\\frac{1}{\\mathcal{w}^2+\\mathcal{h}^2}$ becomes rather large and may cause gradient explosion; therefore in implementations it is usually replaced by the constant 1.\nReference This post refers to the following:\nDistance-IoU Loss: Faster and Better Learning for Bounding Box Regression\nThe correct way to use IoU in object detection\nAll about IoU loss functions\n","permalink":"https://simplestory.github.io/en/posts/20200926_ffb198e96e96a613e471e2f2eb1ae86c/","summary":"\u003cblockquote\u003e\n\u003cp\u003eIoU is an important metric in object detection; it measures how good a predicted box is by computing the ratio of the intersection to the union of the predicted box and the ground truth box. But the usual bounding box adjustment function generally adopts the L2 norm, that is, the loss is computed in an MSE manner, and some studies have shown that this is not the optimal optimization function, hence the IOU loss functions.\u003c/p\u003e\n\u003c/blockquote\u003e","title":"Several Kinds of IOU Loss"},{"content":" Papers keep piling up before I know it, and it is quite exhausting. Back to the point: neural networks are already applied in many fields and even surpass human ability in image recognition, but the improvement in model accuracy comes at the cost of higher computing requirements. Real engineering applications often cannot provide much computing power, and some algorithms have to run on embedded platforms, which means those fancy algorithms simply cannot be deployed. Following mobilenetv1, Google proposed its improved version mobilenetv2, which is better suited to lightweight deployment and can keep a decent accuracy while maintaining running speed.\nDepthwise separable convolution First let us look at the depthwise separable convolution already used in mobilenetv1. The general idea is to split an ordinary convolution into two steps: a depthwise convolution and a pointwise convolution, which reduces the amount of computation while achieving the same effect.\nSuppose the input of a standard convolution is $L_i$ of dimension $h_i\\times w_i\\times d_i$, the kernel parameter is $K\\in R^{k\\times k\\times d_i\\times d_j}$, and the final output is $L_j$ of dimension $h_i\\times w_i\\times d_j$. So the computational cost of the standard convolution is $h_i\\times w_i\\times k\\times k\\times d_i\\times d_j$.\nWith a depthwise separable convolution, the input is $L_i$ of dimension $h_i\\times w_i\\times d_i$, the depthwise convolution kernel parameter is $K_1\\in R^{k\\times k\\times d_i\\times d_i}$, the pointwise convolution kernel parameter is $K_2\\in R^{1\\times 1\\times d_i\\times d_j}$, and the final output is again $L_j$ of dimension $h_i\\times w_i\\times d_j$. So the computational cost is $k\\times k\\times h_i\\times w_i\\times d_i + 1\\times 1\\times h_i\\times w_i\\times d_i\\times d_j$, that is $h_i\\times w_i\\times d_i\\times (k^2 + d_j)$. It differs from the standard convolution by a factor of $\\frac{k^2d_j}{k^2+d_j}$, so the depthwise separable convolution really is an efficient convolution design.\nIn the paper, mobilenetv2 uses $3\\times 3$ kernel parameters throughout the separable convolutions.\nLinear Bottlenecks Let us just call it the linear bottleneck layer (I don\u0026rsquo;t know a good translation\u0026hellip;). This part is mainly about the feature preservation ability of the model. It has long been believed that the \u0026ldquo;manifold of interest\u0026rdquo; in neural networks (presumably meaning something like the effective features) can be embedded in a low-dimensional subspace, hence the idea of reducing the dimension of convolution layers to reduce the dimension of the subspace and thereby obtain more effective features. However, this intuition breaks down when we recall that deep convolutional neural networks actually have non-linear per coordinate transformations.\nTaking the ReLU activation function as an example: applied to a straight line in 1D space, it produces rays in n-dimensional space, and in general produces a piecewise linear curve with n joints. From the shape of the ReLU curve we can also see that a deep network only has a linear classifier function in the non-zero part of the output domain. The authors finally give two requirements for the main features of the model to lie in a low-dimensional subspace of a high-dimensional space:\nWhen the ReLU output is non-zero, ReLU is just a simple linear transformation; Provided that the input manifold lies in a low-dimensional subspace of the input space, ReLU can preserve the information of the input manifold Assuming the second point holds, we can capture effective features by inserting linear bottleneck layers into the convolution block. The authors also show experimentally that adding linear bottleneck layers effectively prevents the nonlinear transformation from destroying too much feature information.\nAn important parameter in the linear bottleneck layer is the expansion ratio, that is, the ratio between the size of the input bottleneck and the internal size.\nI am still a bit confused by this part, so I need to read it a few more times.\nInverted residuals Inspired by the bottleneck idea, the bottleneck actually contains all the necessary information while the expansion layer merely acts as an implementation detail of the tensor\u0026rsquo;s nonlinear transformation, so the authors use shortcut connections directly between bottlenecks. This is shown below.\nThe rough structure of the bottleneck is shown below:\nModel structure As shown\nIt starts with a standard convolution layer with 32 kernels, followed by 19 residual bottleneck modules (although the table in the paper only lists 17, and the official implementation also follows the table). The activation function is ReLU6 and the kernel size is $3\\times 3$. Except for the first bottleneck, the authors use a constant expansion rate throughout the network (experiments show that model robustness is better when it is between 5 and 10)\nThe model implementation also contains some memory-efficiency improvements; see the original paper for details.\nFinally, for the object detection field, the authors also modified the SSD network and proposed a lightweight implementation called SSDLite. On top of mobilenetv2, the first layer of SSDLite is connected to the expansion of layer 15 of mobilenetv2 (output stride 16), and the remaining SSDLite layers are attached on top of the last layer (output stride 32).\nReference This post is mainly based on the following paper\nMobileNetV2: Inverted Residuals and Linear Bottlenecks\n","permalink":"https://simplestory.github.io/en/posts/20200426_02a6a4457d33491491495f7fe2b383d1/","summary":"\u003cblockquote\u003e\n\u003cp\u003ePapers keep piling up before I know it, and it is quite exhausting. Back to the point: neural networks are already applied in many fields and even surpass human ability in image recognition, but the improvement in model accuracy comes at the cost of higher computing requirements. Real engineering applications often cannot provide much computing power, and some algorithms have to run on embedded platforms, which means those fancy algorithms simply cannot be deployed. Following mobilenetv1, Google proposed its improved version mobilenetv2, which is better suited to lightweight deployment and can keep a decent accuracy while maintaining running speed.\u003c/p\u003e\n\u003c/blockquote\u003e","title":"MobileNetV2"},{"content":" Most current object detection/segmentation models are based on Fast/Faster RCNN and FPN; the authors mainly provide a comparable framework for instance segmentation, namely Mask R-CNN.\nIntroduction mask rcnn works by adding a branch on top of faster rcnn to predict segmentation masks in parallel. This branch is in fact a small FCN network applied to each ROI region.\nA brief note on fast/faster rcnn here: fast rcnn introduced ROIPool on top of rcnn, and faster rcnn introduced RPN on top of fast rcnn. Instance segmentation delineates object contours at the pixel level. There are two commonly used strategies:\nSegmenting proposals, that is, segmentation before recognition: candidate segmentation regions are delineated first, and class recognition is then performed on the regions to achieve instance segmentation. This approach is slow and has low accuracy. Semantic segmentation. This differs from instance segmentation: semantic segmentation only separates objects of the same class from each other, whereas instance segmentation must also separate objects within the same class. So this approach starts from the result of per-pixel classification and tries to split pixels of the same class into different instances, that is, it achieves instance segmentation on top of semantic segmentation. mask rcnn is based on the instance-first strategy, and performs it simultaneously with classification and box regression.\nKey points of the model mask rcnn adopts the same two-stage structure as faster rcnn. The first stage is identical to faster rcnn, namely the RPN structure; the second stage has three parallel branches: classification, box regression and segmentation mask.\nDuring training, the model uses a multi-task loss function for each roi:\n$$ L = L_{cls}+L_{box}+L_{mask} $$where $L_{cls}$ is the class loss, $L_{box}$ is the object box regression loss and $L_{mask}$ is the mask loss. The mask branch produces a $Km^2$-dimensional result for each RoI region, where $K$ is the number of classes and $m$ is the resolution of the RoI. The authors apply a sigmoid to every pixel and define the mask loss as the average binary cross-entropy loss. $L_{mask}$ is only computed on the ground-truth instances associated with the current RoI.\nUnlike the other branches, the mask branch preserves the spatial information of the input image and achieves pixel-level correspondence, which requires the roi features to be well aligned and to preserve the previous spatial positions corresponding to pixels. Previously faster rcnn had a roipool step that quantized the roi feature maps. This quantization introduces misalignment between the roi region and the extracted features; it may not affect the classification result, but it has a large negative impact on pixel-level segmentation tasks. So the authors removed the quantization operations in roipool and proposed the ROIAlign layer. This network layer can align the input image with the extracted features. It avoids all the quantization operations in roipool and instead uses bilinear interpolation to compute the pixel values of the corresponding points, as shown below:\nThe proposal of roialign can be said to be a major contribution of this paper: the core operation that the earlier roipool performed on instances was coarse spatial quantization for feature extraction, whereas roialign faithfully preserves the exact spatial location, and roialign is also insensitive to max/average pooling. It largely solves the long-standing problem of detection and segmentation with large-stride features\nMain structure To demonstrate the generality of the model, the authors instantiated several mask rcnn models with different structures. The main difference lies in the backbone of the model and the detection head network.\nAs for the backbone. The authors mainly evaluated resnet and resnext networks with depths of 50 and 101 layers. In the paper the authors use the $network-depth-features$ naming convention to name the networks.\nHere is an example of the $network-depth-features$ naming convention. In the original faster rcnn implementation, feature extraction is performed at the fourth-to-last stage of the backbone, called C4, and since that backbone is resnet-50, the whole backbone is denoted $ResNet-50-C4$ The authors also tried an FPN structure to extract ROI features.\nAs for the model detection head network. The authors chose the detection heads of faster rcnn with resnet and faster rcnn with fpn, plus a mask branch that contains only convolution layers. The paper figure is included below for details:\nImplementation details Model training stage. The hyperparameters used for training are consistent with fast/faster rcnn. The mask loss $L_{mask}$ is only defined on positive rois (with a positive to negative roi ratio of 1:3). The rpn anchors use 5 scales and 3 aspect ratios. For the convenience of later experiments, the authors trained the rpn separately.\nModel inference stage. After nms, the mask branch of the model is applied only to the 100 highest-scoring detection boxes. The mxm floating-point mask output is resized back to the roi size and binarized with a threshold of 0.5. Since the mask branch is not applied to all detection boxes, mask rcnn adds only a small amount of overhead on top of faster rcnn.\nFor more specific details, refer back to the original paper and the related source code\nReference This post is mainly based on the following\nMask R-CNN\n","permalink":"https://simplestory.github.io/en/posts/20200210_ba14ebd4efe67efd08cdc6410e758633/","summary":"\u003cblockquote\u003e\n\u003cp\u003eMost current object detection/segmentation models are based on Fast/Faster RCNN and FPN; the authors mainly provide a comparable framework for instance segmentation, namely Mask R-CNN.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003emask rcnn works by adding a branch on top of faster rcnn to predict segmentation masks in parallel. This branch is in fact a small FCN network applied to each ROI region.\u003c/p\u003e\n\u003chr\u003e\n\u003cul\u003e\n\u003cli\u003eA brief note on fast/faster rcnn here: fast rcnn introduced ROIPool on top of rcnn, and faster rcnn introduced RPN on top of fast rcnn.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eInstance segmentation delineates object contours at the pixel level. There are two commonly used strategies:\u003c/p\u003e","title":"Mask RCNN"},{"content":" Yes, it has arrived.\nIntroduction With a $320\\times 320$ resolution input, yolo3 keeps its accuracy at $28.2mAP$ and runs in roughly $22ms$.\nCompared with yolo2, this model does not bring any major changes; the authors tried many tricks to improve model performance, and finally selected the ones that proved useful and combined them into yolo3.\nBounding box and class prediction In yolo2, bounding boxes are obtained by predicting the center coordinates of the box relative to the top-left corner of the image and the width and height relative to the anchors:\n$$ \\begin{aligned} b_x \u0026 = \\sigma(t_x)+c_x \\\\ b_y \u0026 = \\sigma(t_y)+c_y \\\\ b_w \u0026 = p_we^{t_w} \\\\ b_h \u0026 = p_he^{t_h} \\\\ \\end{aligned} $$The bounding box prediction method of yolo2 is carried over here. yolo3 uses logistic regression to score the objectness of each anchor. When an anchor covers a ground-truth object over a larger area than any other box, that box is assigned a score of 1; when a box is not the best bounding box but does overlap some ground-truth object by more than a threshold (the authors chose 0.5 in the paper), the authors still ignore it. So the authors assign exactly one anchor to each ground-truth object; for the anchors that are not assigned, no coordinate or class loss is incurred, only the objectness loss. This way unnecessary boxes are removed and computation is reduced.\nAs for class prediction, yolo3 uses multi-label classification for each box to predict the classes the box may contain. The authors do not use softmax here because they found it has no impact on the high performance of the model, and an independent logistic classifier with a binary cross-entropy loss makes it easy to transfer the model to other datasets.\nModel improvements For training, the authors combine features across scales for prediction. yolo3 uses 3 different aspect ratios to predict boxes, in a way similar to the $FPN$ feature pyramid. The authors add several convolution layers after the feature extractor of the model to obtain a 3D tensor (containing bounding boxes, objectness probabilities and per-class probabilities). Then they take the feature map from two layers back, upsample it and concatenate it with the output of the last layer, which gives meaningful semantic information from earlier layers as well as fine-grained detail. The same procedure is repeated for one more merged layer, so the final model predicts at 3 different scales. For the choice of anchors, the authors follow yolo2 and use $k-means$ clustering to obtain candidate box sizes. Unlike yolo2, the authors choose 9 sizes and distribute them evenly among 3 different aspect ratios. The structure is shown below:\nFeature extractor In yolo3 the authors use a brand-new feature extractor. The new convolution stack uses consecutive $3\\times 3$ and $1\\times 1$ kernels and adds residual blocks. The whole feature extractor has 53 layers, hence it is called $Darknet-53$. The structure is shown below:\nTraining and shortcomings yolo3 is still trained on datasets without hard negative mining, while using multi-scale training, a large number of data augmentation methods, batch normalization and so on. On the new evaluation method of the COCO dataset, the mAP of the model is on par with the SSD model, but SSD runs about 3 times faster than yolo3. Under the old evaluation method, on the other hand, the mAP of the model is comparable to RetinaNet and leaves SSD far behind.\nyolo3 still has some shortcomings. In earlier versions, yolo had always performed poorly on small object detection; after multi-scale training the model improved a lot on small objects, but its performance on medium and large objects is worse than before.\nReference This post is mainly based on the following:\nYOLOv3: An Incremental Improvement\nIn-depth understanding of YOLOv3\nThe yolo series: yolo v3 [in-depth analysis]\n","permalink":"https://simplestory.github.io/en/posts/20200115_1789a504337212c61f0b46c458695b7c/","summary":"\u003cblockquote\u003e\n\u003cp\u003eYes, it has arrived.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eWith a $320\\times 320$ resolution input, yolo3 keeps its accuracy at $28.2mAP$ and runs in roughly $22ms$.\u003c/p\u003e\n\u003cp\u003eCompared with yolo2, this model does not bring any major changes; the authors tried many tricks to improve model performance, and finally selected the ones that proved useful and combined them into yolo3.\u003c/p\u003e\n\u003ch2 id=\"bounding-box-and-class-prediction\"\u003eBounding box and class prediction\u003c/h2\u003e\n\u003cp\u003eIn yolo2, bounding boxes are obtained by predicting the center coordinates of the box relative to the top-left corner of the image and the width and height relative to the anchors:\u003c/p\u003e","title":"YOLO_v3"},{"content":" After the yolov1 algorithm was released it caused a strong reaction among the public, and many researchers compared the algorithm with Fast Rcnn; yolov1 showed a fair number of deviations in object localization and had low recall. So in almost the same year the authors released yolov2, along with yolo9000 for detection over a huge number of classes — the iteration speed of this algorithm leaves one baffled (it really is awesome).\nThe authors mainly improve the algorithm model around a few points: raising the recall of the model and reducing object localization deviations while maintaining model accuracy, and they also propose a joint training method to realize the super-many-class detection algorithm yolo9000.\nBetter The authors use many tricks to simplify the model while improving accuracy. The first is batch normalization, which can significantly improve the convergence of the model while requiring no other form of regularization, so layers such as the Dropout layer in yolov1 can be removed. The authors add a BN layer after every convolution layer, improving the model\u0026rsquo;s $mAP$ by at least $2\\%$. When training the classification network, yolov1 used an input image resolution of $224\\times 224$, while in actual detection the image resolution used is $448\\times 448$; yolov2, by contrast, directly uses a size of $448\\times 448$ when fine-tuning the classification network. This operation gives the model time to adapt to the change in resolution. After raising the resolution, the model\u0026rsquo;s $mAP$ improves by about $4\\%$.\nAs is well known, yolov1 does not use the anchor boxes approach but predicts bounding boxes directly through fully connected layers; however, the authors found that predicting the box offsets is simpler than predicting the box coordinates and can simplify the network. So in yolov2 the fully connected layers are removed and anchor boxes are used to predict bounding boxes. Secondly, one pooling layer is removed so that the convolution layers of the network output a higher resolution, and an image input of $416\\times 416$ is used instead of $448\\times 448$. The reason for choosing $416\\times 416$ here is that the authors want the final feature map to have an odd size (downsampling by 32 gives a final feature map of size $13\\times 13$), so that there is a single center point on the feature map. An object, especially a large one, generally appears at the center of the image, so having a center point there to predict the bounding box works a bit better than the four surrounding points. After adopting the anchor boxes mechanism the $mAP$ of the model does not improve noticeably, and even drops slightly, but the recall does improve by a certain amount; in short, there is still room for improvement.\nAfter introducing the anchor boxes mechanism, the authors face two problems:\nThe size of the boxes. In the past the box sizes were chosen manually and then optimized by the network. Choosing good sizes improves the detection performance of the model enormously. The instability of the model. Most of the instability comes from the predicted bounding box coordinates $(x,y)$. As for choosing the box sizes, the authors use the k-means clustering algorithm to select the anchor sizes (honestly, this move is slick). The goal of introducing anchor boxes is to give the detection boxes higher IOU values, but the usual K-means clustering algorithm uses the Euclidean distance formula, which does not meet the requirement, so the authors change the distance formula and use IOU to measure distance:\n$$ d(box, centroid) = 1-IOU(box, centroid) $$Since a convolutional neural network is translation invariant and the positions of the anchor boxes are fixed by each grid cell, only the width and height of the anchors need to be computed by k-means. Through experiments, the authors finally choose k = 5.\nAs for the stability problem of the model, the authors follow yolov1\u0026rsquo;s approach, that is, predicting the offsets of the bounding box center relative to the top-left corner of the corresponding grid cell, and using a sigmoid function to constrain the box center within the current cell:\n$$ \\begin{aligned} b_x \u0026 = \\sigma(t_x)+c_x \\\\ b_y \u0026 = \\sigma(t_y)+c_y \\\\ b_w \u0026 = p_we^{t_w} \\\\ b_h \u0026 = p_he^{t_h} \\\\ Pr(object) * IOU(b, object) \u0026 = \\sigma(t_o) \\end{aligned} $$The explanation figure from the paper is attached:\nwhere $p_h$ and $p_w$ denote the sizes of the bounding boxes produced by k-means clustering\nSo the authors generate bounding boxes by using size clustering together with direct prediction of the box center position.\nTo let the model handle the detection of both large and small objects well, the authors introduce Fine-Grained Features. Instead of using only the last feature layer of $13\\times 13$ to detect objects as yolov1 did, the model also adds information from the preceding feature layer ($26\\times 26$), which gives rise to the passthrough layer. In the later code the authors borrow from ResNet and do not process the high-resolution image directly, but first apply a $1\\times 1\\times 64$ convolution. The logic of implementing the passthrough layer is roughly: take the lower-level feature map $26\\times 26\\times 512$, use $1\\times 1\\times 64$ to reduce the dimension to $26\\times 26\\times 64$, then split it into 4 parts and stack them along the channel dimension: $13\\times 13\\times 256$, which combined with the last layer\u0026rsquo;s $13\\times 13\\times 1024$ gives: $13\\times 13\\times (1024+256) = 13\\times 13\\times 1280$. This is roughly the processing of passthrough. After passing through the passthrough layer, the features are processed with $3\\times 3$ and $1\\times 1$ operations, and the output is $13\\times 13\\times 125$.\nTo make the model more robust, the authors also adopt multi-scale training. After training a certain number of batches, a size is randomly chosen from a size range as the input size of the training images (the paper changes it once every 10 batches). Because the model downsamples by 32, the input image size should be a multiple of 32, for example $\\{320, 352, ..., 608\\}$.\nFaster The feature extraction networks used by usual object detection models are based on VGG-16, a very powerful but somewhat bulky network. In the earlier yolov1 the authors used a feature extraction network based on the Googlenet architecture, which runs faster than VGG-16 but has lower accuracy than VGG-16. To improve both the efficiency and the performance of the model as much as possible, the authors redesigned yolov2.\nAnd so Darknet-19 was born. This model is similar to VGG-16, mainly using $3\\times 3$ convolution kernels and doubling the number of channels after each pooling operation. Following the conclusion of NIN (Network in Network), the authors use global average pooling while interleaving $1\\times 1$ convolution kernels to compress the number of feature channels. BN layers are of course indispensable; BN layers not only accelerate model convergence but also regularize the model. In the end, Darknet-19 has 19 convolution layers and 5 max pooling layers; the concrete structure is shown below:\nOnce the feature extraction network is built, it is time to train it. The authors train it for $160$ iterations on the standard ImageNet 1000-class classification dataset with SGD, setting the initial learning rate to $0.1$. As for updating the learning rate, the authors use a polynomial decay strategy:\n$$ lr_{cur} = (lr_{begin}-lr_{end})\\cdot (1-\\frac{gstep}{dstep})^{power} + lr_{end} $$where $lr_{begin}$ is the starting learning rate, $lr_{end}$ is the final learning rate, $dstep$ is the number of decay steps, $gstep$ is the number of global steps, and the following holds:\n$$ gstep = \\min(gstep, dstep) $$For the concrete parameter design, see the authors\u0026rsquo; original paper.\nAfter the initial training at a resolution of $224\\times 224$, the input image resolution is raised to $448\\times 448$ and the model is fine-tuned to adapt to large-resolution images.\nAfter Darknet-19 training is finished, the authors remove its last convolution layer and add 3 convolution layers with $3\\times 3\\times 1024$ kernels, each followed by a $1\\times 1$ convolution layer whose number of channels depends on the number of classes to detect. As for the passthrough layer mentioned earlier, the authors use it on the last $3\\times 3\\times 512$ layer and the second-to-last convolution layer.\nThat is the end of the YoLov2 part; what follows is where YoLo9000 takes the stage.\nStronger To detect as many objects as possible, the authors propose a method for jointly training on classification datasets and detection datasets. During model training, the authors mix images from the two datasets; when a classification image is encountered, only the classification part of the network structure is used to backpropagate the loss, while when a detection image is encountered, backpropagation is based on the whole loss function. For mixing the datasets, one problem faced is how to unify the labels, and moreover the loss function used by most classification algorithms is the softmax function, which assumes that classes are mutually independent, which does not fit the logic here.\nTo address the problems above, the authors introduce hierarchical classification, similar to the species tree in biology. Here WordNet is used (the labels of the Imagenet dataset are taken from it); it is a cognition-based language database structured as a directed graph. The authors do not use the whole graph structure (after all, the graph is enormous) but build the graph structure using only the concepts in Imagenet. To build the graph, the paths of the various Imagenet labels to the root node in WordNet are compared: words with a unique path are added to the graph, and for those with multiple paths to the root node the shortest path is chosen and added to the graph, finally producing a $WordTree$.\nGiven a synset, the conditional probability of each node is predicted, for each hyponym of that synset. For example, at the terrier node the model predicts:\n$$ Pr(Norfolk\\ terrier\\vert terrier) \\\\ Pr(Yorkshire\\ terrier\\vert terrier) \\\\ Pr(Bedlington\\ terrier\\vert terrier) \\\\ ... $$To compute the absolute probability of a particular node, one simply follows the path from that node to the root of the tree and multiplies the conditional probabilities of the nodes along the path. For example, for the Norfolk terrier node:\n$$ Pr(Norfolk\\ terrier) = Pr(Norfolk\\ terrier\\vert terrier) \\\nPr(terrier\\vert hunting\\ dog) \\ \u0026hellip; \\ * \\ Pr(mammal\\vert animal) \\ Pr(animal\\vert physical\\ object) $$ Here, for classification purposes, $Pr(physical\\ object) = 1$ is assumed, that is, the image always contains an object.\nA WordTree is built on top of the 1000 Imagenet classes to train Darknet-19; after adding all intermediate nodes, the number of labels expands from 1000 to 1369. During training, ground truth labels are propagated upwards, for example a label of \u0026ldquo;Norfolk terrir\u0026rdquo; should also be labeled \u0026ldquo;dog\u0026rdquo; and \u0026ldquo;mammal\u0026rdquo;. To compute the conditional probabilities, the model predicts a vector of length 1369 and computes the softmax over all synsets sharing the same concept, as shown below:\nAt classification time, the authors assume that each image contains one object; for the detection task, the value of $Pr(physical\\ object)$ is no longer fixed at 1 but is given by the object detector of Yolov2. The detection part of the model gives a bounding box for the object along with its probability, and then traverses down the WordTree to find the highest-confidence path for each softmax branch until a certain threshold is reached; the node farthest from the root along that path is then the predicted label.\nWhen merging the Imagenet dataset and the COCO dataset, because the sizes of the two datasets differ greatly, the authors resample the COCO dataset, so the final ratio of the two datasets is roughly $4:1$. The model keeps the YoLov2 structure but is changed to use 3 anchors (instead of 5) to limit the output size. During classification, the model backpropagates only the classification loss part, and backpropagates the objectness loss when the IOU between the predicted box and the ground truth label box is at least 0.3; during detection, the model backpropagates normally.\nFor the concrete details, refer to the original paper\nReference This post refers to the following:\nYOLO9000: Better, Faster, Stronger\n","permalink":"https://simplestory.github.io/en/posts/20200110_aa4020065e766e48007d2b3ae229757d/","summary":"\u003cblockquote\u003e\n\u003cp\u003eAfter the yolov1 algorithm was released it caused a strong reaction among the public, and many researchers compared the algorithm with Fast Rcnn; yolov1 showed a fair number of deviations in object localization and had low recall. So in almost the same year the authors released yolov2, along with yolo9000 for detection over a huge number of classes — the iteration speed of this algorithm leaves one baffled (it really is awesome).\u003c/p\u003e\n\u003c/blockquote\u003e","title":"YOLO_v2 and YOLO9000"},{"content":" Object detection models come in single-stage and two-stage variants. Faster Rcnn is the representative of the two-stage kind; among single-stage models, besides SSD we also have another mainstream model, YoLo.\nIntroduction Yolo turns the object detection problem into a regression problem over bounding boxes and class probabilities. It uses a single convolutional network to handle the prediction of multiple bounding boxes together with the class probabilities of those boxes, so the model runs very fast, though there is some loss in accuracy, since the candidate box generation process of something like Faster RCnn is gone. Secondly, the Yolo model has a lower background false positive rate and good generalization properties; the authors even ran object detection on abstract paintings, and the result was not bad :)\nDesign First the model divides the input image into $S \\times S$ grids. If the center of an object falls inside a grid cell, that cell is responsible for predicting the object (specifically, during training the bounding box with the largest IOU with the ground truth box is selected to be responsible for detecting the corresponding ground truth instance. In this way each detector has its own job and only detects one class of object). Each grid cell predicts $B$ bounding boxes along with the confidence of these boxes. Here confidence means whether there is an object in the box and the confidence in that judgment: $Pr(Object)*IOU^{truth}_{pred}$. If there is no object in the box then $Pr(Object)$ is 0; otherwise the $IOU$ between the predicted box and the ground truth box is used as its confidence value.\nEach bounding box contains 5 variables: $x,y,w,h,conf$. The first four are coordinate variables, where $x,y$ are the center of the box and $w,h$ denote the width and length. Note here that the center coordinates are relative to the grid cell size, while the width and height are relative to the whole image.\nEach grid cell predicts $C$ class conditional probabilities ($C$ is the number of classes to predict). These probabilities are computed under the condition that an object is present: $Pr(Class_i\\vert Object)$. During training, this probability is multiplied by the confidence of the box to obtain the predicted probability of each box for each class:\n$$ Pr(Class_i\\vert Object) * Pr(Object) * IOU^{truth}_{pred} = Pr(Class_i) * IOU^{truth}_{pred} $$In summary, the final output tensor of the model has size $S\\times S\\times (B*5+C)$. The authors chose the parameters $S=7$ and $B=2$, and the training dataset is $PASCAL \\ VOC$, so the number of labels is $C=20$. A classic figure from the paper is included below:\nStructure The overall model structure is a convolutional network to extract features plus fully connected layers to output the predicted probabilities and coordinate values. For the feature extraction part, the network built by the authors has a total of 24 convolution layers (for yolo tiny, 9 convolution layers are used along with a reduced number of kernels), and $1\\times 1$ downsampling layers plus $3\\times 3$ convolution layers are used to replace the inception part of GoogLeNet. The structure figure from the paper is attached as well:\nTraining The authors use the Darknet framework for model training and inference. In the pretraining stage, the authors take the first 20 convolution layers of the model and attach a global average pooling layer and a fully connected layer to form a classification network, which is trained on the $ImageNet$ dataset. Since adding both convolution and connected layers to the pretrained model improves performance, the authors add four randomly initialized convolution layers (so the final model has 24 convolution layers) and two fully connected layers. Because object detection requires a higher resolution, the input images are raised from the pretraining size $224\\times 224$ to $448\\times 448$. The last layer predicts class probabilities and box coordinates simultaneously; here the box coordinates are normalized, as mentioned above (the center coordinates are relative to the grid cell size while the width and height are relative to the whole image). The activation function used in the network layers is $Leak\\ ReLU$:\n$$ \\phi(x) = \\begin{cases} x, \\ \u0026\\text{if x \u003e 0} \\\\ 0.1x, \\ \u0026\\text{otherwise} \\end{cases} $$The loss function chosen for the model is the sum-squared error loss function, because it is simple to optimize (yeah, sometimes you just overthink it); of course the authors made some adjustments to it to fit this model. Several shortcomings are listed here:\nThe localization weight of some boxes that are not good enough is the same as the localization weight of boxes that are predicted well; Among the grid cells, there are sometimes many cells that contain no object, and the confidence of these cells is 0, so their gradients usually exceed the gradients of the boxes that contain objects, which makes the model unstable. To compensate for these problems, the authors increase the coordinate loss of the bounding boxes through the parameter $\\lambda_{coord}$ and decrease the confidence loss of boxes containing no object through $\\lambda_{noobj}$; the paper sets $\\lambda_{coord}=5$ and $\\lambda_{noobj}=0.5$.\nAt the same time, the sum-squared error treats the error on large boxes and small boxes the same, which does not match our requirement that a small deviation in a large box should count for less than a small deviation in a small box. The authors take a clever approach: take the square root of the width and height of the box and then compute the squared error.\nThe final loss function is as follows:\n$$ \\begin{aligned} Loss =\\ \u0026 \\lambda_{coord}\\sum_{i=0}^{S^2}\\sum_{j=0}^B\\mathbb{I}_{ij}^{obj}[(x_i-\\hat{x}_i)^2+(y_i-\\hat{y}_i)^2] \\\\ \u0026+\\lambda_{coord}\\sum_{i=0}^{S^2}\\sum_{j=0}^B\\mathbb{I}_{ij}^{obj}\\left[\\left(\\sqrt{w_i}-\\sqrt{\\hat{w}_i}\\right)^2-\\left(\\sqrt{h_i}-\\sqrt{\\hat{h}_i}\\right)^2\\right] \\\\ \u0026+\\sum_{i=0}^{S^2}\\sum_{j=0}^B\\mathbb{I}_{ij}^{obj}(C_i-\\hat{C}_i)^2 \\\\ \u0026+\\lambda_{noobj}\\sum_{i=0}^{S^2}\\sum_{j=0}^B\\mathbb{I}_{ij}^{obj}(C_i-\\hat{C}_i)^2 \\\\ \u0026+\\sum_{i=0}^{S^2}\\mathbb{I}_i^{obj}\\sum_{c \\in classes}(p_i(c)-\\hat{p}_i(c))^2 \\end{aligned} $$Notes on the above:\n$\\mathbb{I}_i^{obj}$ is an indicator function indicating whether an object is present in the corresponding cell $i$;\n$\\mathbb{I}_{ij}^{obj}$ is also an indicator function, indicating whether the $j$th bounding box of cell $i$ is responsible for predicting an object.\nThe first two terms of the function are the coordinate prediction loss of the model, the third term is the confidence prediction loss of bounding boxes that contain objects, and the fourth term is the confidence prediction loss of bounding boxes that contain no object. The last term is the class loss of the model (note that this function only penalizes the classification loss of those cells that contain objects).\nFor the concrete parameter settings of training, refer to the original paper. To avoid overfitting, the authors also use Dropout layers and common data augmentation techniques.\nLimitations Each cell only predicts 2 bounding boxes, and then each cell finally takes only the one with the highest IOU with the ground truth box as the final detection box, that is, each cell predicts at most one object, which is not good for detecting small objects; Although the loss function uses square roots to alleviate the problem that the same position error has the same effect on large and small boxes, the effect is small and does not fundamentally solve the problem; The model learns to predict bounding boxes from the data, so it is hard to generalize to objects with new or different aspect ratio configurations, and the model still uses fairly coarse features to generate bounding boxes. Reference This post is mainly based on the following paper:\nYou Only Look Once: Unified, Real-Time Object Detection\n","permalink":"https://simplestory.github.io/en/posts/20200101_83ba412c3f5a7a54968bf73cc917cd13/","summary":"\u003cblockquote\u003e\n\u003cp\u003eObject detection models come in single-stage and two-stage variants. Faster Rcnn is the representative of the two-stage kind; among single-stage models, besides SSD we also have another mainstream model, YoLo.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eYolo turns the object detection problem into a regression problem over bounding boxes and class probabilities. It uses a single convolutional network to handle the prediction of multiple bounding boxes together with the class probabilities of those boxes, so the model runs very fast, though there is some loss in accuracy, since the candidate box generation process of something like Faster RCnn is gone. Secondly, the Yolo model has a lower background false positive rate and good generalization properties; the authors even ran object detection on abstract paintings, and the result was not bad :)\u003c/p\u003e","title":"YOLO_v1"},{"content":" First of all, I must congratulate the People\u0026rsquo;s Republic of China on its 70th anniversary. To celebrate the arrival of the National Day holiday, I took the time to go through the SSD object detection model and then struck while the iron was hot to update the blog\u0026hellip; The biggest difference between the SSD algorithm and the previous RCNN series algorithms is that it is single-stage, whereas the RCNN series is multi-stage. See the details below.\nBasic framework The previous RCNN series algorithms basically all follow the pattern of generating candidate boxes, extracting features, classifying and regressing, whereas SSD and similar algorithms use dense sampling on feature maps to obtain prior boxes and then classify and regress. Simply put, the distinction between multi-stage and single-stage comes down to whether there is a candidate box generation process. Multi-stage contains this process, so the related algorithms are slower but more accurate; single-stage algorithms do not have this process, so the related algorithms are faster but less accurate.\nThe $SSD$ algorithm generates bounding boxes of fixed size and scores for the objects present in the boxes based on a forward-propagation convolution algorithm, and then applies a non-maximum suppression to produce the final detection results. The convolutional network used in the first half here is the standard structure network for image classification (truncated before the classification layer), which the authors call the $base \\ network$. Some auxiliary structures are attached after the $base \\ network$ to form the $SSD$ object detection network. This model has some key characteristics, listed below.\nMulti-scale feature maps for detection The authors attach some convolution feature layers after the truncated base network, and the scale of these feature layers decreases gradually. The $SSD$ model extracts the features of several convolution layers of different sizes for object detection, which forms multi-scale feature map detection: large-scale features can be used to detect small objects, and small-scale features to detect large objects. Here the authors also compare with the $YOLO$ model; $YOLO$ operates on a single-scale feature map, and obviously the amount of information obtained this way is less than SSD\u0026rsquo;s multi-scale detection, so SSD is better than $YOLO$ in accuracy.\nConvolutional predictors for detection In the added convolution feature layers mentioned above, each layer can use a series of convolutions to produce a series of fixed-size detection predictions (these predictions are either scores for object classes or bounding box offset). This differs from the $YOLO$ algorithm: $YOLO$ goes through fully connected layers after the convolution layers before detection, whereas $SSD$ directly uses convolutions to extract detection results from the different feature maps. For a convolution feature layer of size $m \\times n$ with $p$ channels, the size of the following detection convolution layer is basically $3 \\times 3 \\times p$, that is, small kernels are used to extract features further. The output bounding box offsets are measured relative to the prior boxes, and the prior boxes are in turn relative to the position of each feature map.\nDefault boxes and aspect ratios The authors associate a series of prior boxes of different sizes and aspect ratios with each feature map, similar to the $Anchor \\ box$ of $Faster \\ RCNN$. Each prior box yields class scores and its corresponding bounding box parameters. For the case where each point on the feature map has $k$ prior boxes, the model computes confidences for $c$ classes (note that $SSD$ treats background as one class and counts it in the number of classes) and $4$ offsets.\nNote: in the prediction process, the class with the highest confidence is the class the bounding box belongs to, and when the first confidence is the highest, it means the bounding box contains no object.\nThis means that $(c+4)k$ convolution kernels are applied at each position on the feature map, and for a feature map of size $m \\times n$, $(c+4)kmn$ outputs are generated in total. This approach is very similar to the $anchor \\ boxes$ of $Faster \\ RCNN$, but the difference is that $SSD$ applies prior boxes on several feature maps of different resolutions, which allows us to efficiently discretize the space of possible output box sizes.\nThe overall structure is as follows:\nThe details of the prior boxes are as follows:\nModel training Training the $SSD$ model differs from training other typical detectors that use region proposals, mainly because $SSD$ needs to assign ground truth information to specific outputs in the fixed set of detector outputs (here are the authors\u0026rsquo; own words: The key difference between training SSD and training a typical detector that uses region proposals, is that ground truth information needs to be assigned to specific outputs in the fixed set of detector outputs).\nDuring training, $SSD$ needs to determine which prior box corresponds to which ground truth instance, and trains the model according to these mappings. According to the paper, the matching principle between prior boxes and $ground \\ truth$ is as follows:\nFor each $ground \\ truth$ in the image, find the prior box with the largest $IoU$ with it, and that box matches it; this guarantees that every $ground \\ truth$ matches some prior box; For the remaining unmatched prior boxes, if the $IoU$ with some $ground \\ truth$ is greater than a preset threshold, then that prior box also matches this $ground \\ truth$. The matching principle above results in one $ground \\ truth$ matching multiple prior boxes, but one prior box is not allowed to correspond to multiple $ground \\ truth$.\nThe objective function for training $SSD$ comes from $MultiBox$, but the authors extended it to handle the multi-class classification problem. Define $x^P_{ij} \\in \\\\{1,0\\\\}$ as an indicator function; $x^P_{ij}=1$ means that the $i$-th prior box matches the $j$-th $ground \\ truth$ and the class of the $ground \\ truth$ is $P$. Under the matching strategy above we can have $\\sum_ix^P_{ij} \\ge 1$. The overall loss function is the weighted sum of the localization loss and the confidence loss, as follows:\n$$ L(x,c,l,g) = \\frac{1}{N}(L_{conf}(x,c)+\\alpha L_{loc}(x,l,g)) $$where $N$ denotes the number of positive samples among the prior boxes, $c$ is the predicted class confidence, $l$ is the predicted position of the bounding box corresponding to the prior box, and $g$ is the position parameter of the $ground \\ truth$. The localization loss uses $Smooth \\ L1 \\ loss$; as in $Faster \\ RCNN$, the authors regress the offsets of the prior box center $d(cx,cy)$ and of its width $w$ and height $h$. Note here that the ground truth $g$ must first be encoded, because the prediction $l$ is an encoded value.\n$$ \\begin{aligned} L_{loc}(x,l,g) \u0026 = \\sum_{i\\in Pos}^N \\sum_{m\\in \\{cx,cy,w,h\\}}x_{ij}^ksmooth_{L1}(l_i^m-\\hat{g}_j^m) \\\\ \\hat{g}_j^{cx} \u0026 = (g_j^{cx}-d_i^{cx})/d_i^w \\\\ \\hat{g}_j^{cy} \u0026 = (g_j^{cy}-d_i^{cy})/d_i^h \\\\ \\hat{g}_j^{w} \u0026 = log(\\frac{g_j^w}{d_i^w}) \\\\ \\hat{g}_j^{h} \u0026 = log(\\frac{g_j^h}{d_i^h}) \\\\ \\end{aligned} $$The confidence loss uses $softmax \\ loss$.\n$$ L_{conf}(x,c) = - \\sum_{i \\in Pos}^Nx_{ij}^Plog(\\hat{c}_i^P)-\\sum_{i \\in Neg}log(\\hat{c}_i^0) $$where\n$$ \\hat{c}_i^P = \\frac{exp(c_i^P)}{\\sum_Pexp(c_i^P)} $$To handle the problem of objects having different sizes, some methods suggest processing the image at different sizes and then merging the results; a similar effect can be obtained by making predictions from the feature maps of several different layers in a single network, and parameters can also be shared across all object scales. Previous experiments already verified that low-level feature maps produced by lower convolution layers can effectively improve semantic segmentation, because the lower network layers capture more detail of the input objects. In $SSD$ the authors use both low-level and high-level network layers for object detection.\nFeature maps taken from different network layers have different sizes, and in the $SSD$ network the prior boxes do not need to correspond to the actual receptive field of each network layer; the authors designed the tiling of the prior boxes so that a particular feature map can respond to objects of a particular size. Suppose $m$ feature maps are used for prediction; the prior box size on each feature map is computed with the formula below:\n$$ s_k = s_{min}+\\frac{s_{max}-s_{min}}{m-1}(k-1), \\ k \\in [1,m] $$Here $s_{min}=0.2$ and $s_{max}=0.9$ are set, meaning the minimum size is 0.2 and the maximum size is 0.9. The scale of the prior boxes follows a linear increase rule: as the size of the feature map decreases, the scale of the prior boxes increases linearly. $s_k$ in the formula above denotes the ratio of the prior box to the image. The authors impose different aspect ratios on the prior boxes, denoted $a_r \\in \\\\{1,2,3,\\frac{1}{2},\\frac{1}{3}\\\\}$, and compute the width and height of the prior boxes with the formulas below.\n$$ \\begin{aligned} w_k^a \u0026= s_k\\sqrt{a_r} \\\\ h_k^a \u0026= s_k/\\sqrt{a_r} \\end{aligned} $$For the aspect ratio of 1, the authors add one more prior box (with scale $s_k^*=\\sqrt{s_k s_{k+1}}$), so in the end each position of a feature map produces 6 prior boxes. The authors set the center coordinates of the prior boxes to $(\\frac{i+0.5}{\\vert f_k \\vert},\\frac{j+0.5}{\\vert f_k \\vert})$, where $\\vert f_k \\vert$ denotes the size of the k-th feature map and $i,j \\in [0,\\vert f_k \\vert]$. In fact we can design the distribution of the prior boxes to suit a particular dataset. By combining the prior boxes generated from multiple feature maps of different sizes and aspect ratios the whole image can be fully covered, so $SSD$ can be regarded as dense sampling. Many of the generated prior boxes are negative samples, which shows that balancing positive and negative samples is necessary. The authors use hard negative mining for the balance: the prior boxes are sorted by their confidence loss, the top ones are picked, and the ratio of positive to negative samples is kept at $1:3$.\nFor data augmentation, the authors randomly choose one of the following for each training image:\nUse the original image in full; Sample a patch region so that the minimum $jaccard \\ overlap$ with the objects is 0.1, 0.3, 0.5, 0.7 or 0.9; Randomly sample a patch region where each patch region has a size $[0.1,1]$ times the original image size and an aspect ratio between 0.5 and 2. The $jaccard \\ overlap$ (intersection over union, that is, IOU) is computed as follows:\n$$ \\begin{aligned} J(A,B) \u0026 = \\frac{\\vert A\\cap B\\vert}{\\vert A\\cup B\\vert} \\\\ \u0026 = \\frac{\\vert A\\cap B\\vert}{\\vert A \\vert +\\vert B\\vert - \\vert A\\cap B\\vert} \\end{aligned} $$There is one point to note. In the $base \\ network$, the authors use a $VGG16$ model pretrained on $ILSVRC \\ CLS-LOC$, convert $fc6$ and $fc7$ into convolution layers, and change the pooling layer $pool5$ from the original $2\\times 2$ with $stride=2$ to $3\\times 3$ with $stride=1$. Here the authors use the $Atrous$ algorithm to let the model adapt to this change (in fact the $conv6$ layer uses dilated or atrous convolution to expand the receptive field without increasing the model parameters and complexity).\nFor more details of the model, refer to the original paper.\nAppendix While implementing this model, I was rather confused by the two parameters minsize and maxsize (these two parameters are used to generate the prior box size of each feature layer). After comparing with some other source codes, the rough computation process is as follows:\nIn theory, the computation should follow the formula given in the original paper:\n$$ s_k = s_{min}+\\frac{s_{max}-s_{min}}{m-1}(k-1), \\ k \\in [1.m] $$where $s_k$ is the ratio of the prior box size of each feature layer to the original image size, $S_{min}$ and $S_{max}$ denote the minimum and maximum ratio respectively — here $S_{min} = 0.2$ and $S_{max} = 0.9$ are set according to the paper — $m$ denotes the number of feature layers (after comparing several code bases, the m used is 5 rather than 6), and $k$ is the index of the feature layer.\nIn actual use, however, we need to take rounding into account, so the formula used is as follows (rounding up is used here, though other rounding methods can also be used):\n$$ s_k = \\lceil{s_{min} \\times 100}\\rceil+\\left\\lceil{\\frac{\\lceil{s_{max}\\times 100}\\rceil-\\lceil{s_{min}\\times 100}\\rceil}{m-1}}\\right\\rceil(k-1), \\ k \\in [1,m] $$This gives a series of numbers:\n$$ s1 \\sim s6: 20, 38, 56, 74, 92, 110 $$Dividing them by 100 converts them into the real ratios:\n$$ s1 \\sim s6: 0.20, 0.38, 0.56, 0.74, 0.92, 1.10 $$Finally multiply by the original image size (300 is used here):\n$$ 60, 114, 168, 222, 276, 330 $$For $s1$ there is also an $s1^{\\prime}= 0.5\\times s1$, and finally we get:\n$$ 30, 60, 114, 168, 222, 276, 330 $$This gives 7 values, corresponding to 6 groups of $minsize$ and $maxsize$, from which the prior box size of each feature layer can be computed.\nReference This post refers to:\nSSD:Single Shot MultiBox Detector\nPrinciple and implementation of SSD for object detection (author: Xiao Xiaojiang)\nSource code reference: kuhung/SSD_keras\n","permalink":"https://simplestory.github.io/en/posts/20191003_34df20bab5e85dc75bfc94ef569cced9/","summary":"\u003cblockquote\u003e\n\u003cp\u003eFirst of all, I must congratulate the People\u0026rsquo;s Republic of China on its 70th anniversary. To celebrate the arrival of the National Day holiday, I took the time to go through the SSD object detection model and then struck while the iron was hot to update the blog\u0026hellip; The biggest difference between the SSD algorithm and the previous RCNN series algorithms is that it is single-stage, whereas the RCNN series is multi-stage. See the details below.\u003c/p\u003e\n\u003c/blockquote\u003e","title":"SSD"},{"content":" This feeling of posting several articles in a row is very exciting, very refreshing. Without further ado, as the title says, this is a faster and stronger object detection algorithm, and it also comes from Ross Girshick\u0026hellip; Honestly, this guy spent several papers in a row criticizing the models he had written before — impressive. Faster Rcnn adopts an efficient algorithm to generate candidate boxes, and also introduces the concept of Anchor.\nBasic structure In $Fast \\ RCNN$, the method of proposing candidate boxes was not discussed at all; $selective \\ search$ was simply mentioned for choosing candidate boxes, and later this became the main reason for slowing the model down. In the $Faster \\ RCNN$ model the authors made an improvement and proposed a new candidate box extraction method, namely $RPN$. Combined with this method, the running speed of the model was greatly improved, and the accuracy also improved. The rough structure of $Faster \\ RCNN$ is as follows:\nIts structure mainly consists of two parts: first a deep fully convolutional network used to generate candidate regions, and then the $Fast \\ RCNN$ model used for classification and bounding box regression. $RPN$ is applied to the convolved feature maps.\nRegion Proposal Networks The $RPN$ network takes an image of arbitrary size as input and outputs a set of object candidate boxes together with classification scores. To generate candidate regions, $RPN$ slides a small network over the convolutional feature map, and this small network takes an $n \\times n$ window on the feature map as input. Each sliding window generates a low-dimensional feature, which is fed into two branches: a bounding box regression layer ($reg$) and a box classification layer ($cls$). The structure is roughly as follows:\nThere is an important point here, namely the $Anchor$. At each sliding window position we need to predict multiple candidate boxes simultaneously; let the maximum number of candidate boxes per position be $k$, so the input of the $reg$ layer is $4k$ (encoding the coordinates of the candidate boxes) and the input of the $cls$ layer is $2k$ (that is, whether the candidate box is an object). The $Anchor$ is in fact the center of the sliding window, related to the scale and aspect ratio (my understanding is that the mapping of the current sliding window center onto the original pixel space is the $Anchor$). In the paper\u0026rsquo;s experiments, 3 scales and 3 aspect ratios were used, that is, 9 $anchor$s are generated per sliding window position; for a convolutional feature map of size $W \\times H$, $WHK$ $anchors$ are generated in total. Compared with the previous image pyramids and feature pyramids, our multi-scale detection design based on $Anchors$ relies only on a single-scale image and feature map, and uses sliding windows over a single-size feature map.\n$Anchor$ has another important property, namely translation invariance. If an object in an image is translated, the candidate region should be translated as well, and the same function can predict the candidate box at any position. This property also reduces the number of model parameters.\nTo train the $RPN$, the authors assign a binary class label to each $Anchor$. An $anchor$ is assigned a positive label in the following two cases:\nThe candidate box with the highest $IoU$ with a ground truth instance; A candidate box with an $IoU$ above 0.7 with any ground truth instance A candidate box with an $IoU$ below 0.3 is labeled negative, and $anchors$ with an $IoU$ between 0.3 and 0.7 do not participate in $RPN$ training. The training loss function is:\n$$ L(\\{p_i\\},\\{t_i\\}) = \\frac{1}{N_{cls}}\\sum_iL_{cls}(p_i,p_i^*)+\\lambda\\frac{1}{N_{reg}}\\sum_ip_i^*L_{reg}(t_i,t_i^*) $$where $i$ is the index of an $anchor$ within a $minibatch$ and $p_i$ denotes the probability that the $i$th $anchor$ is an object. $p_i^{\\*}$ is the ground truth label indicator, equal to 1 when the $anchor$ is positive and 0 when it is negative. $t_i$ is the coordinate vector of the predicted box, and $t_i^{\\*}$ is the coordinate vector of the ground truth instance. $L_{cls}$ is the log loss over the two classes; for the box regression loss, the following is used:\n$$ L_{reg}(t_i,t_i^*)=R(t_i-t_i^*) $$where $R$ is the $smooth \\ L_1$ function. The $p_i^*L_{reg}$ term in the formula above means that the regression loss is only computed when the $anchor$ is positive. The outputs of the $cls$ and $reg$ layers consist of $\\{p_i\\}$ and $\\{t_i\\}$, respectively.\nThe two losses are normalized by $N_{cls}$ and $N_{reg}$ respectively, with $\\lambda$ controlling the balance. But the authors show experimentally that the model is not sensitive to $\\lambda$ over a wide range, and that the normalization mentioned above is not essential and can be simplified.\nBounding box regression is done in the same way as in $RCNN$, and here the ground truth bounding boxes are also parameterized as $(t_x^*, t_y^*, t_w^*, t_h^*)$:\n$$ \\begin{aligned} t_x \u0026 = (x-x_a)/w_a \\\\ t_y \u0026 = (y-y_a)/h_a \\\\ t_w \u0026 = log(w/w_a) \\\\ t_h \u0026 = log(h/h_a) \\\\ t_x^* \u0026 = (x^*-x_a)/w_a \\\\ t_y^* \u0026 = (y^*-y_a)/h_a \\\\ t_w^* \u0026 = log(w^*/w_a) \\\\ t_h^* \u0026 = log(h^*/h_a) \\end{aligned} $$Using the logarithm to represent the difference in width and height also has the effect of converging quickly when the difference is large and more slowly (to guarantee accuracy) when the difference is small.\nModel training For the detection network the authors use $Fast \\ RCNN$. During training the $RPN$ and $Fast \\ RCNN$ are trained separately. There are three methods for training such a network with shared features:\n$Alternating \\ training$. First the $RPN$ is trained, then the candidate boxes are used to train $Fast \\ RCNN$; after being fine-tuned by $Fast \\ RCNN$ the network is used in turn to initialize the $RPN$, and this process can be iterated. $Approximate \\ joint \\ training$. This method fuses $RPN$ and $Fast \\ RCNN$ into a single network for training. In each $SGD$ iteration, the forward pass through the $RPN$ does not change its parameters and precomputed candidate boxes are used to train $Fast \\ RCNN$, while the backward pass proceeds normally. For the backward pass of the shared convolution layers, the propagated information comes from both the $RPN$ loss and the $Fast \\ RCNN$ loss. This method ignores the derivative information with respect to the candidate box coordinates. $Non-approximate \\ joint \\ training$. This method fixes the flaw of the previous one and includes the candidate box coordinate information. It requires a $RoI \\ pooling$ layer that is differentiable with respect to the box coordinates, which can be solved with a $RoI \\ warping$ layer. The authors adopt $4-step \\ Alternating \\ Training$, with the four steps as follows:\nTrain the $RPN$, initializing the network with an $ImageNet$ pretrained model and fine-tuning it end to end with the region proposal loss; Train $Fast \\ RCNN$ using the candidate boxes generated in the first step; this network is also initialized with the $ImageNet$ pretrained model. Up to this point the two networks do not share the computation of the convolution layers; Train the $RPN$ using the detection network, while freezing the convolution layers and only adjusting the layers belonging to the $RPN$ model; Keeping the convolution layers frozen, fine-tune the layers belonging to the $Fast \\ RCNN$ model. In this way both networks share the computation of the convolution layers. Both $Faster \\ RCNN$ and $Fast \\ RCNN$ belong to the $two \\ stage$ algorithms. The characteristic of such algorithms is that the algorithm first generates a series of candidate boxes as samples, and then performs classification and regression on these boxes. This gives high accuracy but slow speed. The single-stage algorithms ($one \\ stage$) instead turn the object localization problem into a regression problem, gaining considerably more speed but losing accuracy.\nReference There are many details about the $Faster \\ RCNN$ model that are not mentioned here; refer to the original paper and the related code open-sourced by the authors.\nThis post is mainly based on the following articles:\nFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks\nUnderstanding Faster RCNN in one article\n","permalink":"https://simplestory.github.io/en/posts/20190922_5e8040095d43e97ccedd09771f9eab38/","summary":"\u003cblockquote\u003e\n\u003cp\u003eThis feeling of posting several articles in a row is very exciting, very refreshing. Without further ado, as the title says, this is a faster and stronger object detection algorithm, and it also comes from Ross Girshick\u0026hellip; Honestly, this guy spent several papers in a row criticizing the models he had written before — impressive. Faster Rcnn adopts an efficient algorithm to generate candidate boxes, and also introduces the concept of Anchor.\u003c/p\u003e\n\u003c/blockquote\u003e","title":"Faster RCNN"},{"content":" Ha, since I happen to have some free time these days, let me quickly summarize the RCNN series papers I read before; this is the second post in the series. The author Ross Girshick could not stand how slowly the model he had proposed earlier (RCNN) ran, so he proposed an improved model (Fast RCNN). Below is a brief analysis of that model.\nThe slowness of RCNN Compared with image recognition tasks, object detection is considerably harder because of the extra step of localizing the object positions. There are two main problems to solve: one is that the model needs a large number of object candidate boxes; the other is that these candidate boxes only roughly provide the approximate positions of the objects, so the model has to perform a regression operation. $RCNN$ is a multi-stage training model. The $RCNN$ model first fine-tunes a convolutional neural network on the object candidate boxes using a log loss, then passes the resulting feature maps to $SVM$s for object classification; these $SVM$s replace the $softmax$ layer that would otherwise need fine-tuning. In the third training stage the model learns bounding box regression. For training the $SVM$s and the bounding box regression, $RCNN$ writes the features of every object candidate box to disk, which undoubtedly costs an enormous amount of time and space. Secondly, the speed of object recognition is too slow: on a GPU, recognizing an image takes about 47 seconds per image. Another important reason for the slowness of $RCNN$ is that the model does not share the computation of the convolution layers.\nBasic structure $Fast \\ RCNN$ is a single-stage training model. It takes the whole image and a set of candidate boxes as input; the model first extracts features from the whole image with a set of convolution layers and max pooling layers, then, on the extracted feature map, uses a $RoI \\ pooling$ layer to extract a fixed-length feature vector for each candidate box, and then feeds it into fully connected layers that finally split into two branches for object recognition and bounding box regression respectively. The concrete framework is as follows:\n$RoI \\ pooling$ divides an $h \\times w$ $RoI$ region into sub-windows of $h/H \\times w/W$, then applies $max \\ pooling$ on each sub-window to output a fixed-length feature vector.\nPretraining and fine-tuning the network For network initialization, the authors use three pretrained $ImageNet$ models and make the following changes:\nThe last $max \\ pooling$ layer of the original network is replaced by a $RoI \\ pooling$ layer with the parameters $H \\times W$ specified to match the size of the network\u0026rsquo;s first fully connected layer; The last fully connected network and $softmax$ layer of the original network are replaced by two branch networks; The original network is modified to have two inputs The authors propose an efficient model training method that has the advantage of shared features. In $Fast \\ Rcnn$ training, the $minibatch$ of stochastic gradient descent is sampled hierarchically: first $N$ samples are sampled, then $R/N$ $RoI$s are sampled from each sample. The $RoI$s coming from the same image share the computation and memory of the forward and backward passes.\nThe loss function used by the model is a multi-task loss function. $Fast \\ RCNN$ has two output layers: one outputs the probabilities $p=(p_0,\\dots,p_K)$ over $K+1$ classes for each $RoI$, and the other outputs the bounding box regression offsets $t^k=(t_x^k,t_y^k,t_w^k,t_h^k)$ over the $K$ classes. Each $RoI$ is labeled with a ground truth class $u$ and a ground truth bounding box $v$ of that class. The authors use a multi-task loss function $L$ on each labeled $RoI$ to jointly train classification and bounding box regression:\n$$ L(p,u,t^u,v) = L_{cls}(p,u)+\\lambda[u \\ge 1]L_{loc}(t^u,v) $$where $L_{cls}(p,u)=-logp_u$ is the log loss, and the second loss term is composed of the ground truth regression box $v=(v_x,v_y,v_w,v_h)$ and the predicted regression box $t^u=(t_x^u,t_y^u,t_w^u,t_h^u)$; $L_{loc}$ ignores background $RoI$ classes, so for bounding box regression we have:\n$$ L_{loc}(t^u,v) = \\sum_{t \\in \\{x,y,w,h\\}} smooth_{L_1}(t_i^u-v_i) $$And the $smooth$ function is:\n$$ smooth_{L_1}(x) = \\begin{cases} 0.5x^2 \u0026\\text{if $\\vert x \\vert\\lt 1$} \\\\ \\vert x \\vert-0.5 \\ \u0026\\text{otherwise} \\end{cases} $$Using the $L_1$ loss here is less sensitive than the $L_2$ loss used by $RCNN$. The hyperparameter $\\lambda$ in the multi-task loss function controls the balance between the two individual loss functions. In the paper\u0026rsquo;s experiments the authors normalized the ground truth bounding boxes (to have mean 0 and variance 1), and chose the hyperparameter in the loss function to be 1.\nI will not go into the sampling of the $minibatch$ in detail here; see the authors\u0026rsquo; paper. Next let me talk about the backward propagation of the $RoI \\ pooling$ layer. Let $x_i \\in R$ be the i-th input of $RoI \\ pooling$ and $y_{rj}$ be the j-th output of the r-th $RoI$; then the computation of $RoI \\ pooling$ can be written as\n$$ y_{rj}=x_{i^*(i,j)} $$where $i^*(r,j)=argmax_{i^\\prime\\in R(r,j)}x_i^\\prime$, and $R(r,j)$ is the index set of the input sub-window. A single $x_i$ may correspond to several different outputs $y_{rj}$. Its backward propagation form is:\n$$ \\frac{\\partial{L}}{\\partial{x_i}} = \\sum_r\\sum_j[i=i^*(r,j)]\\frac{\\partial{L}}{\\partial{y_{rj}}} $$In the backward propagation of the $RoI$ pooling layer, the gradient of the loss function $L$ with respect to the input node $x_i$ is the accumulation of the gradients of $L$ with respect to the outputs $y_{rj}$ of each possible candidate region $r$ ($x_i$ is selected as the maximum by the j-th output of the r-th candidate region). The indicator function in the formula above is specifically:\n$$ [i=i^*(r,j)] = \\begin{cases} 1, \u0026 i=i^*(r,j) \\ge 1 \\\\ 0, \u0026 \\text{otherwise} \\end{cases} $$For whole-image classification, the time consumed by the fully connected layers is less than that of the convolution layers, but in this model a large number of $RoI$s are used for object detection, so the computation of the fully connected layers accounts for roughly $50\\%$ of the overall model. So the authors apply $SVD$ truncation to the large fully connected layers. A fully connected layer can be parameterized by a weight matrix $W$ of size $u \\times v$, and applying $SVD$ to this matrix gives:\n$$ W \\approx U\\Sigma_tV^T $$where $U$ is the $u \\times t$ left singular matrix of $W$, $\\Sigma_t$ is the $t \\times t$ diagonal matrix of singular values of $W$, and $V$ is the $v \\times t$ right singular matrix of $W$. This simple compression operation yields a good speedup.\nAbout the model Later in the paper the authors also discuss the model in the form of questions. The first is the multi-task loss: it avoids designing a pipelined sequential training model, and the experimental results also show that this design performs better than multi-stage training. The $Fast \\ RCNN$ model also replaces the $SVM$s of the $RCNN$ model with a $softmax$ layer; the $softmax$ layer introduces competition between classes when predicting scores for $RoI$s, so the final result is better than with $SVM$s. Finally, one point to note: more candidate boxes do not mean better model performance — the authors found experimentally that sparse object candidate boxes can improve model performance.\nReference This post refers to the following paper:\nFast R-CNN\n","permalink":"https://simplestory.github.io/en/posts/20190921_817fe11366b1149fdc44a5a4c92685c4/","summary":"\u003cblockquote\u003e\n\u003cp\u003eHa, since I happen to have some free time these days, let me quickly summarize the RCNN series papers I read before; this is the second post in the series. The author Ross Girshick could not stand how slowly the model he had proposed earlier (RCNN) ran, so he proposed an improved model (Fast RCNN). Below is a brief analysis of that model.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"the-slowness-of-rcnn\"\u003eThe slowness of RCNN\u003c/h2\u003e\n\u003cp\u003eCompared with image recognition tasks, object detection is considerably harder because of the extra step of localizing the object positions. There are two main problems to solve: one is that the model needs a large number of object candidate boxes; the other is that these candidate boxes only roughly provide the approximate positions of the objects, so the model has to perform a regression operation. $RCNN$ is a multi-stage training model. The $RCNN$ model first fine-tunes a convolutional neural network on the object candidate boxes using a log loss, then passes the resulting feature maps to $SVM$s for object classification; these $SVM$s replace the $softmax$ layer that would otherwise need fine-tuning. In the third training stage the model learns bounding box regression. For training the $SVM$s and the bounding box regression, $RCNN$ writes the features of every object candidate box to disk, which undoubtedly costs an enormous amount of time and space. Secondly, the speed of object recognition is too slow: on a GPU, recognizing an image takes about 47 seconds per image. Another important reason for the slowness of $RCNN$ is that the model does not share the computation of the convolution layers.\u003c/p\u003e","title":"Fast RCNN"},{"content":" Object recognition is much more complex than plain image classification: the algorithm model has to identify which objects are where in the image and draw boxes around them. Ross Girshick proposed an algorithm model that uses a convolutional neural network together with a support vector machine for object detection. Below is a brief analysis of this classic model.\nBasic structure The first problem is object localization, which the authors solve through the region recognition approach, using $selective \\ search$. The second problem is the scarcity of training data; the paper proposes supervised pre-training on a large auxiliary dataset, followed by fine-tuning of specific parts on a small dataset. The experimental results also confirm that this approach is effective. The rough structure is as follows:\nIt mainly consists of three parts:\nGenerating class-independent candidate regions; A convolutional network used to extract a fixed-length feature vector from each region; A set of class-specific linear SVMs Module design For selecting candidate regions there are many implementations, such as $objectness$, $selective \\ search$, $category-independent \\ object \\ proposal$ and so on. The authors chose $selective \\ search$; here it must be said that the way $R-CNN$ proposes candidate boxes is not very reasonable, because the $selective \\ search$ method is simply too slow and seriously slows the model down. For feature extraction, the authors use a $CNN$ to extract a fixed-length feature from each candidate region. First, all the pixels of the tight bounding box around the candidate region are warped to the input size suitable for the $CNN$ network; before that, the authors expand the tight bounding box so that, at the warped size, the original surrounding warped image context has exactly enough pixels of the box. In the recognition stage, the score of each candidate box is obtained through the $SVM$, and then a greedy non-maximum suppression ($NMS$) is used to consolidate the results. Considering that the training data is too large, the authors also use standard hard negative mining to improve model performance.\nThe training process of the model is also divided into three parts:\nCNN fine-tuning SVM training Bounding box regression training Since the training of the first two is basically the same as before, I will not go into detail here; let me mainly mention the bounding box regression training.\nThe input of this part is $N$ training data pairs $\\{(P^i, G^i)\\}_{i=1,\\dots,N}$, where $P^i=(P^i_x,P^i_y,P^i_w,P^i_h)$ denotes the pixel coordinates of the center of the candidate box together with its length and width, and likewise $G=(G_x,G_y,G_w,G_h)$ corresponds to the ground-truth object bounding box. Our goal is to learn a mapping from candidate box $P$ to ground-truth box $G$. Define four variables $d_x(P)$, $d_y(P)$, $d_w(P)$, $d_h(P)$; the first two specify a scale-invariant translation of the center of the bounding box of $P$, and the last two specify the translation of the width and height of the bounding box of $P$ in log space. Then the mapping from candidate box $P$ to the predicted target box $\\hat{G}$ can be written as:\n$$ \\begin{aligned} \\hat{G}_x \u0026 = P_wd_x(P)+P_x \\\\ \\hat{G}_y \u0026 = P_hd_y(P)+P_y \\\\ \\hat{G}_w \u0026 = P_wexp(d_w(P)) \\\\ \\hat{G}_h \u0026 = P_hexp(d_h(P)) \\end{aligned} $$This can be treated as a linear regression problem: each of $d_x(P)$, $d_y(P)$, $d_w(P)$, $d_h(P)$ is modeled by the feature vector $\\phi(P)$ from the candidate box pooling layer of the convolutional network, that is:\n$$ d_*(P)=w^T_*\\phi(P) $$where $w_*$ are the parameters of the model, which can be learned through the regularized least squares objective (ridge regression):\n$$ w_*=argmax_{\\hat{w}_*}\\sum_i^N(t_*^i-\\hat{w}_*^T\\phi(P^i))^2+\\lambda\\vert\\vert\\hat{w}_*\\vert\\vert^2 $$For this paper, the regression target $t_*$ is defined as:\n$$ \\begin{aligned} t_x \u0026 = (G_x-P_x)/P_w \\\\ t_y \u0026 = (G_y-P_y)/P_h \\\\ t_w \u0026 = log(G_w/P_w) \\\\ t_h \u0026 = log(G_h/P_h) \\end{aligned} $$That is, after inputting $\\phi$, $(t_x,t_y,t_w,t_h)$ is used to correct the position of the candidate box.\nReference This post refers to the following paper:\nRegion-based Convolutional Networks for Accurate Object Detection and Segmentation\n","permalink":"https://simplestory.github.io/en/posts/20190919_13243638926ee79bc7810c80a460408c/","summary":"\u003cblockquote\u003e\n\u003cp\u003eObject recognition is much more complex than plain image classification: the algorithm model has to identify which objects are where in the image and draw boxes around them. Ross Girshick proposed an algorithm model that uses a convolutional neural network together with a support vector machine for object detection. Below is a brief analysis of this classic model.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"basic-structure\"\u003eBasic structure\u003c/h2\u003e\n\u003cp\u003eThe first problem is object localization, which the authors solve through the region recognition approach, using $selective \\ search$. The second problem is the scarcity of training data; the paper proposes supervised pre-training on a large auxiliary dataset, followed by fine-tuning of specific parts on a small dataset. The experimental results also confirm that this approach is effective. The rough structure is as follows:\u003c/p\u003e","title":"RCNN"},{"content":" First of all, happy Teachers\u0026rsquo; Day to all teachers, and thank you for your spirit of giving roses to others with a lingering fragrance on your hands. Then, after almost a month, I have rolled back to update the blog again; before that I was busy with joining the company, hoping to fit into the company\u0026rsquo;s development work as soon as possible, so I was \u0026ldquo;busy\u0026rdquo; for a month. Next let us continue with the introduction of classic image recognition network models. In previous posts we already introduced the AlexNet and VGG networks, both of which pushed the depth of neural networks of that time to the extreme, and subsequent neural network models basically performed no better. This is because as the network gets deeper, during error backpropagation the weights of the earlier network layers basically do not get updated and stay around some fixed value. In 2015, Kaiming He proposed a shortcut connection that propagates the error to the earlier network layers, greatly improving network performance. Below is a brief analysis of the ResNet network.\nShortcut connections Increasing the number of neural network layers should in theory give a better performing model, but the existence of vanishing gradients limits how deep the network can go. For this reason, Kaiming He proposed a network building block that solves the problem of gradients not being able to propagate to the bottom layers of the network, namely the shortcut connection. The basic structure is shown below:\nThis structure is in fact a residual network. Here $x$ denotes the input of this building block and $\\mathcal{H}(x)$ denotes the mapping function of the building block. Assuming a multi-layer nonlinear network structure can fit any complex function, that is, this structure can fit the residual function $\\mathcal{H}(x)-x$, it is better to fit the residual function $\\mathcal{F}(x) := \\mathcal{H}(x)-x$ than to fit the original mapping function $\\mathcal{H}(x)$, because fitting the original function still cannot avoid the vanishing gradient phenomenon, and from the figure we can see that implementing the residual function is very simple.\nThere is a very important component in the shortcut connection, namely the $Identity \\ Mapping$; the building block in the figure above can be written as: $y = \\mathcal{F}(x,\\{\\mathcal{W}_i\\})+x$, where $x$ and $y$ denote the input and output of the block respectively (before the activation function), and $\\mathcal{F}(x,\\{\\mathcal{W}_i\\})$ denotes the function mapping learned by the residual block. Expanding the figure above gives: $\\mathcal{F} = W_2\\sigma(W_1x)$, where $\\sigma$ denotes the $ReLU$ activation function and the bias values are ignored here. This connection introduces no additional parameters and does not increase the computation of the model. There is one point to note: $x$ and the output of $\\mathcal{F}$ must have the same dimension; if they do not, the dimension can be matched with a linear projection $W_s$: $y = \\mathcal{F}(x,\\{\\mathcal{W}_i\\})+W_sx$.\nBasic structure The authors set up three groups of network models for the $ImageNet$ dataset, taking these as examples to explain the structure of $ResNet$\nPlain convolutional neural network The design of the basic network mostly comes from the $VGG$ network; the convolution kernels of the convolution layers are basically $3 \\times 3$ and follow the following two simple design principles:\nFor network layers with the same output size, set the same number of convolution kernels; If the size of the feature map is halved, double the number of convolution kernels to reduce the computational complexity of each layer For convolution layers with stride 2, the authors perform downsampling directly in that layer. The overall network ends with a global average pooling layer and a 1000-way fully connected layer with $softmax$. The network model has $34$ weight layers.\nConvolutional neural network with residual blocks It is modified from the plain neural network designed before by adding shortcut connections. Shortcut connections can be used directly when the dimensions before and after the connection are the same; for the case of different dimensions, there are the following two ways of handling it:\nUse zero padding to expand the dimension for matching, which introduces no additional parameters; Use a $1 \\times 1$ convolution kernel to expand the dimension The concrete structure figure is as follows:\nThe left side is the $VGG$ network (used as a reference), the middle is the plain convolutional neural network, and the right is the convolutional neural network with residual blocks; the dashed shortcut connections indicate that the dimension is increased during the connection.\nIn later experiments the authors also adopt many variants of the $ResNet$ network, with the concrete structure as follows:\nIdentity Mapping Not long after publishing the $ResNet$ paper, the authors published a paper on the mathematical principles of the $Identity \\ Mapping$ in the shortcut connections of the $ResNet$ network to further confirm the reliability of this model. Here is a rough analysis.\nA residual block can be written in the following form:\n$$ \\begin{aligned} y_l \u0026 = h(x)+\\mathcal{F}(x_l,\\mathcal{W}_l) \\\\ x_{l+1} \u0026 = f(y_l) \\end{aligned} $$where $x_l$ and $x_{l+1}$ are the input and output of the $l$-th residual block, $\\mathcal{F}$ is the residual function and $f$ is the $ReLU$ activation function, with $\\mathcal{W}_l$ defined as follows:\n$$ \\mathcal{W}_l=\\{W_{l,k}\\vert 1\\le k\\le K\\} $$$l$ is the weight of the $l$-th block (including the bias values) and $K$ is the number of network layers in the residual block.\nTo prove the correctness of the $Identity \\ Mapping$ choice, the authors made several assumptions here, and to simplify the analysis they also unified the $f$ function (all being $Identity \\ Mapping$), so that $x_{l+1}=y_l$.\nLet $h(x_l)=x_l$, that is, $Identity \\ Mapping$ From the assumption we obtain:\n$$ x_{l+1} = x_l+\\mathcal{F}(x_l, \\mathcal{W}_l) $$Recursively:\n$$ \\begin{aligned} x_{l+2} \u0026 = x_{l+1}+\\mathcal{F}(x_{l+1},\\mathcal{W}_{l+1}) \\\\ \u0026 = x_l+\\mathcal{F}(x_l,\\mathcal{W}_l)+\\mathcal{F}(x_{l+1},\\mathcal{W}_{l+1}) \\\\ \u0026 = \\dots \\end{aligned} $$which simplifies to:\n$$ x_L = x_l+\\sum_{i=1}^{L-1}\\mathcal{F}(x_i,\\mathcal{W}_i) $$This form has two benefits:\nThe feature $x_L$ of a residual block at any depth can be expressed by any shallow residual block $l$ plus the residual terms $\\sum_{i=1}^{L-1}\\mathcal{F}$, that is, the model is in a residual fashion between any blocks $L$ and $l$; For a residual block $L$ at any depth, the equation $x_L = x_0+\\sum_{i=0}^{L-1}\\mathcal{F}(x_i,\\mathcal{W}_i)$ sums the outputs of all preceding residual functions, unlike the matrix-vector product of ordinary network models (Addition: the matrix-vector product form of an ordinary network model is: $x_L=\\prod_{i=0}^{L-1}\\mathcal{W}_ix_0$)\nThis form also performs very well in backpropagation. Let $\\epsilon$ be the loss function; then:\n$$ \\begin{aligned} \\frac{\\partial{\\epsilon}}{\\partial{x_l}} \u0026 = \\frac{\\partial{\\epsilon}}{\\partial{x_L}}\\cdot\\frac{\\partial{x_L}}{\\partial{x_l}} \\\\ \u0026 = \\frac{\\partial{\\epsilon}}{\\partial{x_L}}\\left(1+\\frac{\\partial{}}{\\partial{x_l}}\\sum_{i=1}^{L-1}\\mathcal{F}(x_i,\\mathcal{W}_i)\\right) \\end{aligned} $$Here $\\frac{\\partial{\\epsilon}}{\\partial{x_L}}$ propagates directly without passing through weight layers, while the other term propagates through weight layers. For $\\frac{\\partial{}}{\\partial{x_l}}\\sum_{i=1}^{L-1}\\mathcal{F}(x_i,\\mathcal{W}_i)$, the value over a $minibatch$ is not always -1, that is, vanishing gradients do not occur (because $\\frac{\\partial{\\epsilon}}{\\partial{x_l}}$ will not be zero).\nLet $h(x_l)=\\lambda_lx_l$, where $\\lambda_l$ is a modulation parameter Likewise from the assumption we obtain $x_{l+1}=\\lambda_lx_l+\\mathcal{F}(x_{l},\\mathcal{W}_{l})$, and recursively\n$$ x_L=(\\prod_{i=1}^{L-1}\\lambda_i)x_l+\\sum_{i=1}^{L-1}(\\prod_{j=i+1}^{L-1}\\lambda_j)\\mathcal{F}(x_i,\\{\\mathcal{W}_i\\}) $$Simplification gives the following:\n$$ x_L=(\\prod_{i=l}^{L-1}\\lambda_i)x_l+\\sum_{i=1}^{L-1}\\hat{\\mathcal{F}}(x_i,\\mathcal{W}_i) $$where $\\hat{\\mathcal{F}}(x_i,\\mathcal{W}_i)$ absorbs the scalar. Backpropagating the formula above gives:\n$$ \\frac{\\partial{\\epsilon}}{\\partial{x_l}} = \\frac{\\partial{\\epsilon}}{\\partial{x_L}}\\left((\\prod_{i=l}^{L-1}\\lambda_i)+\\frac{\\partial{}}{\\partial{x_l}}\\sum_{i=l}^{L-1}\\hat{\\mathcal{F}}(x_i,\\mathcal{W}_i)\\right) $$When the network is very deep ($L$ is large), if $\\lambda_i \\lt 1$ for all $i$, the factor $\\prod_{i=l}^{L-1}\\lambda_i$ grows explosively; if $\\lambda_i \\gt 1$ for all $i$, the factor $\\prod_{i=l}^{L-1}\\lambda_i$ becomes very small or even zero — both of which hinder the network\u0026rsquo;s learning.\nChoosing the activation function Above we argued for the importance of $Identity \\ Mapping$, and this was based on an important assumption, namely that the activation function $f$ is an $Identity \\ Mapping$. So next the authors discuss the case where the activation function takes other forms. The concrete detail figures are as follows:\nIn the figure, $a$ is the reference block and keeps the original structure, with $ReLU$ as the only activation; $b$ places the $BN$ layer after the addition operation, that is, the activation includes the $BN$ layer and $ReLU$; the authors\u0026rsquo; experiments show that this structure is less expressive than the original one, because the $BN$ layer in this structure affects the information flow; $c$ places $ReLU$ before the addition operation, so the activation becomes an $Identity \\ Mapping$, but this makes the output of the function $\\mathcal{F}$ non-negative, whereas intuitively the function $\\mathcal{F}$ should take values in $(+\\infty, -\\infty)$, so the result is that the forward-propagated signal of the residual block increases monotonically, which affects the expressiveness of the model.\nImproving the residual block For the original structure we have:\n$$ \\begin{aligned} y_l \u0026 = h(x_l)+\\mathcal{F}(x_l,\\mathcal{W}_l) \\\\ x_{l+1} \u0026 = f(y_l) \\end{aligned} $$For the next block we have:\n$$ \\begin{aligned} y_{l+1} \u0026 = h(x_{l+1})+\\mathcal{F}(x_{l+1},\\mathcal{W}_{l+1}) \\\\ \u0026 = x_{l+1}+\\mathcal{F}(x_{l+1},\\mathcal{W}_{l+1}) \\\\ \u0026 = f(y_l)+\\mathcal{F}(f(y_l),\\mathcal{W}_{l+1}) \\end{aligned} $$From the formula we can see that the function $x_{l+1}=f(y_l)$ affects both this block and the next block.\nThe authors propose an asymmetric structure to solve this problem, with the concrete structure as follows:\nexpressed as:\n$$ x_{l+1} = x_l+\\mathcal{F}(\\hat{f}(x_l), \\mathcal{W}_l) $$From the figure we can also see that $post-activation$ and $pre-activation$ are equivalent; the mathematical expression is similar to $x_L = x_l+\\sum_{i=1}^{L-1}\\mathcal{F}(x_i,\\mathcal{W}_i)$, so the backward propagation form is similar too, that is, there is no vanishing or exploding gradient, and the activation function in the structure above takes the $Identity \\ Mapping$ form.\nChoosing the position of the activation function after the improvement For the position of the activation function in the improved residual block, the authors considered two structures, as follows:\nExperiments show that the structure shown in figure $e$ achieves better performance than the others, mainly for two reasons:\nThe activation function is an $Identity \\ Mapping$. The authors found experimentally that the closer $h(x_l)$ and the activation function $f$ in the shortcut connection block are to $Identity \\ Mapping$, the better the model\u0026rsquo;s representational ability; Using a $BN$ layer in the pre-activation improves the regularization ability of the model. Although the original structure also has a $BN$ layer for regularization, the value passed through the shortcut connection is added afterwards, so the output of the whole block is not regularized, whereas pre-activation regularizes both the input and the output of the block. This structure is also easy to optimize: the activation function $f=ReLU$ of the original block affects signal propagation in the negative case, and this effect strengthens as the number of residual blocks grows. The activation function of the improved block uses $Identity \\ Mapping$, which can propagate the error directly to the next layer.\nReference This post refers to the two papers published by Kaiming He in 2015 and 2016:\nDeep Residual Learning for Image Recognition\nIdentity Mappings in Deep Residual Networks\n","permalink":"https://simplestory.github.io/en/posts/20190910_cefb8aa8290a0b4cfddf3f88bf5a2a3c/","summary":"\u003cblockquote\u003e\n\u003cp\u003eFirst of all, happy Teachers\u0026rsquo; Day to all teachers, and thank you for your spirit of giving roses to others with a lingering fragrance on your hands. Then, after almost a month, I have rolled back to update the blog again; before that I was busy with joining the company, hoping to fit into the company\u0026rsquo;s development work as soon as possible, so I was \u0026ldquo;busy\u0026rdquo; for a month. Next let us continue with the introduction of classic image recognition network models. In previous posts we already introduced the AlexNet and VGG networks, both of which pushed the depth of neural networks of that time to the extreme, and subsequent neural network models basically performed no better. This is because as the network gets deeper, during error backpropagation the weights of the earlier network layers basically do not get updated and stay around some fixed value. In 2015, Kaiming He proposed a shortcut connection that propagates the error to the earlier network layers, greatly improving network performance. Below is a brief analysis of the ResNet network.\u003c/p\u003e\n\u003c/blockquote\u003e","title":"ResNet"},{"content":" Neural networks already have several normalization algorithms. They all look very similar and basically share a unified form. This post refers to the Zhihu column SIGAI\u0026rsquo;s article on normalization algorithms; the link is at the bottom of the post.\nThe basic problem When using a neural network to learn from a dataset, because the sources of the data may differ considerably, the values of different attributes may also span very different ranges; all of this causes errors in how the data is treated and leads to poor learning by the network. To solve this problem we normalize the data, that is, we regularize data whose value ranges vary widely into the same scale, which is beneficial for the network\u0026rsquo;s learning. Up to now normalization has many variants, the common ones being Batch Normalization (BN), Layer Normalization (LN), Instance Normalization (IN) and Group Normalization (GN). In fact their processing is similar (subtract the mean, divide by the standard deviation, then apply a linear mapping); the difference lies in the dimension along which the statistics are computed. The concrete formula is as follows:\n$$ y = \\gamma \\left(\\frac{x-\\mu(x)}{\\sigma(x)}\\right)+\\beta $$Note there is one more difference here: for $BN$, $IN$ and $GN$, the parameters $\\gamma$ and $\\beta$ are vectors whose dimension equals the number of channels $C$, whereas for $LN$ both parameters are matrices whose dimension equals the $normalized\\ shape$.\nFor convenience in the following explanation, the dimension of the dataset is defined in advance as: $x \\in R^{N \\times C \\times H \\times W}$, which specifically means that the dataset contains $N$ samples, each with $C$ channels, height $H$ and width $W$.\nBatch Normalization This method is the earliest normalization algorithm and its effect is also good — arguably the best. The computation dimension is $N \\times H \\times W$, keeping the number of channels. The concrete formulas are as follows:\n$$ \\begin{aligned} \\mu_c(x) \u0026 = \\frac{1}{NHW}\\sum_{n=1}^N\\sum_{h=1}^H\\sum_{w=1}^W x_{nchw} \\\\ \\sigma_c(x) \u0026 = \\sqrt{\\frac{1}{NHW}\\sum_{n=1}^N\\sum_{h=1}^H\\sum_{w=1}^W (x_{nchw}-\\mu_c(x))^2+\\epsilon} \\end{aligned} $$That is, keeping the number of channels fixed, the first channel of the first sample is added to the first channel of the second sample and so on up to the first channel of the $N$th sample, and dividing by $N\\times H \\times W$ gives a single number, which is the mean of the first channel (not an $H\\times W$ matrix); the standard deviation is computed in the same way.\n$BN$ has a drawback: it requires a large amount of data to reasonably estimate the mean and variance of the training data, which may easily lead to insufficient memory, and for $RNN$ models whose training sequences have variable lengths, $BN$ is also hard to apply.\nNote one point here: when a BN layer is used in the network, there is no need to use a Dropout layer any more\nLayer Normalization $LN$ has a big advantage: it does not need batch training, and can normalize a single sample on its own. The computation dimension is $C \\times H \\times W$, keeping the number of samples. The concrete formulas are as follows:\n$$ \\begin{aligned} \\mu_n(x) \u0026 = \\frac{1}{CHW}\\sum_{c=1}^C\\sum_{h=1}^H\\sum_{w=1}^W x_{nchw} \\\\ \\sigma_n(x) \u0026 = \\sqrt{\\frac{1}{CHW}\\sum_{c=1}^C\\sum_{h=1}^H\\sum_{w=1}^W (x_{nchw}-\\mu_n(x))^2+\\epsilon} \\end{aligned} $$That is, keeping the number of samples fixed, the data corresponding to the channels, height and width of the sample is summed and divided by $C \\times H \\times W$ to obtain the mean of that sample; the standard deviation is analogous.\nInstance Normalization In generative models, the mean and standard deviation of each channel in the dataset affect the style of the final generated image, so the authors of $IN$ believe that an image can first be normalized at the channel level and then normalized again with the mean and standard deviation of the target style image, thus obtaining the style of the target image. $IN$ also does not need batch training and can normalize a single sample on its own. The computation dimension is $H \\times W$, keeping the number of samples and channels. The concrete formulas are as follows:\n$$ \\begin{aligned} \\mu_{nc}(x) \u0026 = \\frac{1}{HW}\\sum_{h=1}^H\\sum_{w=1}^W x_{nchw} \\\\ \\sigma_{nc}(x) \u0026 = \\sqrt{\\frac{1}{HW}\\sum_{h}^H\\sum_{w=1}^W (x_{nchw}-\\mu_{nc}(x))^2+\\epsilon} \\end{aligned} $$That is, keeping the number of samples and channels fixed, the sum of the data over height and width for a single channel of a single sample divided by $H \\times W$ is the required mean.\nGroup Normalization $GN$ is mainly applied to learning tasks that consume a lot of GPU memory, such as image segmentation. Because such tasks are limited by memory size, the $batchsize$ is usually a single-digit number, and $BN$ simply cannot work well. $GN$ is a compromise between $LN$ and $IN$ and can also normalize a single sample. It divides the channels into $G$ groups, each with $C/G$ channels, then computes the mean and standard deviation of the elements in each group and independently normalizes the channels of the corresponding group with the obtained parameters. The concrete formulas are as follows:\n$$ \\begin{aligned} \\mu_{ng}(x) \u0026 = \\frac{1}{(C/G)HW}\\sum_{c=gC/G}^{(g+1)C/G}\\sum_{h=1}^H\\sum_{w=1}^W x_{nchw} \\\\ \\sigma_{ng}(x) \u0026 = \\sqrt{\\frac{1}{(C/G)HW}\\sum_{c=gC/G}^{(g+1)C/G}\\sum_{h=1}^H\\sum_{w=1}^W (x_{nchw}-\\mu_{ng}(x))^2+\\epsilon} \\end{aligned} $$Reference This post mostly refers to the following article:\nZhihu column SIGAI (author: Yi Xiangnan)\n","permalink":"https://simplestory.github.io/en/posts/20190811_58bd041ca5bf7b009cdc8d07392dc061/","summary":"\u003cblockquote\u003e\n\u003cp\u003eNeural networks already have several normalization algorithms. They all look very similar and basically share a unified form. This post refers to the Zhihu column SIGAI\u0026rsquo;s article on normalization algorithms; the link is at the bottom of the post.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"the-basic-problem\"\u003eThe basic problem\u003c/h2\u003e\n\u003cp\u003eWhen using a neural network to learn from a dataset, because the sources of the data may differ considerably, the values of different attributes may also span very different ranges; all of this causes errors in how the data is treated and leads to poor learning by the network. To solve this problem we normalize the data, that is, we regularize data whose value ranges vary widely into the same scale, which is beneficial for the network\u0026rsquo;s learning. Up to now normalization has many variants, the common ones being Batch Normalization (BN), Layer Normalization (LN), Instance Normalization (IN) and Group Normalization (GN). In fact their processing is similar (subtract the mean, divide by the standard deviation, then apply a linear mapping); the difference lies in the dimension along which the statistics are computed. The concrete formula is as follows:\u003c/p\u003e","title":"Telling Apart Several Normalization Algorithms"},{"content":" VGG (Visual Geometry Group) is another classic convolutional neural network model after AlexNet. It won second place in the 2014 ILSVRC competition (first place went to GoogleLeNet), and the variants used most often in work and personal projects are VGG16 and VGG19, where 16 and 19 denote the number of layers in the network. Following on from the AlexNet model discussed previously, this post analyzes the VGG model.\nBasic architecture The training input images are $224 \\times 224$ RGB images, and the only preprocessing is subtracting the mean RGB value of the training set from every pixel. The convolution layers use a kernel size of $3 \\times 3$ with stride 1, and some convolution kernels use a $1 \\times 1$ size; the max pooling layers use a kernel size of $2 \\times 2$ with stride 2. Of the three fully connected layers at the end, the first two have 4096 channels, followed by a $softmax$ layer; the hidden layer activation function is $ReLU$, and the LRN local response normalization used in AlexNet is not used. The detailed structure is shown in the table below:\nAll the networks above follow the basic architecture described earlier, and the only difference is the depth of the network (also shown in the table). As for the width of the network layers, it starts at $64 \\times 64$ from the first layer, then doubles after each max pooling layer until reaching $512 \\times 512$. The table below shows the number of parameters of each network model.\nSome tricks When building the network, the authors used only small $3 \\times 3$ convolution kernels with stride 1 throughout: two stacked $3 \\times 3$ convolution layers (with no pooling in between) have the same receptive field as a $5 \\times 5$ convolution layer, and likewise three stacked $3 \\times 3$ convolution layers are equivalent to a $7 \\times 7$ convolution layer.\nReplacing large-kernel convolution layers with stacks of small-kernel convolution layers has the following main advantages:\nIt uses several nonlinear rectification layers instead of a single one, which strengthens the expressive power of the model; It reduces the parameter size of the model This operation can also be seen as regularizing the original convolution layers, forcing them to be decomposed through small kernels while introducing nonlinearity; the rectification function also brings in extra nonlinearity.\nTraining uses mini-batch gradient descent to optimize the multinomial logistic regression objective (momentum 0.9, batch size 256); weight decay regularization is used with an $L_2$ regularization coefficient of $5 \\times 10^{-4}$; Dropout regularization is used in the first two fully connected layers (with the Dropout parameter set to 0.5); the learning rate is initialized to $10^{-2}$, and similar to the AlexNet model, the learning rate is decreased once whenever the validation set score stops improving, and training stops after three decreases or when the number of iterations reaches 370K (that is, 74 epochs).\nAfter some experimentation, the authors initialize the weights with a normal distribution of mean 0 and variance 0.01, and initialize the biases to 0. To accommodate $224 \\times 224$ input images, the authors randomly crop from the rescaled image in each epoch, and also apply random horizontal flipping and random RGB color shifting.\nHandling of training image size Let $S$ be the smallest side of the isotropically rescaled training image (this parameter is also referred to as the scale of the training image). In principle $S$ should be no smaller than 224 (since the input size is $224 \\times 224$); when $S = 224$ the crop covers the entire image, and when $S \u003e\u003e 224$ the crop corresponds to a small region of the original image, which may contain small objects or parts of objects. The authors propose two ways to set the training scale $S$. The first is fixed-scale training, which corresponds to training at a single scale; the second is multi-scale training, where each training image is rescaled individually by randomly sampling $S$ from a range $[S_{min}, S_{max}]$ (the authors use $[256, 512]$). Since different objects have different sizes, this approach can capture objects better, and it can also be viewed as augmenting the training set through scale jittering.\nHandling of the test set Let $Q$ be the test scale (the authors found experimentally that the test scale need not be the same as the training scale). The fully connected layers of the model are converted into convolution layers (the first FC layer becomes a $7 \\times 7$ convolution layer and the last two FC layers become $1 \\times 1$ convolution layers), and the modified model is applied directly to the whole uncropped image (including the augmented versions of the image). The result is a class score map (with as many channels as classes), which is then spatially averaged (summed) to obtain a fixed-size class score vector for the image; finally the $softmax$ class posteriors of the original and flipped images are averaged to obtain the final score for the image.\nSince the fully convolutional network is applied to the whole image, multi-crop evaluation is not needed at test time. However, multi-crop evaluation can complement the dense evaluation of the test set, because when the convolutional network is applied to crops the convolution feature maps are zero-padded, whereas dense evaluation over the test set pads the feature maps with neighboring pixel values (this comes from the convolution and pooling operations), which provides more contextual information.\nSee the original paper for the concrete models and experimental results\nReference This post refers to the paper published by Karen Simonyan et al. in 2015:\nVERY DEEP CONVOLUTIONAL NETWORKS FOR LARGE-SCALE IMAGE RECOGNITION\n","permalink":"https://simplestory.github.io/en/posts/20190730_1036fd1af9bc59d1a811b4e2ab23d60c/","summary":"\u003cblockquote\u003e\n\u003cp\u003eVGG (Visual Geometry Group) is another classic convolutional neural network model after AlexNet. It won second place in the 2014 ILSVRC competition (first place went to GoogleLeNet), and the variants used most often in work and personal projects are VGG16 and VGG19, where 16 and 19 denote the number of layers in the network. Following on from the AlexNet model discussed previously, this post analyzes the VGG model.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"basic-architecture\"\u003eBasic architecture\u003c/h2\u003e\n\u003cp\u003eThe training input images are $224 \\times 224$ RGB images, and the only preprocessing is subtracting the mean RGB value of the training set from every pixel. The convolution layers use a kernel size of $3 \\times 3$ with stride 1, and some convolution kernels use a $1 \\times 1$ size; the max pooling layers use a kernel size of $2 \\times 2$ with stride 2. Of the three fully connected layers at the end, the first two have 4096 channels, followed by a $softmax$ layer; the hidden layer activation function is $ReLU$, and the LRN local response normalization used in AlexNet is not used. The detailed structure is shown in the table below:\u003c/p\u003e","title":"VGG"},{"content":" Today\u0026rsquo;s deep neural networks have already become fully capable, especially in the vision field where they have approached or even surpassed human visual ability, and behind these deep learning networks there is always a basic framework, namely the convolutional neural network (CNN). CNN is mainly applied in the field of computer vision, so here I will briefly introduce the classic CNN model AlexNet. AlexNet was proposed by Alex Krizhevsky et al. in 2012. Although the model is rather rare nowadays (most people use resnet or something similar), it performed very well in the object detection field at the time and laid the foundation for other later CNN frameworks.\nIntroduction In order to learn to recognize several thousand object categories from millions of images, Krizhevsky et al. built a model with strong learning ability, while taking into account that the enormous complexity of some object recognition tasks cannot be solved by enlarging the dataset, but requires a certain amount of prior knowledge to supplement the model. The capacity of a convolutional neural network model can be controlled through depth and width, and it can also accurately capture the two properties of natural images, namely statistical stationarity and locality of pixel regions, so using a CNN for image recognition is a good choice.\nBasic framework No more rambling, here is the screenshot:\nThis framework contains 8 learning layers (5 convolution layers and 3 fully connected layers). Due to the limited GPU memory, the authors ran it on two GPUs and finally merged the computed data together, which is in fact a grouped convolution operation. In the model, the kernels of the second, fourth and fifth convolution layers are only connected to the previous convolution layer on the same GPU, while the third convolution layer is connected to all kernels of the second convolution layer. The response normalization layers come after the first and second convolution layers. Max pooling layers come after the response normalization layers and after the fifth layer, and the ReLU activation function is applied in every convolution layer and fully connected layer. The objective of the model is to maximize the multinomial logistic regression objective function. The following explains some special tricks used by the model.\nActivation function In 2012, the activation functions most used in neural networks were $f(x) = tanh(x)$ or $f(x) = (1+e^{-x})^{-1}$, but the authors adopt the now widely used ReLU activation function ($f(x) = max(0,x)$), which is a non-saturating nonlinearity; its descent is faster than that of the previous two saturating nonlinear functions, which made deep convolutional neural networks several times more efficient in training with the ReLU activation function.\nNon-saturating nonlinear function: \u0026ldquo;nonlinear\u0026rdquo; simply means nonlinear as usually understood, while \u0026ldquo;non-saturating\u0026rdquo; means that as the variable x tends to infinity, the value of the function also tends to infinity.\nLocal response normalization ReLUs have a very good property, namely that they do not require input normalization to prevent them from saturating. But the authors still use the following normalization:\n$$ b^i_{x,y} = \\frac{a^i_{x,y}}{\\left(k+\\alpha\\sum^{min(N-1,i+n/2)}_{j=max(0,i-n/2)}{(\\alpha^j_{x,y})^2}\\right)^{\\beta}} $$where $b^i_{x,y}$ denotes the normalized activation of the responding neuron, $\\alpha^i_{x,y}$ is the activation of the neuron at position $(x,y)$ produced by applying kernel $i$ and the ReLU nonlinearity, $N$ is the total number of kernels in the layer, and the summation runs over the kernel maps at the same spatial position. The order of the kernel maps is arbitrary and is decided only at the start of training; this scheme implements a form of lateral inhibition, creating competition for large activations among the neuron outputs computed with different kernels.\nAs for the choice of hyperparameters, the authors chose: $k=2$, $n=5$, $\\alpha=10^{-4}$ and $\\beta=0.75$.\nOverlapping pooling The pooling layers in a CNN summarize the outputs of neighboring groups of neurons in the same kernel map. The usual pooling operation is done through adjacent pooling units; specifically, a pooling layer can be seen as consisting of merging cells spaced $S$ pixels apart, each summarizing a $Z \\times Z$ neighborhood centered on that cell. If we let $S = Z$, we get the traditional convolutional neural network pooling operation; for $S \u003c Z$ we get overlapping pooling. The authors chose the parameters $S = 2$ and $Z = 3$. Models with overlapping pooling are slightly harder to overfit during training.\nReducing overfitting Data augmentation A simple and common way to reduce overfitting is data augmentation to enlarge the dataset. In the paper, the augmentation methods used by the authors are mainly image translation and horizontal reflection, and altering the intensities of the RGB channels of the training images.\nFirst, image translation and horizontal reflection: the authors randomly extract $224 \\times 224$ patches from $256 \\times 256$ images together with the corresponding horizontally reflected patches, and then train the model on this data. At prediction time, the model uses five $224 \\times 224$ patches (the four corners and the center) together with their horizontally reflected patches, that is, 10 patches in total, and finally the predictions are combined through a $softmax$ layer to obtain the final prediction.\nAs for altering the intensities of the RGB channels of the training images, the authors apply $PCA$ to the set of RGB pixel values across the training set. For each training image, multiple found principal components are added, with magnitudes proportional to the corresponding eigenvalues multiplied by a random variable drawn from a Gaussian with mean 0 and standard deviation 0.1. So for each RGB pixel value $I_{xy} = [I^R_{xy},I^G_{xy},I^B_{xy}]$, the following values are added:\n$$ [\\mathbf{p}_1,\\mathbf{p}_2,\\mathbf{p}_3][\\alpha_1\\lambda_1,\\alpha_2\\lambda_2,\\alpha_3\\lambda_3]^T $$where $\\mathbf{p}_i$ and $\\lambda_i$ are the $i$th eigenvector and eigenvalue of the $3 \\times 3$ covariance matrix of the RGB pixel values, and $\\alpha_i$ is the random variable mentioned above; for all pixels of a particular training image each $\\alpha_i$ is drawn only once, until that image is used again for training, at which point the $\\alpha_i$ are drawn again. This scheme roughly captures the property of natural images that object identity is invariant to changes in the intensity and color of illumination.\nDropout With a certain probability (the authors chose 0.5), the output of a hidden layer neuron is set to 0. Such a neuron then contributes nothing to the forward propagation and is also unaffected by error backpropagation, that is, the neural structure changes. This technique reduces the complex co-adaptation of neurons, because a neuron cannot rely on particular other neurons. The dropout probability of 0.5 chosen by the authors is the geometric mean of the predictive distributions produced by an exponential number of dropout networks. The dropout operation roughly doubles the number of iterations needed for the model to converge.\nSome training details The optimization method is SGD, with a batch size of 128, a momentum parameter of 0.9 and a weight decay parameter of 0.0005. The authors found that a small amount of weight decay is important for the model\u0026rsquo;s learning. The weight update is as follows:\n$$ \\begin{aligned} \\mathbb{v}_{i+1} \u0026 := 0.9\\cdot\\mathbb{v}_i-0.0005\\cdot\\epsilon\\cdot\\mathbb{\\omega}_i-\\epsilon\\cdot\\left\\langle\\frac{\\partial{L}}{\\partial{\\mathbb{\\omega}}}\\vert_{\\mathbb{\\omega}_i}\\right\\rangle_{D_i} \\\\ \\mathbb{\\omega}_{i+1} \u0026 := \\mathbb{\\omega}_i+\\mathbb{v}_{i+1} \\end{aligned} $$where the partial derivative term of the first equation is the average over batch $i$ of the value at $w_i$ of the derivative of the loss on $D_i$ with respect to $w$, $i$ is the iteration index, $v$ is the momentum variable and $\\epsilon$ is the learning rate.\nThe model weights are initialized with a Gaussian distribution of mean 0 and standard deviation 0.01, and the biases of the second, fourth and fifth convolution layers and of the fully connected hidden layers are initialized to the constant 1, which accelerates the early stage of learning by providing positive input to the ReLUs; the remaining neuron biases are set to 0. For all layers the authors use the same learning rate (0.01) and adjust it during training: when the validation error rate stops decreasing under the current learning rate, the learning rate is divided by 10, and training is stopped after the learning rate has been reduced three times.\nIn short, the AlexNet model achieved quite good results in the 2012 ImageNet competition, where the Top-5 error rate was reduced to $15.3\\%$. This model can be said to be a very classic model for getting started in the deep learning vision field, and many of the tricks used in it have also found good use in the subsequent development of models.\nReference This post refers to the paper published by Alex Krizhevsky et al. in 2012:\nImageNet Classification with Deep Convolutional Neural Networks\nAlexNet Stanford Vision Lab\n","permalink":"https://simplestory.github.io/en/posts/20190722_cdebdc3cc4ecd47a04cff48d03a688d3/","summary":"\u003cblockquote\u003e\n\u003cp\u003eToday\u0026rsquo;s deep neural networks have already become fully capable, especially in the vision field where they have approached or even surpassed human visual ability, and behind these deep learning networks there is always a basic framework, namely the convolutional neural network (CNN). CNN is mainly applied in the field of computer vision, so here I will briefly introduce the classic CNN model AlexNet. AlexNet was proposed by Alex Krizhevsky et al. in 2012. Although the model is rather rare nowadays (most people use resnet or something similar), it performed very well in the object detection field at the time and laid the foundation for other later CNN frameworks.\u003c/p\u003e\n\u003c/blockquote\u003e","title":"AlexNet"},{"content":" The previous post mentioned an efficient tree ensemble algorithm, the Xgboost model, but its efficiency is still low when dealing with data that has a large feature dimension and a large number of samples. Compared with models such as GBDT and Xgboost, the Lightgbm model open-sourced by Microsoft uses a variety of techniques to reduce the amount of data and the number of features, which greatly speeds up training while keeping accuracy.\nHow to do To reduce the impact of a large number of instances and a high feature dimension on model training, a natural idea is to reduce the number of data instances and the feature dimension. The schemes adopted by the Lightgbm model are Gradient-based One-Side Sampling (GOSS) and Exclusive Feature Bundling (EFB), respectively.\nGOSS This is essentially a sampling method. Microsoft found in experiments that if an instance has a small gradient, its contribution to model training is also small, because a small gradient means that the sample has already been trained well. GOSS keeps the samples with large gradients, randomly selects samples with small gradients, and compensates them with a constant weight. In this way, GOSS pays more attention to under-trained samples while not changing the original distribution too much. The related algorithm is shown below:\nSpecifically, GOSS first sorts samples by the absolute value of their gradients (from largest to smallest) and takes the first $a \\times 100\\%$ of the samples. Then it randomly selects $b \\times 100\\%$ of the remaining samples. After that, GOSS amplifies each sample with a small gradient by a constant weight (usually $\\frac{1-a}{b}$).\nEFB Next comes the problem of reducing the feature dimension. In fact, we can notice that data with high-dimensional features is usually sparse, and in a sparse feature space many features are mutually exclusive, so we can bundle these exclusive features into a single feature. The specific algorithm is as follows:\nFirst we build a weighted graph, where the weight of an edge represents the amount of conflict between the two connected features (that is, the degree to which they are not mutually exclusive). Then the features are sorted in descending order by degree. Finally we traverse the sorted feature sequence and check whether a feature can be added to an existing bundle (determined by the amount of conflict); if not, a new bundle is created.\nThe above approach is indeed effective, but it is still unsatisfactory when the number of features is large. We can sort the features by their non-zero values to obtain the feature sequence, which avoids building the feature graph.\nThe features have now been placed into bundles, and the next step is to merge the features within the same bundle. The key here is that the original feature values can be distinguished from the bundle, because histogram-based algorithms store discrete bins rather than continuous feature values. This can be achieved by using offsets. Details are as follows:\nSee the paper below for the detailed mathematical derivation\nReference LightGBM: A Highly Efficient Gradient Boosting Decision Tree\n","permalink":"https://simplestory.github.io/en/posts/20190324_161af8745042398050ad585874c3eec3/","summary":"\u003cblockquote\u003e\n\u003cp\u003eThe previous post mentioned an efficient tree ensemble algorithm, the \u003ca href=\"https://simplestory.github.io/2019/03/17/Xgboost/\"\u003eXgboost model\u003c/a\u003e, but its efficiency is still low when dealing with data that has a large feature dimension and a large number of samples. Compared with models such as GBDT and Xgboost, the Lightgbm model open-sourced by Microsoft uses a variety of techniques to reduce the amount of data and the number of features, which greatly speeds up training while keeping accuracy.\u003c/p\u003e\n\u003c/blockquote\u003e","title":"LightGBM Model"},{"content":" Today it occurred to me that I have not updated my blog for a long time; lately I have been busy with written tests and interviews, and with my graduation project. Now that things have come to a pause today, I might as well organize the Xgboost model I reviewed a few days ago and post an article.\nTree ensemble Ensemble algorithms are a broad and very important class of algorithms in machine learning, generally divided into Bagging and Boosting. Ensembling decision trees is of course a customary technique. For tree ensembles, the representative of the Bagging method is Random Forest, which randomly groups attributes and data for decision tree learning and finally aggregates the results; the representative of the Boosting method is AdaBoost and GBDT (gradient boosted decision trees). These algorithms are all quite important and cannot be summed up in a sentence or two, so I will not expand on them. Within tree Boosting, Tianqi Chen proposed a new model, Xgboost. This model makes use of the second derivative of the objective function and also adds a regularization term to prevent the model from overfitting. Below is an interpretation of the Xgboost model.\nObjective function The result of a tree ensemble model is generated from the results of many individual tree models. The individual tree model chosen by Xgboost is CART. The CART model differs somewhat from other decision tree models: the value of its leaves is a definite score rather than an actual class, which helps to implement highly efficient optimization algorithms. Using a bunch of CART trees for prediction, we can simply add up the results of the individual CART models. This gives the following function:\n$$ \\hat{y}_i = \\Phi(\\mathbf{x}_i) = \\sum_{k=1}^K f_k(\\mathbf{x}_i), \\ f_k \\in F \\tag{1} $$where the given dataset has n rows of data and m features. $D=\\{(\\mathbf{x}_i,y_i)\\}(\\vert{D}\\vert=n,\\mathbf{x}_i\\in R^m,y_i\\in R)$ and an additive algorithm is used.\n$F=\\{f(\\mathbf{x})=w_{q(\\mathbf{x})}\\}(q:R^m\\rightarrow T.\\ w\\in R^T)$ is the space of all possible CART trees, and $f(x)$ is a single CART tree model. $T$ is the number of leaf nodes of the tree, $q(x)$ maps a sample to some value from 1 to T, that is, to some leaf node; in fact it represents the structure of the tree, and $w_{q(x)}$ is the prediction of this tree for the sample.\nThe objective function to be optimized can then be defined as:\n$$ L(\\Phi) = \\sum_i l(\\hat{y}_i, y_i)+\\sum_k \\Omega(f_k) \\tag{2} $$where $L(\\hat{y}_i,y_i)$ is the loss function and the latter term is the regularization term representing the complexity of the tree:\n$$ \\Omega(f) = \\gamma T+\\frac{1}{2}\\lambda{\\Vert{w}\\Vert}^2 \\tag{3} $$Usually the mean squared error (MSE) is chosen as the loss function, which gives a very pretty form:\n$$ L^{(t)} = \\sum_{i=1}^n [2(\\hat{y}^{(t-1)}_i-y_i)f_t(x_i)+{f_t(x_i)}^2]+\\Omega(f_t)+const $$For other loss functions (as long as their second derivative exists), Taylor\u0026rsquo;s formula is used to expand $L^{(t)}(y_i,\\hat{y}^{(t-1)}_i+f_t(x_i))$ to the second order, where we let $x=\\hat{y}_i^{(t-1)}+f_t(x_i), \\ x_0=\\hat{y}_i^{(t-1)}$:\n$$ \\begin{aligned} L^{(t)}(y_i,x) \u0026 \\simeq l(y_i,x_0)+l^\\prime(y_i,x_0)\\cdot(x-x_0)+\\frac{1}{2!}l^{\\prime\\prime}(y_i,x_0)\\cdot{(x-x_0)}^2+\\Omega(f)+const \\\\ \u0026 = l(y_i,x_0)+\\frac{\\partial l(y_i,x_0)}{\\partial x_0}f_t(x_i)+\\frac{1}{2}\\frac{\\partial^2 l(y_i,x_0)}{\\partial^2 x_0}f_t^2(x_i)+\\Omega(f)+const \\end{aligned} \\tag{4} $$Let $g_i = \\frac{\\partial l(y_i,x_0)}{\\partial x_0}$, $h_i = \\frac{\\partial^2 l(h_i, x_0)}{\\partial^2 x_0}$, and ignore the constant term, giving:\n$$ \\tilde{L}^{(t)} = \\sum_{i=1}^n[g_if_t(x_i)+\\frac{1}{2}h_if_i^2(x_i)]+\\Omega(f_t) \\tag{5} $$Define $I_j=\\\\{i\\vert q(\\mathbf{x}_i)=j\\\\}$ as the set of indices of the training samples assigned to leaf j. Combining equations (3) and (5) gives:\n$$ \\begin{aligned} \\tilde{L}^{(t)} \u0026 = \\sum_{i=1}^n[g_if_t(x_i)+\\frac{1}{2}h_if_i^2(x_i)]+\\gamma T+\\frac{1}{2}\\lambda \\sum_{j=1}^T{w_j}^2 \\\\ \u0026 = \\sum_{j=1}^T[(\\sum_{i \\in I_j}g_i)w_i+\\frac{1}{2}(\\sum_{i \\in I_j}h_i+\\lambda){w_j}^2]+\\gamma T \\end{aligned} \\tag{6} $$For a determined tree structure $q(x)$, the optimal weight $w$ of leaf node j can be computed as follows:\n$$ w_j^* = -\\frac{\\sum_{i \\in I_j}g_i}{\\sum_{i \\in I_j}h_i+\\lambda} \\tag{7} $$and the corresponding loss is:\n$$ \\tilde{L}^{(t)}(q) = -\\frac{1}{2}\\sum_{j=1}^{T}\\frac{(\\sum_{i \\in I_j}g_i)^2}{\\sum_{i \\in I_j}h_i+\\lambda}+\\gamma T \\tag{8} $$Equation (8) can be used to evaluate how good a tree model is; the smaller the value the better. This value is only used to measure the quality of the tree structure and has nothing to do with the values of the leaf nodes. This can be seen from the derivation: $\\tilde{L}^{(t)}(q)$ depends only on $g_i$ and $h_i$, and these two values depend only on the structure of the tree, having nothing to do with the leaf node values.\nGrowing the tree Obviously a CART tree has infinitely many structures, so growing a tree and then evaluating the structure is not realistic. There is a greedy algorithm that grows the tree layer by layer. That is, starting from the root node, branches are created layer by layer. Suppose $I_L$ and $I_R$ are the instances contained in the left and right nodes after branching, with $I = I_L \\bigcup I_R$; then the criterion for whether to branch is:\n$$ L_{split} = \\frac{1}{2}\\left[\\frac{(\\sum_{i \\in I_L}g_i)^2}{\\sum_{i \\in I_L}h_i+\\lambda}+\\frac{(\\sum_{i \\in I_R}g_i)^2}{\\sum_{i \\in I_R}h_i+\\lambda}-\\frac{(\\sum_{i \\in I}g_i)^2}{\\sum_{i \\in I}h_i+\\lambda}\\right]-\\gamma \\tag{9} $$The positive and larger the value of $L_{split}$, the more worthwhile the branch is, because after branching $\\tilde{L}^{(t)}(q)$ becomes smaller. In fact, $\\gamma$ acts as a threshold; the larger this value, the stricter the branching criterion.\nNote: Xgboost does not need a separate pruning operation, because the complexity of the tree model is already taken into account when growing the tree, which corresponds to the parameter $\\gamma$.\nThe algorithm model for finding the best split point based on the greedy algorithm is as follows:\nFinding split points with the greedy algorithm is a very effective algorithm, but for large datasets that cannot be loaded into memory at once, this method fails. In that case an approximate method is needed to find the split points.\nThe approximate method in fact proposes candidate split points according to the percentiles of the feature distribution. The continuous features are then mapped into the buckets divided by the candidate points, specifically as follows:\nHere Global means the candidate split points are given before learning each tree, while Local means the candidate split points are given before each split\nWeighted quantile sketch In the approximate branching above, one important step is the selection of candidate points. Suppose the set $D_k=\\\\{(x_{1k},h_1),(x_{2k},h_2)\\cdots(x_{nk},h_n)\\\\}$ represents the values of the k-th feature together with the statistics of their second derivatives. Define a rank function $r_k: \\ R \\rightarrow [0,+\\infty)$:\n$$ r_k(z) = \\frac{1}{\\sum_{(x,h) \\in D_k}h}\\sum_{(x,h) \\in D_k, x \\lt z}h \\tag{10} $$This represents the proportion of instances in the sample whose corresponding feature value is less than z. Our goal is to find the optimal split points $\\\\{s_{k1},s_{k2},\\cdots,s_{kl}\\\\}$, which can be done with:\n$$ \\vert r_k(s_{k,j})-r_k(s_{k,j+1})\\vert \\lt \\epsilon, \\ s_{k1}=min_i x_{ik}, s_{kl}=max_i x_{ik} \\tag{11} $$Here $\\epsilon$ is an approximation factor; roughly $\\frac{1}{\\epsilon}$ candidate split points can be assumed. Each data point here has a weight $h_i$.\nSparsity-aware split finding This part mainly deals with missing values in the samples. The method is to consider assigning the missing values to the left and right branches separately, and to pick the side with the larger gain, specifically as follows:\nCache access patterns To speed up model computation, cache access patterns can be used, specifically as follows:\nFor the greedy split-search algorithm: a cache-aware prefetching algorithm is used to alleviate memory access problems. That is, an internal buffer is requested for each thread, storing gradient statistics, and the accumulation is performed in small batches. For the approximate split-search algorithm: an appropriate block size is used to alleviate memory access problems. A block that is too small gives each thread little work and low efficiency; a block that is too large causes cache misses. It is usually set to $2^{16} \\ examples \\ per \\ block$. Note\nThe means by which Xgboost prevents overfitting:\nAdding a regularization term, that is, $\\Omega(f)$ in the objective function Shinkage: after each iteration, the weights on the leaf nodes are multiplied by this coefficient, mainly to dampen the influence of each tree and leave more room for learning later on Column subsampling: similar to random forest, it supports feature subsampling, which prevents overfitting while reducing computation Reference Introduction to Boosted Trees\nXGBoost: A Scalable Tree Boosting System\n","permalink":"https://simplestory.github.io/en/posts/20190317_8fc39e31f30337e40756914df1f8b607/","summary":"\u003cblockquote\u003e\n\u003cp\u003eToday it occurred to me that I have not updated my blog for a long time; lately I have been busy with written tests and interviews, and with my graduation project. Now that things have come to a pause today, I might as well organize the Xgboost model I reviewed a few days ago and post an article.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"tree-ensemble\"\u003eTree ensemble\u003c/h2\u003e\n\u003cp\u003eEnsemble algorithms are a broad and very important class of algorithms in machine learning, generally divided into Bagging and Boosting. Ensembling decision trees is of course a customary technique. For tree ensembles, the representative of the Bagging method is Random Forest, which randomly groups attributes and data for decision tree learning and finally aggregates the results; the representative of the Boosting method is AdaBoost and GBDT (gradient boosted decision trees). These algorithms are all quite important and cannot be summed up in a sentence or two, so I will not expand on them. Within tree Boosting, Tianqi Chen proposed a new model, Xgboost. This model makes use of the second derivative of the objective function and also adds a regularization term to prevent the model from overfitting. Below is an interpretation of the Xgboost model.\u003c/p\u003e","title":"Xgboost Model"},{"content":" Optimization is a very difficult problem; usually one has to carefully design an objective function and related constraints, and also make sure the problem to be optimized is a convex function, but most real-world problems are non-convex. For non-convex optimization problems there are some basic optimization methods: SGD, Momentum, AdaGrad, RMSProp, Adam and so on\nStochastic Gradient Descent (SGD) // stochastic gradient descent\nGiven the dataset $X=\\\\{x^{(1)},x^{(2)},...,x^{(n)}\\\\}$ and the dataset labels $Y=\\\\{y^{(1)}, y^{(2)},...,y^{(n)}\\\\}$\nLearner $f(x;w)$, learning rate (step size) $\\alpha$\nFor enough iterations\n{\nRandomly select data: $\\\\{x^{(j)}, y^{(j)}\\\\}$\nCompute the loss gradient: $\\nabla\\mathbf{w}=\\frac{\\partial{L(y^{(j)},f(x^{(j)};w))}}{\\partial{w}}$\nModify the weights: $\\mathbf{w_i}=\\mathbf{w_i}-\\alpha\\nabla\\mathbf{w}$\n}\nThe above randomly selects one data point from the dataset for gradient descent, so a strong random effect appears during training. To avoid this randomness, multiple data points can be selected for the gradient computation and then averaged.\nAs for the learning rate, in the algorithm above the learning rate stays constant, which in the later stage of training causes the algorithm to oscillate around the optimal solution. To eliminate or alleviate this, we can decrease the learning rate as much as possible when approaching the optimal solution\nCombining the two gives the following gradient descent algorithm\n// learning-rate-decayed mini-batch gradient descent\nGiven the dataset $X=\\\\{x^{(1)},x^{(2)},...,x^{(n)}\\\\}$ and the dataset labels $Y=\\\\{y^{(1)}, y^{(2)},...,y^{(n)}\\\\}$\nRandomly sample $m$ data points, number of training epochs $k$, minimum learning rate decay value $b$, learner $f(x;w)$\nInitial learning rate (step size) $\\alpha_0$\nFor $i \u003c= k$\n{\nRandomly sample $m$ data points: $\\\\{(x^{(i)},y^{(i)}),...,(x^{(m)},y^{(m)}\\\\}$\nCompute the average loss gradient of the sampled data: $\\nabla \\mathbf{w} = \\frac{1}{m}\\sum^m_{j=1}\\frac{\\partial{L(y^{(j)},f(x^{(j)};w)})}{\\partial{\\mathbf{w}}}$\nCompute the decayed learning rate: $a_i=(i-\\frac{i}{k})a_0+\\frac{i}{k}b$\nModify the network weights: $\\mathbf{w}_i=\\mathbf{w}_i-\\alpha\\nabla\\mathbf{w}$\n}\nMomentum Although the algorithm above solves the problems of oscillation around the optimal solution and of randomness, it cannot guarantee that the solution found by the algorithm is the global optimum; the algorithm may afterwards find a local optimum, that is, a saddle point.\nTo escape the saddle point, the physical concept of momentum can be used to break out of it, giving the following momentum stochastic gradient descent method\n// momentum method\nGiven the dataset $X=\\\\{x^{(1)},x^{(2)},...,x^{(n)}\\\\}$ and the dataset labels $Y=\\\\{y^{(1)}, y^{(2)},...,y^{(n)}\\\\}$\nInitial velocity $v$, random sample size $m$, number of training epochs $k$\nLearner $f(x;w)$, initial learning rate $\\alpha$, initial momentum parameter $\\beta$\nFor $i \u003c= k$\n{\nRandomly sample $m$ data points: $\\\\{(x^{(i)},y^{(i)}),...,(x^{(m)},y^{(m)}\\\\}$\nCompute the average loss gradient of the sampled data: $\\nabla \\mathbf{w} = \\frac{1}{m}\\sum^m_{j=1}\\frac{\\partial{L(y^{(j)},f(x^{(j)};w)})}{\\partial{\\mathbf{w}}}$\nUpdate the velocity: $v=\\beta v-\\alpha\\nabla\\mathbf{w}$\nUpdate the parameters: $\\mathbf{w}=\\mathbf{w}+v$\n}\nIn practice, commonly used values of $\\beta$ are 0.5, 0.9 or 0.99, tuned according to the specific problem, but adjusting $\\beta$ is not as important as adjusting $\\alpha$, so it does not really need to be chosen as a hyperparameter — an appropriate value is generally enough. The momentum method exhibits slight oscillation in the later stage of training.\nNAG (Nesterov accelerated gradient) To solve the problem of slight oscillation in the momentum method, NAG uses $\\mathbf{w}-\\beta v_{t-1}$ as an approximation of the value the parameters will take at the next step, so the gradient is computed at the future position; it performs very well on sparse data. The velocity update is written as follows:\n$$ v_t = \\beta v_{t-1}+\\alpha\\nabla_\\mathbf{w} J(\\mathbf{w}-\\beta v_{t-1}) $$where $J(\\cdot)$ is the loss function. When the gradient direction is about to change, NAG can know in advance, thereby weakening this process and reducing useless iterations.\nAdaGrad, Adelta and RMSProp The algorithms above all use a single global learning rate, so all parameters advance with a uniform stride, but the parameters in a network may not all have consistent descent gradients and directions at the same time. The following algorithms configure the learning rate individually for each parameter\nAdaGrad This algorithm simply accumulates the square of the historical gradient of each dimension, and then divides by that historical gradient when updating (the square is used to remove the sign of the gradient, accumulating only the magnitude of the gradient)\n$$cache_i = cache_i+(\\nabla\\mathbf{w}_i)^2$$Update the parameters:\n$$\\mathbf{w}_i=\\mathbf{w}_i-\\frac{\\alpha}{\\sqrt{cache_i}+\\delta}\\nabla\\mathbf{w}_i$$where $\\delta=10^{-7}$ prevents numerical overflow caused by division by zero\nAdaGrad makes the learning rate larger for parameters whose accumulated gradient magnitude is small (\u0026lt;1), making network training faster, and smaller when the accumulated gradient magnitude is large (\u0026gt;1), slowing down network training\nBut with AdaGrad the learning rate decreases very quickly as the number of epochs grows, that is, the algorithm easily reduces the learning rate too much\nAdelta This method improves on the situation in Adgard where the learning rate becomes very small as the number of iterations increases, mainly by replacing $cache$ with the decayed average of the squared gradients of the past. The parameter update formulas are as follows:\n$$ \\begin{aligned} \\mathbf{w}_t \u0026= \\mathbf{w}_{t-1} - \\frac{\\alpha}{\\sqrt{E[g^2]_t}+\\delta}g_t \\\\ E[g^2]_t \u0026= \\frac{1}{t}\\sum_{i=1}^t g_i^2 \\\\ g_t \u0026= \\nabla\\mathbf{w}_t \\end{aligned} $$This denominator is equivalent to the root mean square of the gradient (in mathematical statistics, the root mean square is the square root of the average of the squared data). So it can be abbreviated as:\n$$ \\mathbf{w}_t = \\mathbf{w}_{t-1}-\\frac{\\alpha}{RMS[g]_t}g_t $$The averaging part can be replaced with an incremental form, introducing the hyperparameter $\\gamma$ (usually set to 0.9) to weigh the importance of past versus current gradients:\n$$ E[g^2]_t = \\gamma E[g^2]_{t-1}+(1-\\gamma)g_t^2 $$The learning rate can also be replaced with $RMS[\\Delta\\mathbf{w}]$, so that no learning rate needs to be preset.\n$$ \\begin{aligned} \\Delta\\mathbf{w}_t \u0026= -\\frac{RMS[\\Delta\\mathbf{w}]_{t-1}}{RMS[g]_t}g_t \\\\ \\mathbf{w}_t \u0026= \\mathbf{w}_{t-1}+\\Delta\\mathbf{w}_t \\\\ \\end{aligned} $$RMSProp To solve the problem that AdaGrad\u0026rsquo;s learning rate decays too fast, the RMSProp algorithm introduces a decay factor, striking a balance between \u0026ldquo;past\u0026rdquo; and \u0026ldquo;now\u0026rdquo; when accumulating gradients; $\\beta$ adjusts the amount of decay, with commonly used values of 0.9 or 0.5\n$$cache_i=\\beta cache_i+(1-\\beta)(\\nabla \\mathbf{w}_i)^2$$In the parameter update stage it is the same as AdaGrad\n$$\\mathbf{w}_i=\\mathbf{w}_i-\\frac{\\alpha}{\\sqrt{cache_i}+\\delta}\\nabla\\mathbf{w}_i$$Adam This algorithm is a fine-tuned version of Momentum + RMSProp; by default this optimization method is recommended\nAt the beginning the gradients are very small, so $r$ and $v$ are often close to 0, and therefore we also need to adjust $r$ and $v$\n$$ \\begin{aligned} vb=\\frac{v}{1-\\beta^t_1} \\\\ rb=\\frac{r}{1-\\beta^t_2} \\end{aligned} $$where $t$ denotes the number of training steps, so this algorithm only scales up the respective values according to the decay factors during the first few rounds of training, and very soon $vb$ and $rb$ decay to $v$ and $r$\n// Adam\nGiven the dataset $X=\\\\{x^{(1)},x^{(2)},...,x^{(n)}\\\\}$ and the dataset labels $Y=\\\\{y^{(1)}, y^{(2)},...,y^{(n)}\\\\}$\nInitial velocity $v$, random sample size $m$, number of training epochs $k$\nLearner $f(x;w)$, initial learning rate $\\alpha$, initial momentum parameter $\\beta_1$\nLearning rate decay parameter $\\beta_2$, $\\delta=10^{-7}$\nFor $i \u003c= k$\nRandomly sample $m$ data points: $\\\\{(x^{(i)},y^{(i)}),...,(x^{(m)},y^{(m)}\\\\}$\nCompute the current sampled data gradient: $g = \\frac{1}{m}\\sum^m_{j=1}\\frac{\\partial{L(y^{(j)},f(x^{(j)};w)})}{\\partial{\\mathbf{w}}}$\nUpdate the current velocity: $v=\\beta_1 v+(1-\\beta_1)g$\nUpdate the current learning rate: $r=\\beta_2 r+(1-\\beta_2)g^2$\nUpdate the step count: $t=t+1$\n$$ \\begin{aligned} vb=\\frac{v}{1-\\beta^t_1} \\\\ rb=\\frac{r}{1-\\beta^t_2} \\end{aligned} $$Update the parameters:\n$$w=w-\\frac{\\alpha}{\\sqrt{rb}+\\delta}vb$$","permalink":"https://simplestory.github.io/en/posts/20181103_7b202c9a380844d1b8ae2c4f5ee3461f/","summary":"\u003cblockquote\u003e\n\u003cp\u003eOptimization is a very difficult problem; usually one has to carefully design an objective function and related constraints, and also make sure the problem to be optimized is a convex function, but most real-world problems are non-convex. For non-convex optimization problems there are some basic optimization methods: SGD, Momentum, AdaGrad, RMSProp, Adam and so on\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"stochastic-gradient-descent-sgd\"\u003eStochastic Gradient Descent (SGD)\u003c/h2\u003e\n\u003chr\u003e\n\u003cp\u003e// stochastic gradient descent\u003c/p\u003e\n\u003cp\u003eGiven the dataset $X=\\\\{x^{(1)},x^{(2)},...,x^{(n)}\\\\}$ and the dataset labels $Y=\\\\{y^{(1)}, y^{(2)},...,y^{(n)}\\\\}$\u003c/p\u003e","title":"Deep Learning Optimization"},{"content":" CTR (click-through rate) prediction plays an important role in content recommendation and delivery, and the FFM model is good at handling this kind of data. The FFM model is a variant of the FM model, or rather the FM model is a special case of the FFM model\nRough pipeline Convert the dataset format (using the LIBSVM or FFMs style) Compute the loss function (using the exponential loss function) Optimize the parameters (using the AdaGrad + SG strategy) Converting the data format LIBSVM format (does not store zero-valued features):\n$$ label \\quad feat1:val1 \\quad feat2:val2 \\quad ... \\quad $$FFMs format:\n$$label \\quad field1:feat1:val1 \\quad field2:feat2:val2 \\quad ... \\quad$$For different data types the conversion of the data format differs\nFor Categorical Features\nFor example, given the following sample:\n$$Yes \\quad P:ESPN \\quad A:Nike \\quad G:Male$$Converted to LIBSVM format:\n$$Yes \\quad P-ESPN:1 \\quad A-Nike:1 \\quad G-Male:1$$Converted to FFMs format:\n$$Yes \\quad P:P-ESPN:1 \\quad A:A-Nike:1 \\quad G:G-Male:1$$Numerical Features\nNumerical features have two conversion formats: the first treats each feature as its own field (this method just duplicates the data without adding information); the second discretizes the data and converts the format as categorical features\nFor example:\nAccepted AR Hidx Cite Yes 45.73 2 3 No 1.04 100 50,000 Format one:\n$$Yes \\quad AR:AR:45.73 \\quad Hidx:Hidx:2 \\quad Cite:Cite:3$$Format two:\n$$Yes \\quad AR:45:1 \\quad Hidx:2:1 \\quad Cite:3:1$$For format two, some information may be lost after discretization\nSingle-field Features\nFeatures of this kind reduce the FFM model to the FM model\nComputing the loss function In CTR prediction, the commonly used optimization is:\n$$ min_{\\mathbf{w}} \\quad \\frac{\\lambda}{2}||\\mathbf{w}||^2_2+\\sum_{i=1}^m log(1+exp(-y_i\\phi(\\mathbf{w},\\mathbf{x}_i))) $$where the function $\\phi(\\mathbf{w},\\mathbf{x}_i)$ is replaced by different functions depending on the model; for the FFM model we have:\n$$ \\phi_{FFM}(\\mathbf{w},\\mathbf{x})=\\sum_{j_1=1}^n\\sum_{j_2=j_1+1}^n (\\mathbf{w}_{j1,f2}\\cdot\\mathbf{w}_{j2,f1})x_{j1}x_{j2} $$where $f_1$ and $f_2$ denote the fields of $j_1$ and $j_2$ respectively\nOptimizing the parameters The parameters are optimized with the SG + AdaGrad method\nFirst the gradients are computed:\n$$ \\begin{aligned} \\mathbf{g}_{j1,f2} \\equiv \\nabla_{\\mathbf{w}_{j1,f2}}f(\\mathbf{w}) = \\lambda \\cdot \\mathbf{w}_{j1,f2}+\\kappa\\cdot\\mathbf{w}_{j2,f1}x_{j1}x_{j2} \\\\ \\mathbf{g}_{j2,f1} \\equiv \\nabla_{\\mathbf{w}_{j2,f1}}f(\\mathbf{w}) = \\lambda \\cdot \\mathbf{w}_{j2,f1}+\\kappa\\cdot\\mathbf{w}_{j1,f2}x_{j1}x_{j2} \\end{aligned} \\tag{1} $$where:\n$$ \\kappa = \\frac{\\partial{log(1+exp(-y\\phi_{FFM}(\\mathbf{w},\\mathbf{x})))}}{\\partial{\\phi_{FFM}({\\mathbf{w},\\mathbf{x}}})} =\\frac{-y}{1+exp(y\\phi_{FFM}({\\mathbf{w},\\mathbf{x}}))} $$Then, with the AdaGrad algorithm, the squares of the gradients are accumulated:\n$$ \\begin{aligned} (G_{j1,j2})_d \\leftarrow (G_{j1,f2})_d+(g_{j1,f2})_d^2 \\\\ (G_{j2,f1})_d \\leftarrow (G_{j2,f1})_d+(g_{j2,f1})_d^2 \\end{aligned} \\tag{2} $$where d runs from 1 to k (k is a hyperparameter denoting the dimension of the latent vector)\nFinally the relevant parameters are updated with:\n$$ \\begin{aligned} (w_{j1,f2})_d \\leftarrow (w_{j1,f2})_d-\\frac{\\eta}{\\sqrt{(G_{j1,f2})_d}}(g_{j1,f2})_d \\\\ (w_{j2,f1})_d \\leftarrow (w_{j2,f1})_d-\\frac{\\eta}{\\sqrt{(G_{j2,f1})_d}}(g_{j2,f1})_d \\end{aligned} \\tag{3} $$where $\\eta$ is a hyperparameter — the learning rate. The initial values of $\\mathbf{w}$ are drawn randomly from a uniform distribution over $[0,1/\\sqrt{k}]$, and the initial value of G is set to 1 to prevent $(G_{j1,f2})_d^{-\\frac{1}{2}}$ from becoming too large\nExperiments show that normalizing each sample to a consistent length improves the training accuracy of the model, and that the model is fairly sensitive to its parameters\nOverfitting strategy The FFM model uses early stopping to prevent overfitting, with the following strategy:\nSplit the dataset into a training set and a validation set After each training epoch, compute the model\u0026rsquo;s loss on the validation set If the loss increases, record the number of training epochs, stop training and go to step 4 If needed, retrain the model on the full dataset for the number of epochs obtained in step 3 Some conclusions Compared with the FM model, the FFM model introduces the concept of field, increasing the number of latent vectors. If n features belong to f fields, the second-order term of the FFM model has nf latent vectors, whereas FM can be regarded as a special case of FFM, namely the FFM model when all features belong to a single field\nThe FFM model performs well on data that contains categorical features converted for binary classification If the converted dataset is not sparse enough, the FFM model may not perform very well The FFM model struggles to handle numerical features The model has three hyperparameters:\nk, the dimension of the latent vectors; experiments show that its size has little effect on the model $\\lambda$, the regularization coefficient; too large a value leads to underfitting, while too small a value easily overfits $\\eta$, the learning rate; too large easily overfits, while too small makes the training error decrease slowly Related libraries or frameworks:\nlibffm\nxlearn\nReference paper\nField-aware Factorization Machines for CTR Prediction\n","permalink":"https://simplestory.github.io/en/posts/20181028_fa81994273e28bed9a6b443a7af7f69e/","summary":"\u003cblockquote\u003e\n\u003cp\u003eCTR (click-through rate) prediction plays an important role in content recommendation and delivery, and the FFM model is good at handling this kind of data. The FFM model is a variant of the FM model, or rather the FM model is a special case of the FFM model\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"rough-pipeline\"\u003eRough pipeline\u003c/h2\u003e\n\u003col\u003e\n\u003cli\u003eConvert the dataset format (using the LIBSVM or FFMs style)\u003c/li\u003e\n\u003cli\u003eCompute the loss function (using the exponential loss function)\u003c/li\u003e\n\u003cli\u003eOptimize the parameters (using the AdaGrad + SG strategy)\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch2 id=\"converting-the-data-format\"\u003eConverting the data format\u003c/h2\u003e\n\u003cp\u003eLIBSVM format (does not store zero-valued features):\u003c/p\u003e","title":"How the FFM Model Works"},{"content":" Decision tree, a common machine learning method, is somewhat like a flowchart. A sample is divided into a set according to the value it takes on some feature, and this is done recursively\nBasic algorithm Input: training set $D=\\\\{(\\mathbf{x_1},y_1),(\\mathbf{x_2},y_2),...,(\\mathbf{x_m},y_m)\\\\}$; attribute set $A=\\\\{a_1,a_2,...,a_d\\\\}$\nOutput: a decision tree with node as the root node\nSimple process:\nUsing some splitting criterion, the optimal feature is selected to split the samples, and this is done recursively, finally yielding the branch nodes of the decision tree\nWhat matters is finding the currently optimal splitting attribute so that the samples contained in the branch nodes of the decision tree belong to the same class as much as possible, that is, the purer the node the better\nBasic routine Information entropy and information gain Suppose the proportion of samples of the k-th class in the current sample set D is $p_k \\ (k=1,2,3,...,\\vert y\\vert)$, then the information entropy of D is defined as:\n$$Ent(D) = -\\sum_{k=1}^{\\vert y\\vert} p_k\\log_{2}p_k$$where by convention, if $p=0$ then $p\\log_{2}p = 0$; we have $0 \\le Ent(D) \\le \\log_{2}\\vert y\\vert$, and the smaller the value of $Ent(D)$, the lower the purity of D\nIf attribute a has V possible values $\\\\{a^1,a^2,...,a^V\\\\}$, splitting the sample set D by a produces V branch nodes, where the v-th branch node contains all samples in D whose value on attribute a is $a^v$, denoted $D^v$; then the information gain obtained by splitting the sample set with attribute a is:\n$$Gain(D,a) = Ent(D)-\\sum_{v=1}^V \\frac{\\vert D^v\\vert}{\\vert D\\vert} Ent(D^v)$$Generally speaking, the larger the information gain, the greater the purity improvement obtained by splitting with attribute a, so we can use information gain as the splitting criterion; this is the ID3 decision tree learning algorithm\nGain ratio Above we used information gain as our splitting criterion, but information gain prefers attributes with more categories. If we had a feature column of id, then information gain would preferentially choose the id feature for splitting, which is not our purpose. For this reason, we consider adding a regularization term proportional to the number of categories to the information gain, which gives the gain ratio (also called information gain ratio):\n$$ \\begin{aligned} Gain\\_raito(D,a) = \\frac{Gain(D,a)}{IV(a)} \\\\ IV(a) = -\\sum_{v=1}^V \\frac{\\vert D^v\\vert}{\\vert D\\vert} \\log_{2}\\frac{\\vert D^v\\vert}{\\vert D\\vert} \\end{aligned} $$The gain ratio criterion prefers attributes with fewer values, so the C4.5 decision tree algorithm adds a heuristic design on top of the gain ratio: first find the attributes among the candidate splitting attributes whose information gain is above average, and then choose the one with the highest gain ratio among them\nGini index Information entropy is a measure of the uncertainty of information, and in fact also a measure of the purity of information; the Gini index is also a measure in this respect, computed as follows:\n$$ \\begin{aligned} Gini(D) \u0026 = \\sum_{k=1}^{\\vert y\\vert} \\sum_{k‘ \\neq k} p_kp_{k’} \\\\ \u0026 = \\sum_{k=1}^{\\vert y\\vert} p_k(1-p_k) \\\\ \u0026 = 1-\\sum_{k=1}^{\\vert y\\vert} {p_k}^2 \\end{aligned} $$where $p_k$ denotes the probability that a sample belongs to the k-th class; Gini(D) reflects the probability that two samples drawn at random from dataset D have inconsistent class labels, so the smaller Gini(D) is, the higher the purity of dataset D\nFor attribute a, the Gini index is the weighted sum of the Gini indices of the values of attribute a, that is:\n$$Gini\\_index(D,a) = \\sum_{v=1}^V \\frac{\\vert D^v\\vert}{\\vert D\\vert} Gini(D^v)$$So among the candidate attribute set A, we should choose the attribute that minimizes the Gini index after splitting as the optimal splitting attribute:\n$$a_* = argmin_{a \\in A} Gini\\_index(D,a)$$Splitting based on the Gini index is the CART decision tree algorithm\nOptimization For the overfitting of a decision tree, there are mainly two methods: pre-pruning and post-pruning\nPre-pruning\nPre-pruning means that during the construction of the decision tree, each node is evaluated before and after splitting; if splitting the current node cannot improve the generalization performance of the decision tree, splitting stops and the current node becomes a leaf node\nPost-pruning\nPost-pruning means that after the decision tree has been fully constructed, the non-leaf nodes are examined bottom-up; if replacing the subtree corresponding to a node with a leaf node improves the generalization performance, the subtree is replaced by a leaf node\nComparison of pre- and post-pruning\nItem Pre-pruning Post-pruning Training time cost Small Large Testing time cost Small Small Overfitting risk Small Small Underfitting risk Large Small In terms of generalization performance: post-pruning is better than pre-pruning\nUse in Scikit-learn See the official site for the concrete API parameters:\nsklearn.tree.DecisionTreeClassifier\nReference Zhou Zhihua\u0026rsquo;s Watermelon Book and Li Hang\u0026rsquo;s Statistical Learning Methods\n","permalink":"https://simplestory.github.io/en/posts/20180829_3cb42cd7fd688b586f483bd682fc8209/","summary":"\u003cblockquote\u003e\n\u003cp\u003eDecision tree, a common machine learning method, is somewhat like a flowchart. A sample is divided into a set according to the value it takes on some feature, and this is done recursively\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch1 id=\"basic-algorithm\"\u003eBasic algorithm\u003c/h1\u003e\n\u003cp\u003eInput: training set $D=\\\\{(\\mathbf{x_1},y_1),(\\mathbf{x_2},y_2),...,(\\mathbf{x_m},y_m)\\\\}$;\nattribute set $A=\\\\{a_1,a_2,...,a_d\\\\}$\u003c/p\u003e\n\u003cp\u003eOutput: a decision tree with node as the root node\u003c/p\u003e\n\u003cp\u003eSimple process:\u003c/p\u003e\n\u003cp\u003eUsing some splitting criterion, the optimal feature is selected to split the samples, and this is done recursively, finally yielding the branch nodes of the decision tree\u003c/p\u003e","title":"Decision Tree"},{"content":" This time I am putting together the commonly used scikit-learn packages as a memo\nK-nearest neighbors:\nfrom sklearn.neighbors import KNeighborsClassifier\nLinear regression:\nfrom sklearn.linear_model import LinearRegression\nLogistic regression:\nfrom sklearn.linear_model import LogisticRgression\nDecision tree:\nfrom sklearn.tree import DecisionTreeClassifier\nNaive Bayes:\nGaussian distribution: from sklearn.neive_bayes import GaussianNB Multinomial distribution: from sklearn.neive_baybes import MultinomnalNB Bernoulli distribution: from sklearn.neive_baybes import BernoulliNB Support vector machine:\nClassification: from sklearn.svm import SVC Regression: from sklearn.svm import SVR Ensemble algorithms (sklearn.ensemble):\nbagging: Classification: from sklearn.ensemble import BaggingClassifier\nRegression: from sklearn.ensemble import BaggingRegression\nAdaBoost: Classification: from sklearn.ensemble import AdaBoostClassifier\nRegression: from sklearn.ensemble import AdaBoostRegression\nRandom forest: Classification: from sklearn.ensemble import RandomForestClassifier\nRegression: from sklearn.ensemble import RandomForestRegression\nK-means:\nfrom sklearn.cluster import KMeans\nPCA dimensionality reduction:\nfrom sklearn.decomposition import PCA\nFinally, a scikit treasure map:\nscikit-learn cheat sheet\n","permalink":"https://simplestory.github.io/en/posts/20180826_8cb44c1627565d8ede47d43e49e9c8ae/","summary":"\u003cblockquote\u003e\n\u003cp\u003eThis time I am putting together the commonly used scikit-learn packages as a memo\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eK-nearest neighbors:\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html\"\u003efrom sklearn.neighbors import KNeighborsClassifier\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eLinear regression:\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html\"\u003efrom sklearn.linear_model import LinearRegression\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eLogistic regression:\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html\"\u003efrom sklearn.linear_model import LogisticRgression\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eDecision tree:\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html\"\u003efrom sklearn.tree import DecisionTreeClassifier\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eNaive Bayes:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eGaussian distribution: \u003ca href=\"http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.GaussianNB.html\"\u003efrom sklearn.neive_bayes import GaussianNB\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eMultinomial distribution: \u003ca href=\"http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html#sklearn.naive_bayes.MultinomialNB\"\u003efrom sklearn.neive_baybes import MultinomnalNB\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eBernoulli distribution: \u003ca href=\"http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.BernoulliNB.html#sklearn.naive_bayes.BernoulliNB\"\u003efrom sklearn.neive_baybes import BernoulliNB\u003c/a\u003e\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eSupport vector machine:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eClassification: \u003ca href=\"http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html\"\u003efrom sklearn.svm import SVC\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eRegression: \u003ca href=\"http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVR.html\"\u003efrom sklearn.svm import SVR\u003c/a\u003e\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eEnsemble algorithms (sklearn.ensemble):\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003ebagging:\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eClassification: \u003ca href=\"http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingClassifier.html\"\u003efrom sklearn.ensemble import BaggingClassifier\u003c/a\u003e\u003c/p\u003e","title":"A Scikit-learn Cheat Sheet Collection"},{"content":" Bayesian decision theory considers how to choose the optimal class label based on the relevant probabilities and misclassification losses\nSuppose there are N possible class labels, that is $\\mathbf{y}=\\\\{c_1,c_2,...,c_N\\\\}$, and $\\lambda_{ij}$ is the loss incurred by misclassifying a sample whose true label is $c_j$ as $c_i$. Based on the posterior probability $P(c_i\\vert\\mathbf{x})$, we can obtain the expected loss incurred by classifying sample $\\mathbf{x}$ as $c_i$, that is, the \u0026ldquo;conditional risk\u0026rdquo; on sample $\\mathbf{x}$:\n$$R(c_i\\vert\\mathbf{x})= \\ \\sum_{j=1}^N \\lambda_{ij}P(c_j\\vert\\mathbf{x})$$Our goal is to find a decision rule $h: \\ \\mathbf{x} \\mapsto \\mathbf{y}$ that minimizes the overall risk:\n$$R(h)=E_{\\mathbf{x}}[R(h(\\mathbf{x})\\vert\\mathbf{x})]$$Since for every sample $\\mathbf{x}$, if h can minimize the conditional risk $R(h(\\mathbf{x})\\vert\\mathbf{x})$, the overall risk $R(h)$ will also be minimized, so we have:\nBayesian decision rule:\nTo minimize the overall risk, one only needs to choose, for each sample, the class label that minimizes the conditional risk $R(c\\vert\\mathbf{x})$, that is:\n$$h^*(\\mathbf{x})=argmin_{c\\in\\mathbf{y}} \\ R(c\\vert\\mathbf{x})$$Here $h^*$ is called the Bayes optimal classifier, and the corresponding overall risk $R(h^*)$ is called the Bayes risk; $1-R(h^*)$ reflects the best performance a classifier can achieve, that is, the theoretical upper bound of the model accuracy obtainable through machine learning\nNaive Bayes Naive Bayes learns the joint probability distribution $P(X,Y)$ from the training dataset, specifically learning the following prior probability distribution and conditional probability distributions:\nPrior probability distribution: $\\ P(Y=c_k), \\ k=1,2,...,N$\nConditional probability distribution: $\\ P(X=x\\vert Y=c_k)=P(X^{(1)}=x^{(1)},...,X^{(n)}=x^{(n)}\\vert Y=c_k), \\ k=1,2,...N$\nSo the joint probability distribution is: $P(X,Y)=P(Y=c_k)P(X=x\\vert Y=c_k)$\nDerivation Naive Bayes has an important assumption, the conditional independence assumption, namely:\n$$ \\begin{aligned} P(X=x\\vert Y=c_k) \u0026 =P(X^{(1)}=x^{(1)},...,X^{(n)}=x^{(n)}\\vert Y=c_k) \\\\ \u0026 =\\prod_{j=1}^n P(X^{(j)}=x^{(j)}\\vert Y=c_k) \\end{aligned} \\tag{1} $$By Bayes\u0026rsquo; theorem the posterior probability is:\n$$P(Y=c_k\\vert X=x)=\\frac{P(X=x\\vert Y=c_k)P(Y=c_k)}{\\sum_k P(X=x\\vert Y=c_k)P(Y=c_k)}$$Substituting equation (1) into the above gives:\n$$ \\begin{aligned} \u0026 P(Y=c_k\\vert X=x)=\\frac{P(Y=c_k)\\prod_j P(X^{(j)}=x^{(j)}\\vert Y=c_k)}{\\sum_k P(Y=c_k)\\prod_j P(X^{(j)}=x^{(j)}\\vert Y=c_k)} \\\\ \u0026 \\text{where} \\ k=1,2,...,k \\end{aligned} \\tag{2} $$Then the naive Bayes classifier can be expressed as:\n$$ y=f(x)=argmax_{c_k} \\ \\frac{P(Y=c_k)\\prod_j P(X^{(j)}=x^{(j)}\\vert Y=c_k)}{\\sum_k P(Y=c_k)\\prod_j P(X^{(j)}=x^{(j)}\\vert Y=c_k)} \\tag{3} $$Note that in equation (3) the denominator is the same for all $c_k$, so we have:\n$$ y=argmax_{c_k} \\ p(Y=c_k)\\prod_j P(X^{(j)}=x^{(j)}\\vert Y=c_k) $$Use in Scikit-learn See the official site for the concrete API parameters:\nsklearn.naive_bayes\nReference Zhou Zhihua\u0026rsquo;s Watermelon Book and Li Hang\u0026rsquo;s Statistical Learning Methods\n","permalink":"https://simplestory.github.io/en/posts/20180819_72c962ee57eaf5ebe2b4645dd6827ce3/","summary":"\u003cblockquote\u003e\n\u003cp\u003eBayesian decision theory considers how to choose the optimal class label based on the relevant probabilities and misclassification losses\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eSuppose there are N possible class labels, that is $\\mathbf{y}=\\\\{c_1,c_2,...,c_N\\\\}$, and $\\lambda_{ij}$ is the loss incurred by misclassifying a sample whose true label is $c_j$ as $c_i$. Based on the posterior probability $P(c_i\\vert\\mathbf{x})$, we can obtain the expected loss incurred by classifying sample $\\mathbf{x}$ as $c_i$, that is, the \u0026ldquo;conditional risk\u0026rdquo; on sample $\\mathbf{x}$:\u003c/p\u003e","title":"Bayesian Classification"},{"content":" SVM is a very important model in machine learning, proposed by Vladimir Vapnik, the father of statistical learning theory\nThis post takes binary classification with SVM as the example\nDerivation Given the training dataset $D=\\\\{(\\mathbf{x_1},y_1),(\\mathbf{x_2},y_2),...,(\\mathbf{x_m},y_m)\\\\}, y_i \\in \\\\{-1, +1\\\\}$, the most basic idea of a classifier is to find in the sample space a dividing space based on the training set D that separates the samples of different classes. The dividing hyperplane can be described by the following linear equation:\n$$ {\\mathbf{w}}^T\\mathbf{x}+b=0$$where $\\mathbf{w}=(w_1;w_2;...;w_d)$ is the normal vector, which determines the direction of the hyperplane, and b is the offset term, which determines the distance between the hyperplane and the origin; thus the hyperplane is determined by w and d. Then the distance from any point in space to the hyperplane (w,b) is:\n$$\\gamma = \\frac{\\vert{\\mathbf{w}}^T\\mathbf{x}+b\\vert}{\\Vert\\mathbf{w}\\Vert}$$Suppose the hyperplane (w,b) classifies the samples completely correctly, that is, for $(\\mathbf{x_i}, y_i) \\in D$, if $y_i=+1$ then ${\\mathbf{w}}^T\\mathbf{x_i}+b \u003e 0$; if $y_i=-1$ then ${\\mathbf{w}}^T\\mathbf{x_i}+b \u003c 0$, so we have:\n$$ \\begin{cases} {\\mathbf{w}}^T\\mathbf{x_i}+b \\ge +1, \\ y_i = +1 \\\\ {\\mathbf{w}}^T\\mathbf{x_i}+b \\le -1, \\ y_i = -1 \\end{cases} \\tag{1} $$The formula above can be combined into: $y_i({\\mathbf{w}}^T\\mathbf{x_i}+b) \\ge 1, i=1,2,...,m$\nFor the few training sample points closest to the hyperplane, equation (1) holds with equality; these sample points are called \u0026ldquo;support vectors\u0026rdquo;, and the sum of the distances from the two support vectors of different classes to the hyperplane is:\n$$\\gamma = \\frac{2}{\\Vert\\mathbf{w}\\Vert}$$which is called the \u0026ldquo;margin\u0026rdquo;\nSVM finds the dividing hyperplane with the maximum margin, that is, it satisfies the following:\n$$ \\begin{aligned} \\min_{\\mathbf{w},b} \u0026 \\frac{1}{2}||\\mathbf{w}||^2 \\\\ s.t. \u0026 \\ y_i({\\mathbf{w}}^T\\mathbf{x_i}+b) \\ge 1, i=1,2,...,m \\end{aligned} \\tag{2} $$The formula above is the basic form of SVM\nApplying the Lagrange multiplier method to (2) gives:\n$$ L(\\mathbf{w},b,\\mathbf{\\alpha}) = \\frac{1}{2}{\\Vert\\mathbf{w}\\Vert}^2 + \\sum_{i=1}^m \\alpha_i(1-y_i({\\mathbf{w}}^T\\mathbf{x_i}+b)) \\tag{3} $$where $\\alpha = (\\alpha_1;\\alpha_2;...;\\alpha_m)$. Setting the partial derivatives of $L(w,b,\\alpha)$ with respect to w and b to 0:\n$$ \\begin{aligned} \\frac{\\partial L(\\mathbf{w},b,\\mathbf{\\alpha})}{\\partial \\mathbf{w}} \\ = \\mathbf{w} - \\sum_{i=1}^m \\alpha_iy_i\\mathbf{x_i} = 0 \\\\ \\frac{\\partial L(\\mathbf{w},b,\\mathbf{\\alpha})}{\\partial b} \\ = 0+\\sum_{i=1}^m \\alpha_iy_i = 0 \\end{aligned} $$then we obtain:\n$$ \\begin{aligned} \\mathbf{w} = \\sum_{i=1}^m \\alpha_iy_i\\mathbf{x_i} \\\\ 0 = \\sum_{i=1}^m \\alpha_iy_i \\end{aligned} \\tag{4} $$Substituting (4) into (3) gives:\n$$ \\begin{aligned} L(\\mathbf{w},b,\\mathbf{\\alpha}) \u0026 = \\frac{1}{2}{\\mathbf{w}}^T\\mathbf{w}+\\sum_{i=1}^m \\alpha_{1} -\\sum_{i=1}^m \\alpha_iy_i{\\mathbf{w}}^T\\mathbf{x_i}-\\sum_{t=1}^m \\alpha_iy_ib \\\\ \u0026 =\\frac{1}{2}\\sum_{i=1}^m \\alpha_iy_i{\\mathbf{x_i}}^T\\cdot\\sum_{j=1}^m \\alpha_jy_j\\mathbf{x_j}+\\sum_{i=1}^m \\alpha_i-\\sum_{i=1}^m \\sum_{j=1}^m \\alpha_iy_i\\alpha_jy_j{\\mathbf{x_j}}^T\\mathbf{x_i} \\\\ \u0026 =\\sum_{i=1}^m \\alpha_i + \\frac{1}{2}\\sum_{i=1}^m \\sum_{j=1}^m \\alpha_iy_i\\alpha_jy_j{\\mathbf{x_i}}^T\\mathbf{x_j}-\\sum_{i=1}^m \\sum_{j=1}^m \\alpha_iy_i\\alpha_jy_j{\\mathbf{x_j}}^T\\mathbf{x_i} \\\\ \u0026 =\\sum_{i=1}^m \\alpha_i - \\frac{1}{2}\\sum_{i=1}^m \\sum_{j=1}^m \\alpha_iy_i\\alpha_jy_j{\\mathbf{x_i}}^T\\mathbf{x_j} \\end{aligned} $$so we have:\n$$ \\begin{aligned} \\max_\\mathbf{\\alpha} \u0026 \\sum_{i=1}^m \\alpha_i - \\frac{1}{2}\\sum_{i=1}^m \\sum_{j=1}^m \\alpha_i\\alpha_jy_iy_j\\mathbf{x_i}^T\\mathbf{x_j} \\\\ s.t. \u0026 \\sum_{i=1}^m \\alpha_iy_i=0, \\\\ \u0026 \\alpha_i \\ge 0, i=1,2,...,m \\end{aligned} \\tag{5} $$After solving for $\\mathbf{\\alpha}$, $\\mathbf{w}$ and $\\mathbf{b}$ are obtained and the model is as follows:\n$$ f(\\mathbf{x})={\\mathbf{w}}^T\\mathbf{x}+b=\\sum_{i=1}^m \\alpha_iy_i{\\mathbf{x_i}}^T\\mathbf{x}+b $$Solving (5) usually uses the SMO algorithm to obtain the value of $\\mathbf{\\alpha}$, after which b is computed with (6):\n$$ b = \\frac{1}{\\vert{S}\\vert}\\sum_{s \\in S}(\\frac{1}{y_c}-\\sum_{i \\in s}\\alpha_iy_i{\\mathbf{x_i}}^T\\mathbf{x_s}) \\tag{6} $$Kernel trick In the discussion above we assumed the data is linearly separable, but real data is in most cases not linearly separable. For this reason we can consider applying a mapping to the original data to make it linearly separable, and then hand it to the SVM model for training. Specifically:\nSuppose the mapping function is $\\phi(\\mathbf{x})$; then: $f(\\mathbf{x})={\\mathbf{w}}^T\\phi(\\mathbf{x})+b$\n$$ \\begin{aligned} \\min_{\\mathbf{w},b} \u0026 \\frac{1}{2}||\\mathbf{w}||^2 \\\\ s.t. \u0026 \\ y_i({\\mathbf{w}}^T\\phi(\\mathbf{x_i})+b) \\ge 1, i=1,2,...,m \\end{aligned} $$Its dual problem is:\n$$ \\begin{aligned} \\max_\\mathbf{\\alpha} \u0026 \\sum_{i=1}^m \\alpha_i-\\frac{1}{2}\\sum_{i=1}^m \\sum_{j=1}^m \\alpha_i\\alpha_jy_iy_j\\phi(\\mathbf{x_i})^T\\phi(\\mathbf{x_j}) \\\\ s.t. \u0026 \\ \\sum_{i=1}^m \\alpha_iy_i=0 \\\\ \u0026 \\alpha_i \\ge 0, i=1,2,...,m \\end{aligned} $$The term $\\phi(\\mathbf{x_i})^T\\phi(\\mathbf{x_j})$ appearing in the formula above can be replaced by a function $k(\\mathbf{x_i},\\mathbf{x_j})$, namely the kernel function\n$$k(\\mathbf{x_i},\\mathbf{x_j})=\\langle \\phi(\\mathbf{x_i}),\\phi(\\mathbf{x_j}) \\rangle=\\phi(\\mathbf{x_i})^T\\phi(\\mathbf{x_j})$$so we have:\n$$ \\begin{aligned} \\max_\\mathbf{\\alpha} \u0026 \\sum_{i=1}^m \\alpha_i - \\frac{1}{2}\\sum_{i-1}^m \\sum_{j=1}^m \\alpha_i\\alpha_jy_iy_jk(\\mathbf{x_i},\\mathbf{x_j}) \\\\ s.t. \u0026 \\ \\sum_{i=1}^m \\alpha_iy_i = 0, \\\\ \u0026 \\alpha_i \\ge 0, i=1,2,..,m \\end{aligned} $$After solving:\n$$ \\begin{aligned} f(x) \u0026 ={\\mathbf{w}}^T\\phi(\\mathbf{x})+b \\\\ \u0026 =\\sum_{i=1}^m \\alpha_iy_i\\phi(\\mathbf{x_i})^T\\phi(\\mathbf{x_j})+b \\\\ \u0026 =\\sum_{i=1}^m \\alpha_iy_ik(\\mathbf{x},\\mathbf{x_i})+b \\end{aligned} $$Commonly used kernel functions:\nName Expression Parameters Linear kernel $k(\\mathbf{x_i},\\mathbf{x_j})=\\mathbf{x_i}^T\\mathbf{x_j}$ Polynomial kernel $k(\\mathbf{x_i},\\mathbf{x_j})=(\\mathbf{x_i}^T\\mathbf{x_j})^d$ $d \\ge 1$, the degree of the polynomial Gaussian kernel $k(\\mathbf{x_i},\\mathbf{x_j})=exp(-\\frac{\\Vert\\mathbf{x_i}-\\mathbf{x_j}\\Vert^2}{2\\sigma^2})$ $\\sigma \\gt 0$, the bandwidth of the Gaussian kernel Laplacian kernel $k(\\mathbf{x_i},\\mathbf{x_j})=exp(-\\frac{\\Vert\\mathbf{x_i}-\\mathbf{x_j}\\Vert}{\\sigma})$ $\\sigma \\gt 0$ Sigmoid kernel $k(\\mathbf{x_i},\\mathbf{x_j})=tanh(\\beta \\mathbf{x_i}^T\\mathbf{x_j}+\\theta)$ tanh is the hyperbolic tangent function, $\\beta \\gt 0$, $\\theta \\gt 0$ Usual choices:\nThe linear kernel is generally used for linearly separable data or text data\nThe Gaussian kernel is generally used when the data is not linearly separable or the data situation is unclear\nRegularization of SVM To prevent overfitting, we usually use the soft margin support vector machine, which allows some samples to fall inside the margin; the model equation is then:\n$$ \\min_{\\mathbf{w},b} \\frac{1}{2}{\\Vert\\mathbf{w}\\Vert}^2+C\\sum_{i=1}^m l_{0/1}(y_i({\\mathbf{w}}^T\\mathbf{x_i}+b)-1) \\tag{7} $$where C\u0026gt;0 is a constant and $l_{0/1}$ is the 0/1 loss function:\n$$ l_{0/1}(z)=\\begin{cases} 1, \\ \\text{if } \\ z \\le 0; \\\\ 0, \\ \\text{otherwise} \\end{cases} $$Other functions are usually used to replace $l_{0/1}$, as follows:\n$$ \\begin{cases} \\text{hinge loss: } \\ l_{hinge}(z)=\\max(0,1-z); \\\\ \\text{exponential loss: } \\ l_{exp}(z)=exp(-z); \\\\ \\text{logistic loss: } \\ l_{log}(z)=log(1+exp(-z)) \\end{cases} $$Referring to (7), the more general form is:\n$$\\min_f \\Omega(f)+C\\sum_{i=1}^m l(f(\\mathbf{x_i}),y_i)$$where $\\Omega(f)$ is the structural risk, used to describe certain properties of the model f ($\\Omega(f)$ can also be called the regularization term, commonly $L_0, L_1, L_2$ regularization), and the second term $\\sum_{i=1}^m l(f(\\mathbf{x_i}),y_i)$ is the empirical risk, used to describe how well the model fits the training data, with C striking a balance between the two.\nUse in Scikit-learn See the official Scikit-learn site for the concrete API:\nsklearn.svm.LinearSVC\nsklearn.svm.SVC\nReference Zhou Zhihua\u0026rsquo;s Watermelon Book and Li Hang\u0026rsquo;s Statistical Learning Methods\n","permalink":"https://simplestory.github.io/en/posts/20180816_7ed73172fd758adf99c76ebc108ea063/","summary":"\u003cblockquote\u003e\n\u003cp\u003eSVM is a very important model in machine learning, proposed by Vladimir Vapnik, the father of statistical learning theory\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003e\u003cstrong\u003eThis post takes binary classification with SVM as the example\u003c/strong\u003e\u003c/p\u003e\n\u003ch2 id=\"derivation\"\u003eDerivation\u003c/h2\u003e\n\u003cp\u003eGiven the training dataset $D=\\\\{(\\mathbf{x_1},y_1),(\\mathbf{x_2},y_2),...,(\\mathbf{x_m},y_m)\\\\}, y_i \\in \\\\{-1, +1\\\\}$, the most basic idea of a classifier is to find in the sample space a dividing space based on the training set D that separates the samples of different classes. The dividing hyperplane can be described by the following linear equation:\u003c/p\u003e","title":"SVM Support Vector Machine"},{"content":" I have not updated the blog for a long time, having paused for about half a year, and I have also read half a year\u0026rsquo;s worth of books on machine learning and deep learning; now it is time to write some output\nLinear regression Given a dataset $D=\\\\{(\\mathbf{x_1},y_1),(\\mathbf{x_2},y_2),...,(\\mathbf{x_m},y_m)\\\\}$, where $\\mathbf{x_i}=(x_{i1};x_{i2};...;x_{id}), \\ y_{i} \\in R$. Linear regression tries to learn a linear model\n$$f(x_i)=wx_i+b$$such that $f(x_i)\\simeq y_i$.\nDerivation Consider using MSE (mean squared error) to measure how good the model is; then the loss function $L(w,b)$ is:\n$$ \\begin{aligned} L(w,b) \u0026 =\\sum_{i=1}^m (f(x_i)-y_i)^2 \\\\ \u0026 =\\sum_{i-1}^m (y_i-wx_i-b)^2 \\end{aligned} $$The objective function is:\n$$ \\begin{aligned} (w^*,b^*) \u0026 =argmin_{(w,b)} L(w,b) \\\\ \u0026 =argmin_{(w,b)} \\sum_{i=1}^m (y_i-w_i-b)^2 \\end{aligned} $$Taking the partial derivatives of $L(w,b)$ with respect to $w$ and $b$ respectively, setting the derivatives to zero and solving the resulting system gives\n$$ \\begin{cases} \\nabla_w L(w,b) = 2(w\\sum_{i=1}^m x_i^2 -\\sum_{i=1}^m (y_i-b)x_i )=0 \\\\ \\nabla_n L(w,b) = a(mb-\\sum_{i=1}^m (y_i-wx_i))=0 \\end{cases} $$Solving gives the optimal closed-form solution for $w$ and $b$:\n$$w=\\frac{\\sum_{i=1}^m y_i(x_i-\\overline{x})}{\\sum_{i=1}^m x_i^2 - \\frac{1}{m}(\\sum_{i=1}^m x_i)^2}$$$$b=\\frac{1}{m}\\sum_{i=1}^m (y_i-wx_i)$$where $\\overline{x}=\\frac{1}{m} \\sum_{i=1}^m x_i$ is the mean of $x$\nUse in Scikit-learn See the official Scikit-learn site for the concrete API: sklearn.linear_model.LinearRegression\nLogistic regression This is a classification method\nLogistic distribution:\nLet X be a continuous random variable; $\\mathbf{X}$ follows a logistic distribution means that $\\mathbf{X}$ has the following distribution function and density function:\n$$ \\begin{cases} F(x)=P(X \\leq x)=\\frac{1}{1+e^{-(x-\\mu)/\\gamma}} \\\\ f(x)=F\\prime(x)=\\frac{e^{-(x-\\mu)/\\gamma}}{\\gamma(1+e^(x-\\mu)/\\gamma)^2} \\end{cases} $$where $\\mu$ is the location parameter and $\\gamma \u003e 0$ is the shape parameter\nThe binomial logistic regression model gives the following conditional probabilities:\n$$P(Y=1|x)=\\frac{exp(w\\cdot x+b)}{1+exp(w\\cdot x+b)}$$$$P(Y=0|x)=\\frac{1}{1+exp(w\\cdot x+b)}$$Here $x\\in R^n$ is the input, $Y\\in \\\\{0,1\\\\}$ is the output, $w\\in R^n$ is the weight vector, $b\\in R$ is the bias and $w\\cdot x$ is the inner product.\nFor convenience of expression, merging w and b gives:\n$$P(Y=1|x)=\\frac{exp(w\\cdot x)}{1+exp(w\\cdot x)}$$$$P(Y=0|x)=\\frac{1}{1+exp(w\\cdot x)}$$From the assumptions and conclusions above, the logistic regression model returns the probability that the sample is positive, that is, $P(Y=1\\vert x)$\nDerivation From the maximum likelihood function we obtain the loss function $L(w)$:\n$$L(w)=\\prod_{i=1}^N P(Y=1|x_i)^{y_i}[1-P(Y=0|x_i)]^{1-y_i}$$To prevent overflow caused by the chained multiplication, take the logarithm:\n$$L(w)=\\sum_{i=1}^N [y_i\\log{P(Y=1|x_i)}+(1-y_i)\\log{1-P(Y=0|x_i)}]$$which simplifies to:\n$$L(w)=\\sum_{i=1}^N [y_i(w\\cdot x_i)-\\log{(1+exp(w\\cdot x_i))}]$$Maximizing $L(w)$ gives the estimate of w. Gradient descent and quasi-Newton methods are usually used to solve it\nUse in Scikit-learn sklearn.linear_model.LogistRegression\nReference Zhou Zhihua\u0026rsquo;s Watermelon Book and Li Hang\u0026rsquo;s Statistical Learning Methods\n","permalink":"https://simplestory.github.io/en/posts/20180813_135c495c1e86bceb067f1bc3135c278b/","summary":"\u003cblockquote\u003e\n\u003cp\u003eI have not updated the blog for a long time, having paused for about half a year, and I have also read half a year\u0026rsquo;s worth of books on machine learning and deep learning; now it is time to write some output\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch1 id=\"linear-regression\"\u003eLinear regression\u003c/h1\u003e\n\u003cp\u003eGiven a dataset $D=\\\\{(\\mathbf{x_1},y_1),(\\mathbf{x_2},y_2),...,(\\mathbf{x_m},y_m)\\\\}$, where $\\mathbf{x_i}=(x_{i1};x_{i2};...;x_{id}), \\ y_{i} \\in R$. Linear regression tries to learn a linear model\u003c/p\u003e\n$$f(x_i)=wx_i+b$$\u003cp\u003esuch that $f(x_i)\\simeq y_i$.\u003c/p\u003e\n\u003ch2 id=\"derivation\"\u003eDerivation\u003c/h2\u003e\n\u003cp\u003eConsider using MSE (mean squared error) to measure how good the model is; then the loss function $L(w,b)$ is:\u003c/p\u003e","title":"Linear Regression and Logistic Regression"},{"content":" G++ is the C++ compiler under Unix/Linux, and gdb is a powerful debugging tool under Unix/Linux. Most of the time, the C/C++ code we write does not run on Windows but on a Linux system. Here I record some usage of g++ and gdb in case they are needed.\nUsing g++ When g++ performs compilation work, it needs 4 steps in total:\nPreprocessing, generating a .i file, mainly expanding header files and macro definitions: g++ -E test.cpp -o test.i Compilation, converting the preprocessed file into assembly language and generating a .s file: g++ -S test.i -o test.s Assembly, turning the assembly into object code (machine code) and generating a .o file, mainly converting the .s file into a binary file: g++ -c test.s -o test.o Linking, linking the object code to generate an executable program: g++ test.o -o test The steps above can also be done with a single command:\ng++ test.cpp -o test\nWhen compiling multiple files, there are two approaches:\n$ g++ a.cpp b.cpp c.cpp -o test #compile all files at once $ g++ a.cpp -o a.o #compile separately $ g++ b.cpp -o b.o $ g++ c.cpp -o c.o $ g++ a.o b.o c.o -o test Here the separate compilation approach is recommended, since when you modify one or more of the files you only need to compile that file or those files, rather than compiling from scratch.\nCommonly used g++ options:\nCommand Description -Wall Output some warning and error messages during compilation -c Compile only, do not link -E Preprocess the source file -S Convert the preprocessed file into an assembly file -o Specify the output file name -I Specify a path containing header files -l Specify an additional library -L Specify an additional library path -shared Generate a dynamic library file Under Linux the file type does not depend on the suffix, but generally speaking: .o is an object file, equivalent to a .obj file in windows .so is a shared library, that is, a shared object used for dynamic linking, roughly the same as .dll .a is a static library, several .o files combined together, used for static linking .la are some shared libraries automatically generated by libtool, mainly recording some configuration information.\nUsing gdb The .cpp must be compiled with the -g option for the generated executable to be debuggable at source level with gdb\ng++ -g test.cpp -o test\nIt is worth noting that -g has 4 levels:\ng0 is the same as not adding -g. That is, no information is included g1 includes only minimal information; generally it is used only when you do not need to debug but only need backtrace information, and you really care about program size or have some other confidentiality/special requirement. g2 is the gdb default level and includes most of the information you need. g3 includes some extra information, for example macro definition information. When you need to debug macro definitions, use -g3 Generally speaking, gdb mainly helps you accomplish the following four things:\nStart the program, and run the program according to your own requirements.\nLet the debugged program stop at a breakpoint at a specified position.\nWhen the program stops, inspect what is happening in the program at that moment.\nDynamically change the execution environment of the program.\nSeveral ways to run a program under gdb:\ngdb ${program} then, after entering gdb, type run(short form r) ${arg1} ${arg2} … ${argN} gdb --args ${program} ${arg1} ${arg2} … ${argN}, then after entering gdb run run After entering gdb, type file ${program}. Then use set args ${arg1} ${arg2} … ${argN} to set the program arguments, and then run run Commonly used gdb debugging commands:\nCommand Description start Start executing the program and stop before the first statement of the main function backtrace/bt View the function calls at each level and display the stack information step/s Single-step debugging, stepping into the current function next/n Single-step debugging, stepping over the current function until/u Execute until the current loop is finished continue/c Continue running the program finish Execute until the current function returns list/l List the source code, 10 lines at a time list \u0026lt;line number\u0026gt; List the source code starting from the given line list \u0026lt;function name\u0026gt; List the source code of the function print/p \u0026lt;expression\u0026gt; Print the value of the expression frame/f x Switch to frame x, where x is shown in the bt command, starting from 0, with 0 meaning the top of the stack up/down x Move x frames towards the top/bottom of the stack; when x is not given it defaults to 1 info/i locals Print the local variables of the current stack frame set var x Modify the value of the current variable x break \u0026lt;line number\u0026gt; Set a breakpoint at the specified line of a source file break \u0026lt;function name\u0026gt; Set a breakpoint at the specified function of a source file break \u0026lt;line number/function name\u0026gt; if \u0026lt;condition\u0026gt; Set a conditional breakpoint; when the condition is met the program breaks at the location watch \u0026lt;variable\u0026gt; Write watch on the specified variable; when the variable changes the program stops running rwatch \u0026lt;variable\u0026gt; Read watch on the specified variable; when the variable is read the program stops running awatch \u0026lt;variable\u0026gt; Read/write watch on the specified variable; the program stops running when the variable changes or is read delete \u0026lt;line number/function name\u0026gt; Delete a breakpoint info/i b View breakpoint information save breakpoint \u0026lt;file name\u0026gt;.dp Save the breakpoints into the specified file -x \u0026lt;file name\u0026gt;.dp This option reads the specified file and restores the breakpoint settings in it quit Quit gdb debugging Reference macky0668 — basic usage of the g++ C++ compiler under linux\nXi Shan Ju Shi who loves tinkering — a series on commonly used GDB commands\nchenyu who is changing himself — a summary of basic gdb debugging commands and usage under linux\nGetting started with gdb debugging, a high-quality guide written by an expert\n","permalink":"https://simplestory.github.io/en/posts/20180302_5ed219da904f7b6718ebebb4c8b47d3c/","summary":"\u003cblockquote\u003e\n\u003cp\u003eG++ is the C++ compiler under Unix/Linux, and gdb is a powerful debugging tool under Unix/Linux. Most of the time, the C/C++ code we write does not run on Windows but on a Linux system. Here I record some usage of g++ and gdb in case they are needed.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch3 id=\"using-g\"\u003eUsing g++\u003c/h3\u003e\n\u003cp\u003eWhen g++ performs compilation work, it needs 4 steps in total:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003ePreprocessing, generating a .i file, mainly expanding header files and macro definitions: \u003ccode\u003eg++ -E test.cpp -o test.i\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003eCompilation, converting the preprocessed file into assembly language and generating a .s file: \u003ccode\u003eg++ -S test.i -o test.s\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003eAssembly, turning the assembly into object code (machine code) and generating a .o file, mainly converting the .s file into a binary file: \u003ccode\u003eg++ -c test.s -o test.o\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003eLinking, linking the object code to generate an executable program: \u003ccode\u003eg++ test.o -o test\u003c/code\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThe steps above can also be done with a single command:\u003c/p\u003e","title":"Using g++ and gdb"},{"content":" At the time of the 2018 Spring Festival, heaven knows what I went through, posting a blog article on the fourth day of the new year. Django was actually configured before the New Year; I had just learned some knowledge about developing web applications with python at that time, so I configured it along the way.\nEnvironment used for the configuration\nSystem: Ubuntu16.04 (LTS) python version: python3.6 (using Anaconda to create a virtual environment) Preparation Uninstalling an old version of Django If you are upgrading from an older version of Django, you need to deal with the old version a little.\nIf the previous version of Django was installed through pip or easy_install, running pip or easy_install again will automatically handle the old version of Django, so you can install directly without any extra work. If it was previously installed through python setup.py install, uninstalling means deleting the Django library from the python packages; the path of the Django library can be found with the following command: $ python -c \u0026#34;import django; print(django.__path__)\u0026#34; Installing Apache (optional) If Django is used in a real environment rather than a test development environment, Apache and mod_wsgi need to be installed. Before installing Apache, APR, APR-Util and Pcre need to be installed.\nInstalling APR: Download the APR package, decompress it, and run the following commands in the decompressed folder (the folder name should not contain the version number) to install:\n$ ./configure --prefix=/usr/local/apr $ make $ make install Installing APR-Util: Likewise, download the APR-Util package, decompress it, and run the following commands in the decompressed folder (the folder name should not contain the version number) to install:\n$ ./configure --prefix=/usr/local/apr-util --with-apr=/usr/local/apr $ make $ make install Installing Pcre: Download the Pcre package; note that it is pcre rather than pcre2. Then decompress it and run in the decompressed folder:\n$ ./configure --prefix=/usr/local/pcre $ make $ make install Installing Apache2: Download the Apache2 package, decompress it, enter the folder and run the following commands:\n$ ./configure --prefix=/usr/local/apache2 --with-apr=/usr/local/apr --with-apr-util=/usr/local/apr-util/ --with-pcre=/usr/local/pcre $ make $ make install Installing mod_wsgi Install it with the following commands:\n$ sudo apt-get install apache2-dev $ sudo pip install mod_wsgi Installing Django First create a virtual environment:\n$ conda create -n django_py36 python=3.6 Install Django in the virtual environment:\n(django_py36)$ sudo pip install Django At this point the Django installation is finished.\nReference Django Documentation\nApache documentation\nfresheer\u0026rsquo;s blog — installing Apache and mod_wsgi on ubuntu\n","permalink":"https://simplestory.github.io/en/posts/20180219_c851e812699d1207837d404c0af593ce/","summary":"\u003cblockquote\u003e\n\u003cp\u003eAt the time of the 2018 Spring Festival, heaven knows what I went through, posting a blog article on the fourth day of the new year. Django was actually configured before the New Year; I had just learned some knowledge about developing web applications with python at that time, so I configured it along the way.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003e\u003cstrong\u003eEnvironment used for the configuration\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eSystem: Ubuntu16.04 (LTS)\u003c/li\u003e\n\u003cli\u003epython version: python3.6 (using Anaconda to create a virtual environment)\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"preparation\"\u003ePreparation\u003c/h2\u003e\n\u003ch4 id=\"uninstalling-an-old-version-of-django\"\u003eUninstalling an old version of Django\u003c/h4\u003e\n\u003cp\u003eIf you are upgrading from an older version of Django, you need to deal with the old version a little.\u003c/p\u003e","title":"Configuring Django on Ubuntu"},{"content":" Borrowing a line from the TensorFlow Chinese community: TensorFlow, an open-source magic tool for artificial intelligence.\nEnvironment used for the configuration\nSystem: Ubuntu16.04 (LTS) python version: python3.6 (using Anaconda to create a virtual environment) Nvidia card: Geforce 920M (if there is no Nvidia card, only the CPU version can be installed) Installing Tensorflow tensorflow has two versions, CPU and GPU; I created two python virtual environments and installed these two versions separately. Below I record this painful installation process\ntensorflow CPU Create a virtual environment conda create -n tensorflow python=3.6 source activate tensorflow #enter the virtual environment Install tensorflow Installing the CPU version is fairly simple — just use pip pip install tensorflow If the installation reports an error, you can choose to download and install it: Go to pypi and pick the appropriate version to download, then run the following command in the directory where the file is placed:\n#the file name here is your own pip install tensorflow-1.6.0rc0-cp36-cp36m-manylinux1_x86_64.whl Verify the installation Enter the python environment and test:\n$ python \u0026gt;\u0026gt;\u0026gt; import tensorflow as tf \u0026gt;\u0026gt;\u0026gt; hello = tf.constant(\u0026#39;Hello TensorFlow!\u0026#39;) \u0026gt;\u0026gt;\u0026gt; sess = tf.Session() \u0026gt;\u0026gt;\u0026gt; print(sess.run(hello)) Hello Tensorflow! \u0026gt;\u0026gt;\u0026gt; a = tf.constant(12) \u0026gt;\u0026gt;\u0026gt; b = tf.constant(23) \u0026gt;\u0026gt;\u0026gt; print(sess.run(a+b)) 35 \u0026gt;\u0026gt;\u0026gt; Or import the tensorflow module and run sess = tf.Session(); the output is roughly as follows:\nAt this point the tensorflow CPU version is installed\ntensorflow GPU This will be a painful process with a high error rate; it took me two days to get this GPU version installed.\nCreating the virtual environment conda create -n tensorflow_gpu python=3.6 Before installing the tensorflow GPU version, CUDA and cuDNN must be installed. CUDA is a general parallel computing architecture introduced by NVIDIA, which enables GPUs to solve complex computing problems. It includes the CUDA instruction set architecture (ISA) and the parallel computing engine inside the GPU. cuDNN, on the other hand, is a GPU acceleration library for deep learning neural networks, emphasizing performance, ease of use and low memory overhead. NVIDIA cuDNN can be integrated into higher-level machine learning frameworks.\nOne piece of advice here:\nYou can download the CUDA documentation first, but do not rush to install; read the official guide provided by NVIDIA carefully, then find a few good blog posts to read, so as to have a rough understanding of the CUDA installation process and a general idea of the problems that may arise during installation.\nInstallation prerequisites Verify whether the computer has a CUDA-capable GPU: Enter the command lspci | grep -i nvidia in the terminal and check the output. If there is no output, try update-pciids (this command is usually in the /sbin directory) to update the computer\u0026rsquo;s PCI hardware database, and then run the previous query command again. The final output should look like the figure below; if your GPU is from the Nvidia manufacturer and is in the CUDA list, it means you have a CUDA-capable GPU.\nVerify whether the Linux version supports CUDA: Use the command uname -m \u0026amp;\u0026amp; cat /etc/*release to check (ubuntu16.04 is supported)\nVerify whether gcc is installed on the system: Enter the command gcc --version to check the gcc version; if it is not installed, install it following the terminal prompts.\nVerify whether the system has the matching kernel header and package development files: The command uname -r shows the kernel version, and then you can run sudo apt-get install linux-headers-$(uname -r) to install it; if it is already installed, the terminal will say so and stop the installation.\nOnce all the items above are satisfied, you can start installing CUDA.\nInstalling CUDA CUDA offers two installation methods: package manager installation and runfile installation. I saw on other blogs that most people choose the runfile method; I tried it several times too, but in the end it always failed, and of the two days of installation time, most was spent on installing CUDA. Finally I chose the package manager installation method.\nDownload the CUDA package: First select the CUDA version on the CUDA download page; I chose CUDA 9.0\nThen install following the installation steps indicated on the page\nThere is also a patch package for CUDA9.0 on the page; after installing CUDA9.0 it can be installed with the dpkg -i command\nConfiguring CUDA: After the steps above, the system environment variables still need to be configured: edit the .bashrc file in the home directory with superuser privileges, add the following content to the file, save and exit.\nexport PATH=\u0026#34;/usr/local/cuda-9.0/bin${PATH:+:${PATH}}\u0026#34; export LD_LIBRARY_PATH=\u0026#34;/usr/local/cuda-9.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}\u0026#34; Then reload the file with the command source ~/.bashrc.\nVerifying the CUDA installation Check the CUDA version: Enter nvcc --version in the terminal and check the output; if the output is the corresponding version number, you can go on to the next step, otherwise you should uninstall and reinstall\nUninstall commands:\nsudo /usr/local/cuda-9.0/bin/uninstall_cuda_9.0.pl (by default the CUDA uninstall script is in the /usr/local/cuda-9.0/bin directory);\nsudo /usr/bin/nvidia-uninstall uninstalls the nvidia driver and its configuration.\nCompile the examples provided by CUDA: Generally the examples provided by CUDA are in the /usr/local/cuda-9.0/samples directory; they can be compiled there, or the whole directory can be copied elsewhere for compilation.\nRun the command make under samples/ to compile; on success it displays Finished building CUDA samples\nAfter compiling, enter the directory /samples/bin/x86_64/linux/release and run ./deviceQuery; the result is roughly as follows:\nThen check the connection between the system and the CUDA-Capable device by running ./bandwidthTest; the output is roughly as follows:\nIf all the items above pass, congratulations, CUDA has been installed successfully! Next is installing cuDNN\nInstalling cuDNN First you need to register an account on the Nvidia official site to download the cuDNN package, then choose the corresponding package to download (make sure it matches the installed CUDA version)\nAfter downloading, extract it with the tar command, and then copy the corresponding files into the corresponding CUDA directories\nsudo cp cuda/include/cudnn.h /usr/local/cuda-9.0/include/ sudo cp cuda/lib64/libcudnn* /usr/local/cuda-9.0/lib64/ sudo chmod a+r /usr/local/cuda-9.0/include/cudnn.h sudo chmod a+r /usr/local/cuda-9.0/lib64/libcudnn* Installing tensorflow GPU Enter the virtual environment: source activate tensorflow_gpu\nYou can go to pypi to choose the corresponding version to download and then install it with pip install.\nYou can also install it directly with the corresponding package link\n$ pip install https://pypi.python.org/pypi/tensorflow-gpu/tensorflow_gpu-1.6.0rc0-cp36-cp36m-manylinux1_x86_64.whl Verifying the installation Import the tensorflow module in python inside the virtual environment and run sess = tf.Session(); the output is roughly as follows:\nOK, the long installation of the tensorflow GPU version is over; now you can put those amazing artificial intelligence algorithms into practice!\nReference QLULIBIN — a detailed tutorial on installing CUDA 9.0 on Ubuntu 16.04\nmultiangle — the process of installing CUDA, cuDNN and the tensorflow-gpu version under ubuntu16.04\nSome pitfalls that may be encountered during installation and how to solve them:\nThe thinking fish — some pitfalls and solutions when installing the NVIDIA graphics driver and CUDA on Ubuntu16.04\nycszen — installing the NVIDIA driver on Ubuntu (a solution after consulting an NVIDIA engineer)\nSome further reading:\nma_fighting — how to check CPU information under Ubuntu, including bitness and multi-core information\nZeroZone — an overview of the Compute Capability of NVIDIA GPUs\nsuperPershing — switching between Intel and NVIDIA graphics cards on Ubuntu\n","permalink":"https://simplestory.github.io/en/posts/20180204_68b9e972b5a534bc31d12d9be53b57ec/","summary":"\u003cblockquote\u003e\n\u003cp\u003eBorrowing a line from the \u003ca href=\"http://www.tensorfly.cn/\"\u003eTensorFlow Chinese community\u003c/a\u003e: TensorFlow, an open-source magic tool for artificial intelligence.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003e\u003cstrong\u003eEnvironment used for the configuration\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eSystem: Ubuntu16.04 (LTS)\u003c/li\u003e\n\u003cli\u003epython version: python3.6 (using Anaconda to create a virtual environment)\u003c/li\u003e\n\u003cli\u003eNvidia card: Geforce 920M (if there is no Nvidia card, only the CPU version can be installed)\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch1 id=\"installing-tensorflow\"\u003eInstalling Tensorflow\u003c/h1\u003e\n\u003cp\u003etensorflow has two versions, CPU and GPU; I created two python virtual environments and installed these two versions separately. Below I record this painful installation process\u003c/p\u003e","title":"Configuring Tensorflow on Linux"},{"content":" Below is a penetration testing memo for Linux machines; I came across it on the internet by chance and copied it over here.\nPenetration testing a Linux machine Command Description netstat -tulpn Show network ports in Linux together with the corresponding ID (PID) watch ss -stplu Observe TCP and UDP ports in real time through sockets lsof -i Show established connections macchanger -m MACADDR INTR Change the MAC address on KALI Linux ifconfig eth0 192.168.2.1/24 Set the IP address in Linux ifconfig eth0:192.168.2.3/24 Add an IP address to an existing network interface in Linux ifconfig eth0 hw ether MAC ADDR Change the address in Linux using ifconfig ifconfig eth0 mtu 1500 Change the MTU size with ifconfig; change 1500 to the MTU you want dig -x 192.168.1.1 Perform a reverse lookup on an IP address host 192.168.1.1 Perform a reverse lookup on an IP address; useful when dig is not installed dig @192.168.2.2 domain.com -t AXFR Perform a DNS zone transfer with dig host -l domain.com nameserver Perform a DNS zone transfer with host nbtstat -A x.x.x.x Obtain the domain name corresponding to an IP address ip addr add 192.168.2.22/24 dev eth0 Add a hidden IP address that does not show up when running ifconfig tcpkill -9 host goole.com Block access to goole.com from the host echo \u0026quot;1\u0026quot;\u0026gt;/proc/sys/net/ipv4/ip_forward Enable IP forwarding, turning the Linux box into a router — which makes it convenient to control the traffic routed through this box echo \u0026quot;8.8.8.8\u0026quot;\u0026gt;/etc/resolv.conf Use Goole's DNS System information commands Very helpful for local enumeration\nCommand Description whoami Show the currently logged-in user on Linux id Show the currently logged-in user and group last Show the last logged-in users mount Show mounted drives df -h Show disk usage in human-readable output echo \u0026quot;user:passwd\u0026quot; | chpasswd Reset a password with a single command getent passwd List the users on Linux strings /usr/local/bin/blah Show the content of a non-text file, for example what is inside a binary uname -ar Show the running kernel version PATH=$PATH:/my/new-path Add a new path, making file system (FS) operations more convenient history Show the user's history of previously executed bash scripts, along with the commands typed Redhat/CentOS/RPM based distributions Command Description cat /etc/redhat-release Show the Redhat/CentOS version number rpm -qa List all installed RPM packages on RPM-based Linux rpm -q --changelog openvpn Check whether an installed RPM has been patched against a CVE; the grep command can be used to filter the output related to CVEs YUM commands Command Description yum update Update all RPM packages with YUM; it also shows which ones are outdated yum update httpd Update a single package, in this example HTTPD (Apache) yum install package Install a package with YUM yum --exclude=package kernel* update Exclude a package from being updated when using YUM yum remove package Remove a package with YUM yum erase package Remove a package with YUM yum list package List information about a YUM package yum provides httpd Show what a package provides yum info httpd Show package information, architecture, version and so on yum local install blah.rpm Use YUM to install a local RPM, installing from the repository yum deplist package Show the provider information of a package yum list installed | more List all installed packages yum grouplist | more Show all YUM groups yum groupinstall \u0026#39;Development Tools\u0026#39; Install a YUM group Debian/Ubuntu/.deb based distributions Command Description cat /etc/debian_version Show the Debian version number cat /etc/*-release Show the Ubuntu version number dpkg -l List all installed packages on Debian/.deb based Linux distributions Linux user management Command Description useradd new-user Create a new Linux user passwd username Reset a Linux user's password; if you are the root user you only need to enter the password userdel username Delete a Linux user User creation commands adduser: automatically specifies the home directory and system shell version for the created user, and asks for the user password at creation time useradd: the basic settings above must be specified with parameter options; if no parameters are used, the created user has no password, no home directory and no specified shell version\nUser deletion commands userdel: delete a user; adding the option -r deletes all files related to the user, and -f forces deletion of the user even if the user is logged in deluser: remove a user from a group (that is, leave the user group) deluser username usergroup\nLinux decompression commands Command Description unzip archive.zip Extract the files in a zip archive on Linux zipgrep *.txt archive.zip Search inside a zip archive tar xf archive.tar Extract the files in a tar archive on Linux tar xvzf archive.tar.gz Extract the files in a tar.gz archive on Linux tar xjf archive.tar.bz2 Extract the files in a tar.bz2 archive on Linux tar ztvf file.tar.gz | grep blah Search inside a tar.gz file gzip -d archive.gz Extract the file in a gzip archive on Linux zcat archive.gz Read a .gz file on Linux without decompressing it zless archive.gz Achieve the same functionality for a .gz archive with the less command zgrep \u0026#39;blah\u0026#39; /var/log/maillog*.gz Search the contents of .gz archives on Linux, for example searching compressed log files vim file.txt.gz Read a .txt.gz file with vim Linux compression commands Command Description zip -r file.zip /dir/* Create a .zip file on Linux tar cf archive.tar files Create a .tar file on Linux tar czf archive.tar.bz2 files Create a tar.gz file on Linux tar cjf archive.tar.bz2 files Create a tar.bz2 file on Linux gzip file Create a .gz file on Linux upx -9 -o output.exe input.exe Compress an .exe file with UPX on Linux Linux file commands Command Description df -h blah Show the size of a file/directory on Linux diff file1 file2 Compare/show the difference between two files on Linux md5sum file Generate an MD5 digest on Linux md5sum -c blah.iso.md5 Check a file's MD5 digest on Linux, assuming the file and the .md5 are in the same path file blah Find out the type of a file on Linux; it also shows whether the file is 32 or 64 bit dos2unix Convert Windows line endings to Unix/Linux ones base64 input-file output-file Base64-encode the input file and output a Base64-encoded file named output-file base64 -d input-file output-file Base64-decode the input file and output a Base64-decoded file named output-file touch -r ref-file new-file Create a new file using the timestamp data from the reference file; drop -r to simply create a file rm -rf Delete files and directories without showing a confirmation prompt Misc commands Command Description init 6 Restart Linux from the command line gcc -o output.c input.c Compile C code gcc -m32 -o output.c input.c Cross-compile C code; on 64-bit Linux this compiles a 32-bit binary unset HISTORYFILE Turn off bash history logging rdesktop x.x.x.x Connect to an RDP server from Linux kill -9 $$ Kill the current session chown user:group blah Change the owner of a file or directory chown -R user:group blah Change the owner of a file or directory as well as the files/directories under it — recursively execute chown chmod 600 file Change the permission settings of a file/directory; see Linux file system permissions for details Clearing bash history $ ssh user@x.x.x.x | cat /dev/null \u0026gt; ~/.bash_history The .bash_history file can hold many history entries; the number of saved commands can be reduced by setting the HISTFILESIZE and HISTSIZE parameters.\nThe meaning of the two parameters:\nHISTSIZE: defines the number of commands output by the history command HISTFILESIZE: defines the number of commands saved in the .bash_history file $ HISTFILESIZE=\u0026lt;num\u0026gt; $ HISTSIZE=\u0026lt;num\u0026gt; The command history -c clears the commands output this time but does not clear the .bash_history file, so after logging in again the old commands will still appear.\nLinux file permissions Value Meaning 777 rwxrwxrwx no restriction, fully readable, writable and executable (RWX); the user can do anything 755 rwxr-xr-x the owner has full access, others can only read and execute 700 rwx------ the owner has full access, no one else can access it 666 rw-rw-rw- everyone can read and write, but not execute 644 rw-r--r-- the owner can read and write, others can only read 600 rw------- the owner can read and write, no one else can access it Penetration testing memo for the Linux file system Path Description / / is also called \"slash\" or root /bin General programs shared by the system, the system administrator and users /boot Boot files, boot loader (grub), kernel, vmlinuz /dev Contains references to system devices, files with special attributes /etc Important system configuration files /home Home directories of system users /lib Library files, including files for all kinds of programs needed by both the system and users /lost+found Files from failed file operations are saved here /mnt Standard mount point for external file systems /media Mount point for external file systems (or for some distributions) /net Standard mount point for entire remote file systems — nfs /opt Generally contains some additional or third-party software /proc A virtual file system containing information related to system resources /root Home directory of the root user /sbin Programs used by the system and the system administrator /tmp Temporary space for the system's use, cleared on reboot /usr Programs, libraries, documentation and so on used by all user-related programs /var Stores all variable files and transient files created by users, such as log files, mail queues, print spoolers, web servers, databases and so on Some interesting files/directories in Linux Very helpful for privilege escalation / post-exploitation\nPath Description /etc/passwd Contains the local Linux users /etc/shadow Contains the hashed local account passwords /etc/group Contains the local account groups /etc/init.d/ Contains the service initialization scripts — worth a look to see exactly what is installed /etc/hostname The system's hostname /etc/network/interfaces Network interfaces /etc/resolv.conf The system's DNS service /etc/profile The system's environment variables ~/.ssh/ SSH keys ~/.bash_history The user's bash history log /var/log/ Linux system log files are generally stored here /var/adm/ UNIX system log files are generally stored here /var/log/apache2/access.log /var/log/httpd/access.log The usual storage paths of the Apache access log files /etc/fstab Mounted file systems Breaking out of shell restrictions Python small tricks:\npython -c \u0026#39;import pty;pty.spawn(\u0026#34;/bin/bash\u0026#34;)\u0026#39; echo os.system(\u0026#34;/bin/bash\u0026#34;) /bin/sh -i Samba commands Connecting to a Samba share from Linux:\n$ smbmount //server/share /mnt/win -o user=username,password=password1 $ smbclient -U user \\\\\\\\server\\\\share $ mount -t cifs -o username=user,password=password //x.x.x.x/share /mnt/share Reference Linux enthusiasts\n","permalink":"https://simplestory.github.io/en/posts/20180120_3e2c28e497c306d34f88081093e32c89/","summary":"\u003cblockquote\u003e\n\u003cp\u003eBelow is a penetration testing memo for Linux machines; I came across it on the internet by chance and copied it over here.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"penetration-testing-a-linux-machine\"\u003ePenetration testing a Linux machine\u003c/h2\u003e\n\u003ctable\u003e\n\u003cthead\u003e\n\u003ctr\u003e\n\u003cth style=\"text-align:left\"\u003eCommand\u003c/th\u003e\n\u003cth style=\"text-align:left\"\u003eDescription\u003c/th\u003e\n\u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003enetstat -tulpn\u003c/td\u003e\n\u003ctd style=\"text-align:left\"\u003eShow network ports in Linux together with the corresponding ID (PID)\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003ewatch ss -stplu\u003c/td\u003e\n\u003ctd style=\"text-align:left\"\u003eObserve TCP and UDP ports in real time through sockets\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003elsof -i\u003c/td\u003e\n\u003ctd style=\"text-align:left\"\u003eShow established connections\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003emacchanger -m MACADDR INTR\u003c/td\u003e\n\u003ctd style=\"text-align:left\"\u003eChange the MAC address on KALI Linux\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003eifconfig eth0 192.168.2.1/24\u003c/td\u003e\n\u003ctd style=\"text-align:left\"\u003eSet the IP address in Linux\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003eifconfig eth0:192.168.2.3/24\u003c/td\u003e\n\u003ctd style=\"text-align:left\"\u003eAdd an IP address to an existing network interface in Linux\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003eifconfig eth0 hw ether MAC ADDR\u003c/td\u003e\n\u003ctd style=\"text-align:left\"\u003eChange the address in Linux using ifconfig\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003eifconfig eth0 mtu 1500\u003c/td\u003e\n\u003ctd style=\"text-align:left\"\u003eChange the MTU size with ifconfig; change 1500 to the MTU you want\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003edig -x 192.168.1.1\u003c/td\u003e\n\u003ctd style=\"text-align:left\"\u003ePerform a reverse lookup on an IP address\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003ehost 192.168.1.1\u003c/td\u003e\n\u003ctd style=\"text-align:left\"\u003ePerform a reverse lookup on an IP address; useful when dig is not installed\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003edig @192.168.2.2 domain.com -t AXFR\u003c/td\u003e\n\u003ctd style=\"text-align:left\"\u003ePerform a DNS zone transfer with dig\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003ehost -l domain.com nameserver\u003c/td\u003e\n\u003ctd style=\"text-align:left\"\u003ePerform a DNS zone transfer with host\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003enbtstat -A x.x.x.x\u003c/td\u003e\n\u003ctd style=\"text-align:left\"\u003eObtain the domain name corresponding to an IP address\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003eip addr add 192.168.2.22/24 dev eth0\u003c/td\u003e\n\u003ctd style=\"text-align:left\"\u003eAdd a hidden IP address that does not show up when running ifconfig\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003etcpkill -9 host goole.com\u003c/td\u003e\n\u003ctd style=\"text-align:left\"\u003eBlock access to goole.com from the host\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003eecho \u0026quot;1\u0026quot;\u0026gt;/proc/sys/net/ipv4/ip_forward\u003c/td\u003e\n\u003ctd style=\"text-align:left\"\u003eEnable IP forwarding, turning the Linux box into a router — which makes it convenient to control the traffic routed through this box\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"text-align:left\"\u003eecho \u0026quot;8.8.8.8\u0026quot;\u0026gt;/etc/resolv.conf\u003c/td\u003e\n\u003ctd style=\"text-align:left\"\u003eUse Goole's DNS\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\u003ch2 id=\"system-information-commands\"\u003eSystem information commands\u003c/h2\u003e\n\u003cp\u003eVery helpful for local enumeration\u003c/p\u003e","title":"Linux Penetration Testing Commands"},{"content":"Server version: 5.7.20-0ubuntu0.16.04.1 (Ubuntu)\nAfter logging into MySQL, you can use the following command to query where the related files are stored:\nshow variables like \u0026#39;%dir%\u0026#39;; The output is shown below:\nUsually these are:\nThe default location of the database files is: /usr/share/mysql/ Configuration file: /etc/mysql/my.cnf\nDatabase directory: /var/lib/mysql Configuration directory: /etc/share/mysql (mysql.server command and configuration files) MySQL related commands: /usr/bin Startup scripts: /etc/init.d/\nOne way to start mysql safely: /usr/bin/mysqld_safe-user=root\u0026amp;\n","permalink":"https://simplestory.github.io/en/posts/20171201_7eec743fae4833726b0d923cf6db1da6/","summary":"\u003cp\u003e\u003cstrong\u003eServer version: 5.7.20-0ubuntu0.16.04.1 (Ubuntu)\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAfter logging into MySQL, you can use the following command to query where the related files are stored:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-sql\" data-lang=\"sql\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eshow\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003evariables\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"k\"\u003elike\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;%dir%\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe output is shown below:\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://simplestory-blog-img.oss-cn-guangzhou.aliyuncs.com/in_posts/20171201/mysql.png\" title=\"mysql\"\u003e\u003c/p\u003e\n\u003cp\u003eUsually these are:\u003c/p\u003e\n\u003cp\u003eThe default location of the database files is: \u003ccode\u003e/usr/share/mysql/\u003c/code\u003e\nConfiguration file: \u003ccode\u003e/etc/mysql/my.cnf\u003c/code\u003e\u003c/p\u003e\n\u003cp\u003eDatabase directory: \u003ccode\u003e/var/lib/mysql\u003c/code\u003e\nConfiguration directory: \u003ccode\u003e/etc/share/mysql\u003c/code\u003e (mysql.server command and configuration files)\nMySQL related commands: \u003ccode\u003e/usr/bin\u003c/code\u003e\nStartup scripts: \u003ccode\u003e/etc/init.d/\u003c/code\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eOne way to start mysql safely: /usr/bin/mysqld_safe-user=root\u0026amp;\u003c/strong\u003e\u003c/p\u003e","title":"MySQL Data File Paths on Linux"},{"content":" Using the command line under Linux is routine; usually we memorize forty or fifty commands, but some rarely used yet useful commands are overlooked by most people. This post collects a small subset of them.\nsudo !! Some commands report an error when run without superuser privileges, after which sudo !! can be used to invoke the previous command. !! grabs the last command.\npython command With a simple python command we can produce a simple web page that displays the folder structure over HTTP, accessible from a browser on port 8000 until an interrupt signal is sent.\n$ python -m SimpleHTTPServer mtr The mtr command combines the functionality of the ping and traceroute commands; mtr may need to be installed with apt/yum before use\n$ sudo apt install mtr #Debian based systems $ yum install mtr #Red Hat based systems Ctrl+x+e To automate daily tasks, administrators need to open an editor by typing vi, vim, nano and so on. Simply pressing Ctrl-x-e quickly from the command line lets you start working in the editor.\nnl The nl command adds line numbers to a file, that is, it adds line numbers to the specified file.\nshuf This command randomly selects lines/files/folders from a file or folder. It can be appended to the ls command to pick things at random.\n$ ls|shuf -n1 #randomly select one $ ls|shuf -n2 #randomly select two. And so on ss ss stands for socket statistics. This command investigates sockets and shows information similar to the netstat command. It can show more TCP and state information than other tools.\nlast This command shows historical information about the last logged-in users (by searching the file \u0026ldquo;/var/log/wtmp\u0026rdquo;, it displays the list of logged-in and logged-out users along with their tty\u0026rsquo;s).\ncurl ifconfig.me This command is used to obtain your external IP address; using curl may require installing the relevant packages.\ntree Displays the current folder in a tree structure.\npstree Displays all processes of the currently running commands and the related child processes, with the output shown in a tree structure.\nEvery command typed in the terminal is recorded in history, which can be viewed with the history command. When typing a command in the terminal you can put a space in front of the command, so that your command is not recorded in history.\nstat The stat command in Linux is used to display status information about a file or file system. When a file name is used as the argument, the command returns all information about the file, including file size, blocks, permissions, access time, modification time, status change time and so on.\npv This command simulates the text output of any pipeline, including piped output. The pv package may need to be installed first before use.\nmount | column -t Displays all mounted file systems in a neat format.\nCtrl + l This is more of a shortcut key, and has the same effect as clear.\nat This command is similar to the cron command and can also be used to schedule a task or execute a command at a specific moment.\n$ echo \u0026#34;ls -l \u0026gt; /dev/pts/0\u0026#34; | at 14:012 $ echo \u0026#34;ls -l \u0026gt; /dev/pts/0\u0026#34; | at 2:12 PM du -h -max-depth=1 This command outputs the sizes of the subdirectories of the current directory in human-readable form. This is very useful when checking system usage.\nexpr expr performs simple arithmetic in the terminal.\nlook Look up a word from the English dictionary in the terminal to avoid confusion.\n$ look car #show all words in the dictionary starting with \u0026#39;car\u0026#39; yes This command keeps outputting the given string until an interrupt command is received. This is very useful in scripting and system administration.\nfactor This command outputs all the factors of the given number.\nping -i n -a IP_address When waiting or keeping your eyes on the terminal waiting for a command\u0026rsquo;s response or for a server connection: a sound is emitted when the server is reachable (make sure your system is not muted). The parameter n is the waiting time of the command.\n#this command PINGs once every 60 seconds $ ping -i 60 -a www.google.com tac This command outputs the content of a text file in reverse order, that is, from the last line to the first line.\nstrace strace is a debugging tool mainly used for troubleshooting on Linux. It may not be installed by default on your system, and you may need to install the required packages with apt or yum.\n$ strace pwd #use the strace command to trace the execution of a command For the concrete command parameters of strace, see the man page.\ndisown -a \u0026amp;\u0026amp; exit disown can keep a task running in the background even after the terminal session is closed.\n$ \u0026lt;command\u0026gt; disown -a \u0026amp;\u0026amp; exit To detach an already long-running task from the terminal again (translator\u0026rsquo;s note: it may already be running in the background, but not detached from the terminal), use the jobs command to find the job number and then use disown %n, where %n is the job number. To verify that the job is indeed running, use the ps or top command. The nohup command is also an alternative to the disown command.\ngetconf LONG_BIT Shows whether your machine architecture is 32bit or 64bit.\nDisplaying the date This can be considered a script: enter the following command at the terminal prompt and you will see the date and time in the top right corner of the terminal.\n$ while sleep 1;do tput sc;tput cup 0 $(($(tput cols)-29));date;tput rc;done \u0026amp; watch -t -n1 \u0026ldquo;date + %T | figlet\u0026rdquo; Before using it, make sure the figlet package is installed on the system. This command displays an animated digital clock in the terminal by piping the output to figlet.\nhost and dig host is a DNS lookup tool, and the dig command is used to resolve domain names, giving more domain name information than nslookup.\ndstat dstat is a versatile tool that generates statistics based on system resources. dstat may not be installed by default on your system. Use apt or yum to install it before using this colorful generator of system information.\nbind -p This command shows all the shortcuts available in the BASH shell.\nlsb_release The lsb_release command prints information specific to the distribution. If lsb_release is not yet installed, you can install the package with apt on Debian-based distributions and with yum on Red Hat based systems.\nNote: the option \u0026lsquo;-a\u0026rsquo; shows all available information about the version, ID, description, release number and codename.\nnc -zv localhost The parameter num is any port number; this command checks whether the port is open or closed.\ncurl ipinfo.io This command outputs and provides the geographic location of the IP address.\nfind . -user root This command outputs the files in the current directory owned by the (root) user (translator\u0026rsquo;s note: that is, with owner root). Below are all the files owned by the \u0026lsquo;root\u0026rsquo; user listed in the current directory.\nsudo apt-get build-dep ffmpeg This command automatically builds the dependencies when the corresponding package is installed. As a result, the package installation process will be very smooth and very easy.\nlsof -iTCP: -sTCP:LISTEN This command outputs the names of the processes/services that are using port .\npdftk pdftk is used to merge several pdf files; make sure the pdftk package is installed on the system before use.\nps -LF -u user_name This command outputs the processes and threads of a user. The option \u0026ldquo;L\u0026rdquo; (list threads), the option \u0026ldquo;-F\u0026rdquo; (full formatting)\nstartx - :1 Share an X session. If you frequently need to log in and out of the desktop (as different users), then startx comes to the rescue. This command creates a new session, thus avoiding repeated logging in and out within one session. To switch between X sessions we can use the key combinations ctrl+Alt+F7 and ctrl+Alt+F8.\nNote: the shortcuts ctrl+Alt+F1~F6 are for console sessions, while ctrl+Alt+F7~F12 serve X sessions. So we have 6 console sessions and 6 X sessions, and there is no need to log in and out frequently. The ordering above applies to most distributions, though different distributions may implement it differently.\nReference Linux.China - open source community\n","permalink":"https://simplestory.github.io/en/posts/20171107_619e2d5ba49593e40e16667bafdcf2c1/","summary":"\u003cblockquote\u003e\n\u003cp\u003eUsing the command line under Linux is routine; usually we memorize forty or fifty commands, but some rarely used yet useful commands are overlooked by most people. This post collects a small subset of them.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch3 id=\"sudo-\"\u003esudo !!\u003c/h3\u003e\n\u003cp\u003eSome commands report an error when run without superuser privileges, after which \u003ccode\u003esudo !!\u003c/code\u003e can be used to invoke the previous command. \u003ccode\u003e!!\u003c/code\u003e grabs the last command.\u003c/p\u003e\n\u003ch3 id=\"python-command\"\u003epython command\u003c/h3\u003e\n\u003cp\u003eWith a simple python command we can produce a simple web page that displays the folder structure over HTTP, accessible from a browser on port 8000 until an interrupt signal is sent.\u003c/p\u003e","title":"Some Linux Commands"},{"content":" Opencv contains more than 2500 functions, and almost any mature algorithm you can think of can be implemented by calling Opencv\u0026rsquo;s functions, which is super convenient. This post combines Python and Opencv to detect cars in surveillance video and check whether they commit a violation (crossing the double yellow line).\nSystem environment: Ubuntu16.04LTS+OpenCV3.3.0+Python2.7.12\nBackground extraction For surveillance the position is relatively fixed, so the background will not change considerably over a long period of time. We can therefore consider using background subtraction to roughly extract the background.\nHere I use the KNN algorithm built into Opencv; this algorithm uses the first small portion of the surveillance video as training frames to train a model and obtain an approximate background image.\nOf course OpenCV already implements several very easy-to-use algorithms. For the specific differences see: background subtraction\n# coding = utf-8 import numpy as np import cv2 cap = cv2.VideoCapture(\u0026#39;filename\u0026#39;) #specify the number of training frames as 30 trainframe = 30 #KNN algorithm background subtraction bgsubstractor = cv2.creatBackgroundSubtractorKNN(detectShadow = True) Preprocessing The video frames are median filtered according to the video resolution. For different video resolutions, different kernel parameters are used. For example: for a video resolution of 240X320, the kernel parameter chosen is 5X5.\nAfter median filtering the image is converted to grayscale for Gaussian blurring, and a suitable kernel parameter can be found experimentally. This is followed by edge detection, where the lowest and highest thresholds are also found through repeated experiments. Finally morphological processing is performed, such as erosion and dilation; the default kernel is used here with 2 iterations.\n#median filter the original frame gray = fg_mask.copy() gray = cv2.medianBlur(gray, 11, 0) #image preprocessing thresh = cv2.threshold(gray, 244, 255, cv2.THRESH_BINARY)[1] thresh = cv2.erode(thresh, None, iterations = 2) thresh = cv2.dilate(thresh, None, iterations = 2) Moving object detection Use the findContours() function in Opencv to obtain the detection boxes in the corresponding video frames.\nNote: a copy of the video frame should be used\nThe contour retrieval mode chosen is cv2.RETR_EXTERNAL, which returns only the outermost contours. The contour approximation method is cv2.CHAIN_APPROX_SIMPLE, which returns the minimum number of key points of the detected box.\n#obtain the detection boxes _, conts, _ = cv2.findContours(thresh.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) Filtering moving objects This part could use the cascade classifier in Opencv for recognition, but due to the lack of a sufficient sample set and time, a method based on the contour area is used here to distinguish objects from non-objects.\nFor details see: object detection with the Opencv cascade classifier\n#object extraction #obtain the approximate list of vehicles carlist carslist = [] cars = [] cars = sorted(conts, contoursSort) for car in cars: area = cv2.contourArea(car) if area \u0026gt; cv2.contourArea(cars[0])/2 and area \u0026gt; 400: carslist.append(car) In the code snippet above there is a function contoursSort(), which is used for sorting.\n#contour area sorting rule function def contoursSort(c1, c2): return cv2.contourArea(c1) \u0026gt; cv2.contourArea(c2) Marking the double yellow line The Hough transform can be used to obtain the position of the fairly close double yellow lines.\n#Hough transform rho = 2 theta = np.pi/360 threshold = 15 min_line_length = 40 max_line_gap = 20 line_frame = np.copy(frame) * 0 lines = cv2.HoughLinesP(masked_edges, rho, theta, threshold, np.array([]), min_line_length, max_line_gap) #draw the lines for x1,y1,x2,y2 in lines[0]: cv2.line(frame, (x1,y1), (x2,y2), (0,255,0), 2) Collision handling First obtain the coordinate points of the current frame box: use the boundingRect function to get the coordinates $(x,y)$ of the top-left corner of the frame box and its width and height $(w,h)$.\nRecord the position of the double yellow line as: the $x$ coordinate of the left yellow line $x_b$, and the $x$ coordinate of the right yellow line $x_c$\nCollision cases: Moving forward and crossing the double yellow line: $x_b \u003c x \u003c x_c$ or $x_b \u003c x+w \u003c x_c$;\nStationary or straddling the double yellow line: $x \u003c x_b$ and $x+w \u003e x_c$\nAfter a collision is detected, draw a box around the violating vehicle. Compute the centroid coordinates of the contour in the $x$ and $y$ directions separately, and based on these draw a rectangle with a suitable distance.\n#mark the violating vehicles (red box) #position of the double yellow line x_b = 155 x_c = 160 for car in carslist: x, y, w, h = cv2.boundingRect(car) #detection value for crossing the double yellow line while moving k1 = (x_b \u0026lt; x \u0026lt; x_c) or (x_b \u0026lt; (x + x+w)/2 \u0026lt; x_c) #detection value for being stationary or straddling the double yellow line k2 = x \u0026lt; x_b and (x + x+w)/2 \u0026gt; x_c if k1 or k2: M = cv2.moments(car) #compute the various properties of the contour cx = int(M[\u0026#39;m10\u0026#39;]/M[\u0026#39;m00\u0026#39;]) #centroid in the x direction cy = int(M[\u0026#39;m01\u0026#39;]/M[\u0026#39;m00\u0026#39;]) #centroid in the y direction cv2.rectangle(frame, (cx-10, cy-10), (cx+20, cy+20), (0, 0, 255), 2) Saving the video Usually we want to save the processed video, which can be done by constructing a VideoWriter class in advance.\nprename = raw_input(\u0026#39;File name (.avi): \u0026#39;) name = prename + \u0026#39;.avi\u0026#39; #obtain the frame rate of the original video fps = int(cap.get(cv2.CAP_PROP_FPS)) #obtain the size of the original video size = (int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)), int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))) #set the video format fourcc = cv2.VideoWriter_fourcc(*\u0026#39;MJPG\u0026#39;) out = cv2.VideoWriter(name, fourcc, fps, size) Summary The above is my implementation process for detecting traffic violations in surveillance video. This process was very hard going, but thanks to my persistence the whole idea gradually became clear and my inner restlessness subsided quite a bit. From being completely lost about using Python-Opencv at the beginning to writing this beginner-level article now, I have learned a lot — not to mention all those morphological transformations, edge detection and other image and video processing algorithms, my Baidu/Google skills have also advanced by leaps and bounds. Judging from the detection results, this program is not entirely satisfactory and there is plenty of room for improvement, for example adding a shadow removal algorithm.\nReference Thanks to the almighty Baidu/Google.\nBelow are some references:\nOpenCV official tutorial (Chinese version)\nQin Xiuli. Shadow removal algorithm based on YUV color space and graph cuts [D]. Wuhan University of Technology. 2010\nWang Zhiyong. Design of a traffic violation monitoring system based on video sequences [D]. Zhengzhou University. 2015\nLuo Yurong. Research and implementation of a seat belt recognition system [D]. Beijing University of Technology. 2008\nWu Peimin. Research on video-based face detection and recognition [D]. Nanchang University. 2011\nFinally, a few blogs/websites:\nGoogle\u0026rsquo;s chief programmer\nJobbole\nthefutureisour\nOpenCV Tutorials\n","permalink":"https://simplestory.github.io/en/posts/20171104_bb6e9cb8623b3bf044c62bbbca472d53/","summary":"\u003cblockquote\u003e\n\u003cp\u003eOpencv contains more than 2500 functions, and almost any mature algorithm you can think of can be implemented by calling Opencv\u0026rsquo;s functions, which is super convenient. This post combines Python and Opencv to detect cars in surveillance video and check whether they commit a violation (crossing the double yellow line).\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003e\u003cstrong\u003eSystem environment: Ubuntu16.04LTS+OpenCV3.3.0+Python2.7.12\u003c/strong\u003e\u003c/p\u003e\n\u003ch2 id=\"background-extraction\"\u003eBackground extraction\u003c/h2\u003e\n\u003cp\u003eFor surveillance the position is relatively fixed, so the background will not change considerably over a long period of time. We can therefore consider using background subtraction to roughly extract the background.\u003c/p\u003e","title":"Detecting Traffic Violations with Opencv"},{"content":" Recently, because a project needed computer vision recognition, I started tinkering with OpenCV. Although the installation process was long and winding, the experience of using it together with Python is actually quite good. Here I record the OpenCV installation and compilation process.\nSystem environment: Ubuntu16.04\nSoftware installed: OpenCV3.3.0+Python2.7.12\nUpdating the system packages: $ sudo apt update $ sudo apt upgrade Installing some developer tools: $ sudo apt-get install build-essential cmake pkg-config The pkg-config package is very likely already installed by default, but just in case, add it in as well. The cmake program is used later to compile OpenCV.\n$ sudo apt-get install libjpeg8-dev libtiff5-dev $ sudo apt-get install libjasper-dev libpng12-dev The packages above are used by OpenCV to process JPEG, PNG, TIFF and other images. For video, the packages installed below can be used to process it frame by frame.\n$ sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev $ sudo apt-get install libv4l-dev libxvidcore-dev libx264-dev The highgui module that handles OpenCV GUI operations depends on the GTK library, so the GTK library also has to be installed\n$ sudo apt-get install libgtk-3-dev Install the optimization module\n$ sudo apt-get install libatlas-base-dev gfortran Install the Python development related files and libraries\n$ sudo apt-get install python2.7-dev python3.5-dev Downloading the OpenCV source code: $ cd ~ $ wget -O opencv.zip https://github.com/Itseez/openCV/archive/3.3.0.zip $ unzip opencv.zip You can check the new version released by OpenCV officially on Github and replace the version number before .zip in the wget command\nLink: OpenCV Github\nThe opencv_contrib library also needs to be downloaded\n$ wget -O opencv_contrib.zip https://github.com/Itseez/opencv_contrib/archive/3.3.0.zip $ unzip opencv_contrib.zip The version number can likewise be replaced with a new one\nLink: opencv_contrib\nNote that when replacing with a new version, keep the version numbers of your opencv and opencv_contrib consistent\nSetting up the Python environment: First make sure you have installed the pip tool\nTo keep the system environment clean, I decided to install virtualenv and virtuanenvwrapper. These tools allow you to work in a virtual environment independent of the system.\nInstall the related tools:\n$ sudo pip install virtualenv virtualenvwrappper $ sudo rm -rf ~/.cache/pip Then we need to update the .bashrc in the user\u0026rsquo;s home directory: open that file and add the following content at the end\n# virtualenv and virtualenvwrapper export $WORKON_HOME=$HOME/.virtualenvs source /usr/local/bin/virtualenvwrapper.sh Finally, when .bashrc is executed again, an error may be reported:\n/usr/bin/python: No module named virtualenvwrapper virtualenvwrapper.sh: There was a problem running the initialization hooks. If Python could not import the module virtualenvwrapper.hook_loader, check that virtualenvwrapper has been installed for VIRTUALENVWRAPPER_PYTHON=/usr/bin/python and that PATH is set properly. Because ubuntu installs two versions of Python, and virtualenvwrapper was installed with pip3 while the system runs Python2 by default, some modules are missing in Python2. Part of the virtualenvwrapper.sh file is as follows:\n# Locate the global Python where virtualenvwrapper is installed. if [ \u0026#34;$VIRTUALENVWRAPPER_PYTHON\u0026#34; = \u0026#34;\u0026#34; ] then VIRTUALENVWRAPPER_PYTHON=\u0026#34;$(command \\which python)\u0026#34; fi When the VIRTUALENVWRAPPER_PYTHON environment variable does not exist, which python is used by default so the environment variable also needs to be added:\nVIRTUALENVWRAPPER_PYTHON=/usr/bin/python3 Choose the Python version to depend on (Python2 or Python3) to create the virtual environment\n$ mkvirtualenv opencvwork -p python2 Here I choose Python2 and create a virtual environment named opencvwork; the related directory is under ~/.virtualenv\nThen enter the virtual environment\n$ workon opencvwork Afterwards the terminal prompt will start with a related marker\nInstall Numpy (inside the virtual environment)\n$ pip install numpy Configuring and compiling OpenCV: Inside the virtual environment\n$ cd ~/opencv-3.3.0/ $ mkdir build $ cd build $ cmake -D CMAKE_BUILD_TYPE=RELEASE \\ -D CMAKE_INSTALL_PREFIX=/usr/local \\ -D INSTALL_PYTHON_EXAMPLES=ON \\ -D INSTALL_C_EXAMPLES=OFF \\ -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib-3.3.0/modules \\ -D PYTHON_EXECUTABLE=~/.virtualenvs/cv/bin/python \\ -D BUILD_EXAMPLES=ON .. If the compilation reports an error: stdlib.h: No such file or directory\nthen you need to add -D ENABLE_PRECOMILED_HEADERS=OFF to the Cmake options\nIn the interface returned after installation, the Python section should list the Interpreter, Libraries, numpy and packages path items (corresponding to the Python version you chose)\nOnce you have confirmed that CMake ran without errors, you can compile OpenCV:\n$ make -j4 The number 4 can be replaced with the number of CPU cores on your computer\n$ make clean $ make After it runs successfully, OpenCV can be installed on Ubuntu:\n$ sudo make install $ sudo ldconfig Finishing the OpenCV installation: After all the commands above have executed successfully, the Python+OpenCV related files will be saved in /usr/local/lib/python-2.7/site-packages/ (corresponding to the Python version you chose earlier)\nThen create a symbolic link:\n$ cd ~/.virtualenv/opencvwork/lib/python2.7/site-packages/ $ ln -s /usr/local/lib/python2.7/site-packages/cv2.so cv2.so Sometimes cv2.so may be under /usr/local/lib/python2.7/dist-packages\nA simple check of the OpenCV installation: $ cd ~ $ workon opencvwork $ python Python 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609] on linux2 Type \u0026#34;help\u0026#34;, \u0026#34;copyright\u0026#34;, \u0026#34;credits\u0026#34; or \u0026#34;license\u0026#34; for more information. \u0026gt;\u0026gt;\u0026gt; import cv2 \u0026gt;\u0026gt;\u0026gt; cv2.__version__ \u0026#39;3.3.0\u0026#39; \u0026gt;\u0026gt;\u0026gt; Installation successful!\nAt the end of the post, the official OpenCV3.3.0 tutorial is attached: OpenCV3.3.0\nReference: Adrian Rosebrock\nXiao Xiao Gong Cheng Shi\nHello World\n","permalink":"https://simplestory.github.io/en/posts/20171020_df3860e05b3ca04bdc2f774c5c1914fd/","summary":"\u003cblockquote\u003e\n\u003cp\u003eRecently, because a project needed computer vision recognition, I started tinkering with OpenCV. Although the installation process was long and winding, the experience of using it together with Python is actually quite good. Here I record the OpenCV installation and compilation process.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003e\u003cstrong\u003eSystem environment: Ubuntu16.04\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSoftware installed: OpenCV3.3.0+Python2.7.12\u003c/strong\u003e\u003c/p\u003e\n\u003ch2 id=\"updating-the-system-packages\"\u003eUpdating the system packages:\u003c/h2\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-shell\" data-lang=\"shell\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e$ sudo apt update\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e$ sudo apt upgrade\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch2 id=\"installing-some-developer-tools\"\u003eInstalling some developer tools:\u003c/h2\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-shell\" data-lang=\"shell\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e$ sudo apt-get install build-essential cmake pkg-config\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe pkg-config package is very likely already installed by default, but just in case, add it in as well. The cmake program is used later to compile OpenCV.\u003c/p\u003e","title":"Installing OpenCV on Linux"},{"content":" Visual Studio Code is a lightweight editor released by Microsoft; by installing various plugins it can be turned into a powerful multi-language, cross-platform compiler. Recently, because of practical needs, I had to debug some C/C++ code, so I started tinkering with Vscode. Below is a record of the problems that troubled me during the process.\nThe system used in this post is ubuntu16.04LTS\nI will not go into the details of installing Vscode on linux, as Baidu has plenty of resources on that.\nInstalling the C/C++ plugin Use ctrl+shift+P to bring up the command line, search for C++, and find the plugin officially released by Microsoft, as shown:\nEditing C/C++ source files Open a directory with Vscode, because compiling C/C++ generates some configuration files that are specific to the files in the directory.\nNow let\u0026rsquo;s write a simple cpp file and try it out:\n// hello.cpp # include\u0026lt;iostream\u0026gt; using namespace std; int main() { cout \u0026lt;\u0026lt; \u0026#34;Hello World!\u0026#34; \u0026lt;\u0026lt; endl; return 0; } Compiling C/C++ source files Save the cpp source file you have written; if you compile and run it directly, Vscode will report an error, because the relevant files have not been configured yet.\nConfiguring the launch.json file According to the error message, choose C/C++(GDB) to open the relevant launch.json file and change the code of the \u0026quot;program\u0026quot; section to: \u0026quot;${workspaceRoot}/${fileBaseNoExtension}.out\u0026quot; Some people may also need to change the \u0026quot;cwd\u0026quot; item to: \u0026quot;${workspaceRoot}\u0026quot; After the change it looks like this:\nConfiguring the tasks.json file Even after configuring launch.json, compilation still will not run smoothly; the tasks.json file also needs to be configured. Use ctrl+shift+P to open the command line, type tasks runner, select configure tasks and press enter to open it Modify \u0026quot;taskName\u0026quot; and \u0026quot;command\u0026quot; under the \u0026quot;task\u0026quot; item and add an \u0026quot;args\u0026quot; item; the result is shown:\nOnce both files have been configured and saved, you can compile and run. Click debug and you will see the terminal flash by; under windows, so that we can clearly see the output in the terminal, we would add a statement to the cpp file: system(pasue), but in linux systems pasue is no longer a command, so it does not work. getchar() can be used instead to achieve the same effect.\n# include\u0026lt;iostream\u0026gt; # include\u0026lt;stdio.h\u0026gt; using namespace std; int main() { cout \u0026lt;\u0026lt; \u0026#34;Hello World!\u0026#34; \u0026lt;\u0026lt; ehdl; getchar(); return 0; } A screenshot of the result is attached:\n","permalink":"https://simplestory.github.io/en/posts/20171001_f9fd02f8e181115a55cec2936857675c/","summary":"\u003cblockquote\u003e\n\u003cp\u003eVisual Studio Code is a lightweight editor released by Microsoft; by installing various plugins it can be turned into a powerful multi-language, cross-platform compiler. Recently, because of practical needs, I had to debug some C/C++ code, so I started tinkering with Vscode. Below is a record of the problems that troubled me during the process.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003e\u003cstrong\u003eThe system used in this post is ubuntu16.04LTS\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eI will not go into the details of installing Vscode on linux, as Baidu has plenty of resources on that.\u003c/p\u003e","title":"Compiling C/C++ under Vscode"},{"content":" \u0026ldquo;Class\u0026rdquo; is a template, and the template packages many \u0026ldquo;functions\u0026rdquo; for use. This post introduces the members of Python classes, member modifiers, and the special members of classes.\nClass members The members of a class can roughly be divided into three categories: fields, methods and properties\nNote: among all members, only the content of ordinary fields is stored in objects, that is, however many objects are created from this class, there will be that many ordinary fields in memory. The other members are all stored in the class, that is, no matter how many objects there are, only one copy is created in memory.\nFields Fields include ordinary fields and static fields; ordinary fields belong to objects and static fields belong to the class.\nclass World: string=\u0026#39;earth\u0026#39; #static field def _init_(self,name): self.name=\u0026#39;name\u0026#39; #ordinary field obj=World(\u0026#39;China\u0026#39;) #directly access the ordinary field print(obj.name) #directly access the static field world.string From the code above we can see that ordinary fields are accessed through objects and static fields through the class. That is, static fields belong to the class while ordinary fields belong to objects, and the class does not store the ordinary fields of objects.\nWhen creating objects from a class, if every object has the same field, use a static field\nMethods Methods include: ordinary methods, static methods and class methods. All three kinds of methods belong to the class in memory; the difference is in how they are called.\nOrdinary methods: called by objects; at least one self parameter; when an ordinary method is executed, the object calling the method is automatically assigned to self;\nClass methods: called by the class; at least one cls parameter; when a class method is executed, the class calling the method is automatically copied to cls;\nStatic methods: called by the class; no default parameter;\nclass Foo: def _init_(self,name): self.name-name #define an ordinary method with at least one self parameter def ord_func(self): print(\u0026#39;ordinary method\u0026#39;) #define a class method with at least one cls parameter @classmethod def class_func(cls): print(\u0026#39;class method\u0026#39;) #define a static method with no default parameter @staticmethod def static_func(): print(\u0026#39;static method\u0026#39;) #call the ordinary method f=Foo() f.ord_func() #call the class method Foo.class_func() #call the static method Foo.static_func() These methods have in common that they all belong to the class (not to objects), so only one copy is kept in memory. What differs is how the methods are called, and the parameters passed when calling them.\nProperties Basic use of properties class Foo: def func(self): pass @property #define a property def prop(self): pass foo_obj=Foo() foo_obj.func() foo_obj.prop #call the property A few points to note about defining and calling properties:\nWhen defining, the @property decorator is added on top of an ordinary method; When defining, a property has only one self parameter; When calling, no parentheses are needed; Method: foo_obj.func(); Property: foo_obj.prop A property is derived from a method; if there were no properties in Python, methods could fully replace them.\nThe existence of properties can create the illusion of accessing a field.\nThe function of a Python property is: a series of logical computations are carried out inside the property, and the result of the computation is finally returned.\nTwo ways of defining properties: Decorator, that is, applying a decorator to a method Static field, that is, defining in the class a static field whose value is a property object Decorator approach: apply the @property decorator to an ordinary method of the class\nFor old-style classes, there is one kind of @property decorator\nclass Goods: @property def price(self): return \u0026#34;four yuan\u0026#34; obj=Goods() result=obj.price #automatically executes the price method decorated with @property and obtains its return value For new-style classes, there are three kinds of @property decorators\nclass Goods(object): @property def price(self): print(\u0026#39;@property\u0026#39;) @price.setter def price(self,value): print(\u0026#39;@price.setter\u0026#39;) @price.deleter def price(self): print(\u0026#39;@price.deleter\u0026#39;) obj=Goods() obj.price #automatically executes the price method decorated with @property and obtains its return value obj.price=890 #automatically executes the price method decorated with @property and assigns 890 to the method\u0026#39;s parameter del obj.price #automatically executes the price method decorated with @property From the above we can see: a property in an old-style class has only one access mode, corresponding to the method decorated with @property; a property in a new-style class has three access modes, corresponding respectively to the three methods decorated with @property, @method_name.setter and @method_name.deleter\nSince a new-style class has three access modes, according to the access characteristics of these properties, the three methods are defined for the same property as: get, set and delete.\nStatic field approach: create a static field whose value is a property object\nWhen creating a property using the static field approach, old-style and new-style classes are no different\nclass Foo: def get_bar(self): return \u0026#39;five yuan\u0026#39; BAR=property(get_bar) obj=Foo() result=obj.BAR #automatically calls get_bar and obtains the method\u0026#39;s return value print(result) The property constructor has four parameters\nThe first parameter is a method name; calling object.property automatically triggers the execution of the method The second parameter is a method name; calling object.property=XXX automatically triggers the execution of the method The third parameter is a method name; calling del object.property automatically triggers the execution of the method The fourth parameter is a string; calling object.property.__doc__ returns it — this parameter is the description of the property class Foo: def get_bar(self): return \u0026#39;five yuan\u0026#39; # requires two parameters def set_bar(self, value): return \u0026#39;set value\u0026#39; + value def del_bar(self): return \u0026#39;five yuan\u0026#39; BAR = property(get_bar, set_bar, del_bar, \u0026#39;description...\u0026#39;) obj = Foo() obj.BAR #automatically calls the method defined in the first parameter: get_bar #automatically calls the method defined in the second parameter: set_bar, passing \u0026#34;alex\u0026#34; as the parameter obj.BAR = \u0026#34;alex\u0026#34; del Foo.BAR #automatically calls the method defined in the third parameter: del_bar obj.BAR.__doc__ #automatically obtains the value set in the fourth parameter: description... Since creating a property with the static field approach provides three access modes, in the same way, according to the access characteristics of the properties, the three methods are defined for the same property as: get, set and delete\nModifiers of class members All the members of a class have been described in detail above; for each member of a class there are two forms:\nPublic members, accessible anywhere Private members, accessible only inside the class Private members and public members are defined differently: when naming a private member, the first two characters are underscores. (Except for special members, for example: __init__, __call__, __dict__ and so on)\nclass C: def __init__(self): self.first=\u0026#39;public field\u0026#39; self._second=\u0026#39;private field\u0026#39; The access restrictions for private members and public members differ:\nStatic fields\nPublic static fields: accessible by the class; accessible inside the class; accessible in derived classes Private static fields: accessible only inside the class class C: name=\u0026#39;public static field\u0026#39; def func(self): print(C.name) class B(C): def show(self): print(C.name) C.name #class access obj=C() obj.func() #access inside the class obj_son=D() obj_son.show() #access in a derived class class C: __name=\u0026#39;private static field\u0026#39; def func(self): print(C.__name) class B(C): def show(self): print(C.__name) C.__name #class access, results in an error obj=C() obj.func() #access inside the class obj_son=D() obj_son.show() #access in a derived class, results in an error Ordinary fields\nPublic ordinary fields: accessible by objects; accessible inside the class; accessible in derived classes Private ordinary fields: accessible only inside the class (They can be accessed through object.__class name__private field name, but forcing access to private members is not recommended)\nThe access to methods and properties is similar to the above, that is: private members can only be accessed inside the class\nSpecial members of classes In Python there are some members with special meanings, as follows:\n__doc__: represents the description information of the class class Foo: \u0026#34;\u0026#34;\u0026#34;descrptioin....\u0026#34;\u0026#34;\u0026#34; def func(self): pass print(Foo.__doc__) #outputs \u0026#34;\u0026#34;\u0026#34;descrption...\u0026#34;\u0026#34;\u0026#34; __module__ and __class__\n__module__ indicates the module in which the currently operated object is located;\n__class__ indicates what the class of the currently operated object is\n__init__: the constructor, which is automatically triggered and executed when the object is created\n__del__: the destructor, which is automatically triggered and executed when the object is released from memory\nThis method generally does not need to be defined; usually the destructor is automatically triggered by the interpreter during garbage collection.\n__call__: adding parentheses after an object triggers execution\nThe execution of the constructor is triggered by the created object, that is: object=class_name(); whereas the execution of __call__ is triggered by adding parentheses after the object, that is: object() or class()()\nclass Foo: def __init__(self): pass def __call__(self,*args,**kwargs): print(\u0026#39;__call__\u0026#39;) obj=Foo() #executes __init__ obj() #executes __call__ __dict__: the members in a class or object\nThe ordinary fields of a class belong to objects; the static fields and methods in a class belong to the class\n__str__: if a class defines a __str__ method, printing the object outputs the return value of that method by default\nclass A: def __str__(self): return \u0026#39;simplestory\u0026#39; obj=A() print(obj) #output: simplestory __getitem__, __setitem__, __delitem__\nUsed for index operations, denoting getting, setting and deleting data respectively\nclass A(object): def __getitem__(self,key): print(\u0026#39;__getitem__\u0026#39;,key) def __setitem__(self,key,value): print(\u0026#39;__setitem__\u0026#39;,key,value) def __delitem__(self,key): print(\u0026#39;__delitem__\u0026#39;,key) obj=A() result=obj[\u0026#39;k1\u0026#39;] #automatically executes __getitem__ obj[\u0026#39;k2\u0026#39;]=\u0026#39;simplestory\u0026#39; #automatically executes __setitem__ del obj[\u0026#39;k1\u0026#39;] #automatically executes __delitem__ __getslice__, __setslice__, __delslice__\nThese three methods are used for slice operations, for example on lists\n__iter__\nThe reason lists, dictionaries and tuples can be iterated with a for loop is that the type internally defines __iter__ a for loop actually iterates over the result of iter\nobj=iter([1,2,3,4,5,6,7]) for i in obj: print(i) __new__ and __metcalss__ class Foo(object): def __init__(self): pass obj=Foo() #obj is an object instantiated through the Foo class Everything in Python is an object According to this theory we can conclude: the obj object is created by executing the constructor of the Foo class, so the Foo class object should also be created by executing the constructor of some class. (The parent class of a class can be checked with the type() function)\n#continuing from the code snippet above print(type(obj)) #output: \u0026lt;class \u0026#39;__main__.Foo\u0026#39;\u0026gt;, meaning the obj object is created by the Foo class print(type(Foo)) #output: \u0026lt;type \u0026#39;type\u0026#39;\u0026gt;, meaning the Foo class object is created by the type class Class creation can in fact be done in two ways:\nThe ordinary way class Foo(object): def func(self): print(\u0026#39;Hello World!\u0026#39;) The special way (the constructor of the type class) def func(self): print(\u0026#39;Hello World\u0026#39;) Foo=type(\u0026#39;Foo\u0026#39;,(object,),{\u0026#39;func\u0026#39;:func}) #first parameter of type: class name #second parameter of type: base classes of the current class #third parameter of type: the members of the class A class is produced by instantiating the type class\nAs for the type class, there is a __metaclass__, used to indicate by what the class is instantiated and created. So we can set a derived class of the type class for __metaclass__, and then all classes created within the scope of __metaclass__ will use the class or function specified by __metaclass__ as the derived class of the type class.\n","permalink":"https://simplestory.github.io/en/posts/20170820_7cb802827b60ef8c770d824ec28a65a0/","summary":"\u003cblockquote\u003e\n\u003cp\u003e\u0026ldquo;Class\u0026rdquo; is a template, and the template packages many \u0026ldquo;functions\u0026rdquo; for use. This post introduces the members of Python classes, member modifiers, and the special members of classes.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"class-members\"\u003eClass members\u003c/h2\u003e\n\u003cp\u003eThe members of a class can roughly be divided into three categories: fields, methods and properties\u003c/p\u003e\n\u003cp\u003eNote: among all members, only the content of ordinary fields is stored in objects, that is, however many objects are created from this class, there will be that many ordinary fields in memory. The other members are all stored in the class,\nthat is, no matter how many objects there are, only one copy is created in memory.\u003c/p\u003e","title":"Python Classes"},{"content":" Python is an interpreted, object-oriented, high-level programming language with dynamic semantics. Object Oriented Programming (OOP) requires \u0026ldquo;classes\u0026rdquo; and \u0026ldquo;objects\u0026rdquo; to implement, so object-oriented programming is the use of \u0026ldquo;classes\u0026rdquo; and \u0026ldquo;objects\u0026rdquo;.\nBrief introduction Procedure oriented: Write code top-down according to the program logic.\nFunctional: Encapsulate the code for some functionality into functions, and afterwards there is no need to write it again — just call the function. This generally suits scenarios where the functions are independent and share no data.\nObject oriented: Classify and encapsulate functions.\nAbout classes and objects A class is a template, and the template contains many functions, each implementing some functionality An object is an instance created from the template; through the object you can execute the functions in the template #create a class class A: def Func(self): print(\u0026#34;Afun\u0026#34;) def Hello(self,name): print(\u0026#34;Hello,%s\u0026#34;%name) #create object obj from class A obj=A() obj.Func #call the Func method obj.Hello(\u0026#39;Alex\u0026#39;) #call the Hello method The first parameter of a function in a class must be self (it is the reference to the object itself);\nFunctions defined in a class are called \u0026ldquo;methods\u0026rdquo;.\nThe three major features of object orientation The three major features of object orientation are: encapsulation, inheritance and polymorphism.\nEncapsulation Encapsulation means hiding the working details of an object from the outside world. When using the encapsulation feature of object orientation, you need to:\nEncapsulate the content somewhere Call the encapsulated content from somewhere class Foo: #__init__ is called the constructor and is executed automatically when an instance of the class is created def __init__(self,name,age): self.name=name self.age=age obj=Foo(\u0026#39;Alex\u0026#39;,20) #encapsulate Alex and 20 in the name and age attributes of obj/self There are two ways to call the encapsulated content:\nCall it directly through the object Call it indirectly through self class Foo: def __init__(self,name,age): self.name=name self.age=age def Detail(self): print(self.name) print(self.age) obj=Foo(\u0026#39;Alex\u0026#39;,20) print(obj.name) #directly call the name attribute of the obj object obj.Detail() #indirectly call the attributes of the obj object In summary, for object-oriented encapsulation, what happens is that the constructor is used to encapsulate the content into the object, and then the encapsulated content is obtained either directly through the object or indirectly through self.\nInheritance Build specialized class objects on the basis of a general class\nclass Animal(): def eat(self): pass def drink(self): pass def shit(self): pass def pee(self): pass class Cat(Animal): def sound(self): print(\u0026#34;Meow\u0026#34;) class Dog(Animal): def sound(self): print(\u0026#34;Woof\u0026#34;) From the code above we can see that both the Cat and Dog classes take Animal as their base class, so both Cat and Dog have the methods of the Animal class; this is inheritance.\nMultiple inheritance\nA Python class can inherit from multiple classes If a Python class inherits from multiple classes, there are two ways of looking things up: depth first and breadth first When the class is an old-style class, in the multiple inheritance case the lookup follows the depth-first manner When the class is a new-style class, in the multiple inheritance case the lookup follows the breadth-first manner (When a class or its parent class inherits from the object class, the class is a new-style class, otherwise it is an old-style class) class A(): def foo(self): print(\u0026#39;A.foo\u0026#39;) class B(A): def foo(self): print(\u0026#39;B.foo\u0026#39;) class C(A): def foo(self): print(\u0026#39;C.foo\u0026#39;) class D(B,C): def foo(self): print(\u0026#39;D.foo\u0026#39;) k=D() Old-style class: first look in class D; if not found in D, continue to look in class B; if not found in B, continue to look in class A; if not found in A, continue to look in class C; if still not found, raise an error.\nNew-style class: first look in class D; if not found in D, continue to look in class B; if not found in B, continue to look in class C; if not found in C, continue to look in class A; if still not found, raise an error.\nIn the process above, once something is found the lookup stops immediately and does not continue.\nPolymorphism Polymorphism means that the same operation can be applied to objects of different classes.\nclass Foo: pass class S1(Foo): def show(self): print(\u0026#39;S1.show\u0026#39;) class S2(Foo): def show(self): print(\u0026#39;S2.show\u0026#39;) def Func(obj): print(obj.show) s1_obj=S1() Func(s1_obj) #pass an object of class S1 into Func(), which executes the show method of class S1 s2_obj=S2() Func(s2_obj) #pass an object of class S2 into Func(), which executes the show method of class S2 Summary The above is an introduction to the object-oriented programming style; this style is implemented on the basis of using classes and objects A class is a template, and the template packages multiple functions for use An object is an instance created from the template, and the instance is used to call the functions packaged in the class Object orientation has three features: encapsulation, inheritance and polymorphism Aside How classes and objects are stored in memory A class and the methods in the class exist only once in memory, whereas every object created from the class needs its own copy in memory, because the values of methods or variables may differ between objects, and between an object and its base class. When an object is created from a class, in addition to encapsulating the values of its own variables, the object also stores a pointer to a class object, which points to the class of the current object.\n","permalink":"https://simplestory.github.io/en/posts/20170810_3193456a2ce85ef1e01f5a219b37cebd/","summary":"\u003cblockquote\u003e\n\u003cp\u003ePython is an interpreted, object-oriented, high-level programming language with dynamic semantics. Object Oriented Programming (OOP) requires \u0026ldquo;classes\u0026rdquo; and \u0026ldquo;objects\u0026rdquo; to implement, so object-oriented programming is the use of \u0026ldquo;classes\u0026rdquo; and \u0026ldquo;objects\u0026rdquo;.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"brief-introduction\"\u003eBrief introduction\u003c/h2\u003e\n\u003ch3 id=\"procedure-oriented\"\u003eProcedure oriented:\u003c/h3\u003e\n\u003cp\u003eWrite code top-down according to the program logic.\u003c/p\u003e\n\u003ch3 id=\"functional\"\u003eFunctional:\u003c/h3\u003e\n\u003cp\u003eEncapsulate the code for some functionality into functions, and afterwards there is no need to write it again — just call the function. This generally suits scenarios where the functions are independent and share no data.\u003c/p\u003e","title":"Python, Object Oriented"},{"content":" Git is an extremely good distributed version control system. The system can record every change to a user\u0026rsquo;s files and also has powerful team collaboration capabilities: everyone can commit changes to the text without worrying about overwriting the operations of others. Github is furthermore a software source code hosting service version-controlled with Git. As of 2015, Github had more than nine million registered users and 21.1 million repositories. Github has become the largest code hosting website and open source community in the world.\nGIT usage tutorial Creating a repository Choose a suitable place and create an empty directory\nmkdir \u0026lt;dir-name\u0026gt; cd \u0026lt;dir-name\u0026gt; #operate inside the repository pwd #you can check the current directory git init #turn the directory into a repository Git can manage, that is, initialize the repository Afterwards a .git directory is created in the current directory; this directory is used by Git to track and manage the repository. A non-empty directory can also be used to create a Git repository\nWriting and saving files Write the file and place it in the \u0026lt;dir-name\u0026gt; directory (or a subdirectory)\nEdit and save the file in plain text, saving the file encoding in UTF-8 format\ngit add \u0026lt;filename1\u0026gt; \u0026lt;filename2\u0026gt; ... #tell Git to add the files to the repository git commit -m \u0026#34;...\u0026#34; #commit the files to the repository; enter the description of this commit after -m; multiple files can be committed at once git status #check the current status of the repository, telling the user whether any file has been modified git diff #check what exactly was modified in the files, or compare files git log #show the commit log from most recent to oldest (the parameter --pretty=oneline can be added to change the display view) git reset --hard HEAD^ #revert the files to the previous version In Git, HEAD represents the current version, the previous version is HEAD^, the one before that is HEAD^^, and N versions back can be written as HEAD~N\ngit reset --hard \u0026lt;fileID\u0026gt; #jump the file version to the file with version ID \u0026lt;fileID\u0026gt; git reflog #records every operation of the user, from which you can check the version number before the file was reverted Working directory and staging area Working directory The directory you can see on the computer is the working directory, used to store files\nRepository and staging area There is a hidden directory .git under the working directory; this is the git repository, and the repository contains the staging area (stage/index)\nRoughly how Git works Adding files to the Git repository takes two steps:\ngit add adds the file into the staging area, that is, adds the modification or related operations on the file to the staging area git commit commits the changes, that is, submits everything in the staging area to the current branch Undoing changes to files git checkout -- \u0026lt;filename\u0026gt; undoes all modifications to in the working directory; the -- in the command is very important, without -- it becomes the command for switching to another branch\nCases of undoing changes to files As for when changes to a file need to be undone, there are roughly two cases:\nIf \u0026lt;filename\u0026gt; has been modified but not yet put into the staging area, then after the command executes the changes are undone and the file returns to exactly the same state as the repository If \u0026lt;filename\u0026gt; has already been added to the staging area and then modified again, then after the command executes the changes are undone and the file returns to the state it had after being added to the staging area In short, it makes the file return to the state of the most recent git commit or git add Summary of undoing and reverting When you have messed up the content of a file in the working directory and want to simply discard the modifications in the working directory, use the command git checkout -- \u0026lt;filename\u0026gt; When you have not only messed up the content of a file in the working directory but also added it to the staging area, and want to discard the modifications, do it in two steps: use the command git reset HEAD \u0026lt;filename\u0026gt; to go back to case 1, then continue with the operation in 1 When you have already committed an inappropriate modification to the repository and want to undo this commit, refer to version reverting — provided it has not been pushed to the remote repository Version reverting: git reset --hard \u0026lt;ID\u0026gt; jumps the file to the version with version number ID; HEAD represents the latest file version\nUndoing deletions in Git Deletion is also a modification operation in Git\nNormally, after committing to the repository, we delete a file directly in the file manager (or with the rm command); git status will tell you which files have been deleted, and there are then two choices:\nIf you really want to delete it from the repository, run the git rm command to delete it and then git commit If you deleted it by mistake, you can run git checkout -- \u0026lt;filename\u0026gt; to restore the mistakenly deleted file to the latest version git checkout in fact replaces the version in the working directory with the version in the repository; whether the working directory version was modified or deleted, it can be \u0026ldquo;restored with one click\u0026rdquo;, The command git rm is used to delete a file. If a file has already been committed to the repository, you never need to worry about deleting it by mistake, but you can only restore the file to the latest version, and you will lose the content after the most recent modification commit Creating an SSH Key ssh-keygen -t rsa -C \u0026#34;youremail@example.com\u0026#34; After creating it, you can find the .ssh directory in the user\u0026rsquo;s home directory, containing the two files id_rsa and id_rsa.pub; these two files are the key pair of the SSH Key. id——rsa is the private key and must not be leaked, while id_rsa.pub is the public key and can be published. Then log into Github to configure it: go to the \u0026ldquo;SSH Key\u0026rdquo; page, click \u0026ldquo;Add SSH Key\u0026rdquo;, fill in any title and paste the content of the id_rsa.pub file into the Key text box\nGithub allows adding multiple Keys\nAssociating a remote repository First create a new repository on Github (or use an existing one), then run the following command in a local repository:\ngit remote add origin git@github.com:\u0026lt;accountname\u0026gt;/\u0026lt;localreponame\u0026gt;.git After adding it, the default name of the remote repository in Git is origin; it can be changed:\ngit push -u origin master The git push command actually pushes the current branch master to the remote, If this is the first time you push the master branch you need to add the -u parameter; Git will not only push the content of the local master branch to the new remote master branch, but also associate the local master branch with the remote master branch, which simplifies the command for later pushes or pulls. Afterwards, as long as you have made a local commit, you can complete the submission with the command: git push origin master When using Git\u0026rsquo;s clone or push command for the first time you will get an SSH warning; just type yes\n#associate a remote repository git remote add origin git@server-name:path/repo-anme.git Cloning a remote repository to local After creating a repository in Github, you can use git clone to clone a local repository git clone git@github.com:\u0026lt;git repository address\u0026gt;\nGit supports multiple protocols, including https\nwindows systems: git clone \u0026lt;URL of the Git remote repository\u0026gt;\nGit branches git branch #view branches git branch \u0026lt;name\u0026gt; #create a branch git checkout \u0026lt;name\u0026gt; #switch branches git checkout -b \u0026lt;name\u0026gt; #create + switch branch git merge \u0026lt;name\u0026gt; #merge some branch into the current branch git branch -d \u0026lt;name\u0026gt; #delete a branch Multiple branches share the staging area, that is, if you only add without committing on branch 1, the staging area has in fact already recorded that modification, and even switching to branch 2 afterwards and committing there is a valid operation When Git cannot merge branches automatically, the conflict has to be resolved first. Commit after resolving the conflict, with the git log --graph command you can see the branch merge graph, and git status can also show the conflicting files\ne.g: git log --graph --pretty=oneline --abbrev-commit\nGit branch management When merging branches, adding the --no-ff parameter allows merging in normal mode; the history after such a merge has branches, so you can tell that a merge was once done, whereas with fast-forward merging you cannot tell that a merge was done\ngit merge --no-ff -m\u0026#34;...\u0026#34; \u0026lt;name\u0026gt; Merging a branch in normal mode (that is, disabling fast forward) requires creating a new commit for this merge, so add the -m parameter and write the commit description\nBranch strategy: In actual development we should manage branches according to a few basic principles: The master branch should be very stable, that is, used only to release new versions, and you do not normally work on it; Work happens on the dev branch, that is, the devd branch is unstable, and at some point the dev branch is merged into the master branch; Everyone works on the dev branch, and each person has their own branch; when it needs to be saved, just merge it into the dev branch Bug branches The git stash command is equivalent to taking a snapshot of the current working environment, so that you can restore the environment later and continue working\nFix the bug on whichever branch it should be fixed on, and create a temporary branch from that branch\nCommit and delete the branch after fixing\ngit stash list #check whether a stash exists git stash apply #restore the stash content; this command does not delete the stash content after running git stash drop #delete the stash content git stash pop #restore the stash content and delete it at the same time When fixing a bug while the work at hand is not finished, first git stash the work in progress, then go and fix the bug, and after fixing it git stash pop back to the work in progress\nFeature branches (adding new features) When developing a new feature it is best to create a new branch, To discard a branch that has never been merged, you can force its deletion with git branch -D \u0026lt;name\u0026gt;\nCollaboration git remote shows information about the remote repositories\ngit remote -v shows detailed information about the remote repositories (it displays the addresses of the for fetching and pushing; if you have no push permission, the push address is not shown)\nPushing branches: Pushing a branch means pushing all the local commits on that branch to the remote repository. When pushing, the local branch must be specified\ngit push \u0026lt;remote repository name\u0026gt; \u0026lt;local repository name\u0026gt; git pull #fetch the latest commits from the remote repository Merge locally after fetching and push again after resolving conflicts\nWorking pattern: First, you can try to push your changes with git push origin branch-name If the push fails, it is because the remote branch is newer than your local version, so you need to try to merge first with git push If the merge has conflicts, resolve them and commit locally Once there are no conflicts or the conflicts have been resolved, pushing again with git push origin branch-name will succeed If git pull says \u0026ldquo;no tracking information\u0026rdquo;, it means the link between the local branch and the remote branch has not been created; use the command git branch --set-upstream branch-name origin/branch-name (this command establishes the association between the local branch and the remote branch)\nTag management When releasing a version, we usually first tag the repository, so that the version at the time of tagging is uniquely determined. At some point in the future, taking a certain version of a certain tag means taking out the version corresponding to the moment the tag was created. A tag is also a snapshot of the repository\nAlthough a Git tag is a snapshot of the repository, it is in fact a pointer to a certain commit (branches can move, tags cannot)\nA tag is just an easy-to-remember name bound to a certain commit\ngit tag \u0026lt;name\u0026gt; #to tag in git you should first switch to the branch that needs the tag git tag #view all tags; by default a tag is created on the commit of the latest submission git tag \u0026lt;name\u0026gt; \u0026lt;commit ID\u0026gt; #tag the commit with the corresponding ID Tags are not listed in chronological order but sorted alphabetically; git show \u0026lt;tagname\u0026gt; can be used to view tag information\ngit tag -a \u0026lt;tagname\u0026gt; -m \u0026#34;...\u0026#34; #create a tag with a description; -a specifies the tag name and -m specifies the description text git tag -d \u0026lt;tagname\u0026gt; #delete a local tag, because created tags are only stored locally and are not automatically pushed to the remote git push \u0026lt;remote repository name\u0026gt; \u0026lt;tagname\u0026gt; #push a certain tag to the remote git push \u0026lt;remote repository name\u0026gt; --tags #push all local tags not yet pushed to the remote in one go If in the first line of code above the parameter -s is used instead of -a, a PGP-signed tag can be created\nDeleting a tag that has already been pushed to the remote:\ngit push \u0026lt;remote repository name\u0026gt; :refs/tags/\u0026lt;tagname\u0026gt; On Github you can freely Fork open source repositories;\nyou have read and write permission on the repository you Forked;\nyou can push Pull requests to other people\u0026rsquo;s repositories to contribute code;\nafter Forking someone else\u0026rsquo;s repository, clone the repository under your own account, so that you can push your modifications\nCustomizing Git git config --global color.ui true #git will display different colors appropriately Ignoring files In the root directory of the Git working directory there is a .gitignore file (create it if it does not exist); fill in the names of the files to be ignored, and Git will automatically ignore these files\nPrinciples for ignoring files:\nIgnore files automatically generated by the operating system, such as thumbnails; Ignore intermediate files and executable files generated by compilation — that is, files automatically generated from other files do not need to be put into the repository; Ignore your own configuration files containing sensitive information, such as configuration files that store passwords Afterwards commit .gitignore to Git as well, and the ignored-file configuration is complete\ngit check-ignore #check the .gitignore file git add -f \u0026lt;filename\u0026gt; #force adding a file We can also version-manage .gitignore\nSetting aliases git config --global alias \u0026lt;alias\u0026gt; \u0026lt;original command name\u0026gt; The \u0026ndash;global parameter is a global parameter, meaning these commands are effective in all Git repositories on the computer\nWhen configuring Git, adding --global applies to the current user; without it, it only applies to the current repository. Each repository\u0026rsquo;s Git configuration file is stored in the .git/config file, with aliases after [alias]; to delete an alias just delete the corresponding line. The current user\u0026rsquo;s Git configuration file is the hidden file .gitconfig in the user\u0026rsquo;s home directory\nSetting up a Git server (linux ubuntu or debian) Install git: sudo apt-get install git; Create a git user to run the git service: sudo adduser git; Set up certificate login: collect the public keys of all users who need to log in, that is, their id_rs.pub files, and import all public keys into the /home/git/.shh/authorized_keys file (the name can be changed); Initialize the Git repository: sudo git init --bare \u0026lt;reponame\u0026gt;; the parameter \u0026ndash;bare creates a bare repository, that is, one without a working directory (Git repositories on a server usually end with .git; after creating it change the owner to git with sudo chown -R git:git \u0026lt;reponame\u0026gt;); Disable shell login: for security reasons, the git user created in step 2 is not allowed to log into the shell; this can be done by editing the /etc/passwd file Find a line like: git:x:1001:1001:,,,:/home/git:/bin/bash and change it to: git:x:1001:1001:,,,:/home/git:/usr/bin/got-shell so that when the user logs in with a shell they are automatically logged out; Clone the remote repository: git clone git@server:/srv/\u0026lt;reponame\u0026gt;; run it on each person\u0026rsquo;s own computer Reference: Liao Xuefeng\u0026rsquo;s Git tutorial\n","permalink":"https://simplestory.github.io/en/posts/20170704_497c40137cedefb55b1afe3bd1e6b13f/","summary":"\u003cblockquote\u003e\n\u003cp\u003eGit is an extremely good distributed version control system. The system can record every change to a user\u0026rsquo;s files and also has powerful team collaboration capabilities: everyone can commit changes to the text without worrying about overwriting the operations of others. Github is furthermore a software source code hosting service version-controlled with Git. As of 2015, Github had more than nine million registered users and 21.1 million repositories. Github has become the largest code hosting website and open source community in the world.\u003c/p\u003e\n\u003c/blockquote\u003e","title":"A Basic Git Tutorial"}]