Neural networks already have several normalization algorithms. They all look very similar and basically share a unified form. This post refers to the Zhihu column SIGAI’s article on normalization algorithms; the link is at the bottom of the post.
The basic problem
When using a neural network to learn from a dataset, because the sources of the data may differ considerably, the values of different attributes may also span very different ranges; all of this causes errors in how the data is treated and leads to poor learning by the network. To solve this problem we normalize the data, that is, we regularize data whose value ranges vary widely into the same scale, which is beneficial for the network’s learning. Up to now normalization has many variants, the common ones being Batch Normalization (BN), Layer Normalization (LN), Instance Normalization (IN) and Group Normalization (GN). In fact their processing is similar (subtract the mean, divide by the standard deviation, then apply a linear mapping); the difference lies in the dimension along which the statistics are computed. The concrete formula is as follows:
$$ y = \gamma \left(\frac{x-\mu(x)}{\sigma(x)}\right)+\beta $$Note there is one more difference here: for $BN$, $IN$ and $GN$, the parameters $\gamma$ and $\beta$ are vectors whose dimension equals the number of channels $C$, whereas for $LN$ both parameters are matrices whose dimension equals the $normalized\ shape$.
For convenience in the following explanation, the dimension of the dataset is defined in advance as: $x \in R^{N \times C \times H \times W}$, which specifically means that the dataset contains $N$ samples, each with $C$ channels, height $H$ and width $W$.
Batch Normalization
This method is the earliest normalization algorithm and its effect is also good — arguably the best. The computation dimension is $N \times H \times W$, keeping the number of channels. The concrete formulas are as follows:
$$ \begin{aligned} \mu_c(x) & = \frac{1}{NHW}\sum_{n=1}^N\sum_{h=1}^H\sum_{w=1}^W x_{nchw} \\ \sigma_c(x) & = \sqrt{\frac{1}{NHW}\sum_{n=1}^N\sum_{h=1}^H\sum_{w=1}^W (x_{nchw}-\mu_c(x))^2+\epsilon} \end{aligned} $$That is, keeping the number of channels fixed, the first channel of the first sample is added to the first channel of the second sample and so on up to the first channel of the $N$th sample, and dividing by $N\times H \times W$ gives a single number, which is the mean of the first channel (not an $H\times W$ matrix); the standard deviation is computed in the same way.
$BN$ has a drawback: it requires a large amount of data to reasonably estimate the mean and variance of the training data, which may easily lead to insufficient memory, and for $RNN$ models whose training sequences have variable lengths, $BN$ is also hard to apply.
Note one point here: when a BN layer is used in the network, there is no need to use a Dropout layer any more
Layer Normalization
$LN$ has a big advantage: it does not need batch training, and can normalize a single sample on its own. The computation dimension is $C \times H \times W$, keeping the number of samples. The concrete formulas are as follows:
$$ \begin{aligned} \mu_n(x) & = \frac{1}{CHW}\sum_{c=1}^C\sum_{h=1}^H\sum_{w=1}^W x_{nchw} \\ \sigma_n(x) & = \sqrt{\frac{1}{CHW}\sum_{c=1}^C\sum_{h=1}^H\sum_{w=1}^W (x_{nchw}-\mu_n(x))^2+\epsilon} \end{aligned} $$That is, keeping the number of samples fixed, the data corresponding to the channels, height and width of the sample is summed and divided by $C \times H \times W$ to obtain the mean of that sample; the standard deviation is analogous.
Instance Normalization
In generative models, the mean and standard deviation of each channel in the dataset affect the style of the final generated image, so the authors of $IN$ believe that an image can first be normalized at the channel level and then normalized again with the mean and standard deviation of the target style image, thus obtaining the style of the target image. $IN$ also does not need batch training and can normalize a single sample on its own. The computation dimension is $H \times W$, keeping the number of samples and channels. The concrete formulas are as follows:
$$ \begin{aligned} \mu_{nc}(x) & = \frac{1}{HW}\sum_{h=1}^H\sum_{w=1}^W x_{nchw} \\ \sigma_{nc}(x) & = \sqrt{\frac{1}{HW}\sum_{h}^H\sum_{w=1}^W (x_{nchw}-\mu_{nc}(x))^2+\epsilon} \end{aligned} $$That is, keeping the number of samples and channels fixed, the sum of the data over height and width for a single channel of a single sample divided by $H \times W$ is the required mean.
Group Normalization
$GN$ is mainly applied to learning tasks that consume a lot of GPU memory, such as image segmentation. Because such tasks are limited by memory size, the $batchsize$ is usually a single-digit number, and $BN$ simply cannot work well. $GN$ is a compromise between $LN$ and $IN$ and can also normalize a single sample. It divides the channels into $G$ groups, each with $C/G$ channels, then computes the mean and standard deviation of the elements in each group and independently normalizes the channels of the corresponding group with the obtained parameters. The concrete formulas are as follows:
$$ \begin{aligned} \mu_{ng}(x) & = \frac{1}{(C/G)HW}\sum_{c=gC/G}^{(g+1)C/G}\sum_{h=1}^H\sum_{w=1}^W x_{nchw} \\ \sigma_{ng}(x) & = \sqrt{\frac{1}{(C/G)HW}\sum_{c=gC/G}^{(g+1)C/G}\sum_{h=1}^H\sum_{w=1}^W (x_{nchw}-\mu_{ng}(x))^2+\epsilon} \end{aligned} $$Reference
This post mostly refers to the following article: