Having studied so many deep learning algorithms, the loss function is always an unavoidable point, but fortunately the loss functions used in general are fairly fixed, with occasional changes made as things go. This post mainly lists some commonly used loss functions.
L1-loss
The L1 loss function, also known as Mean Absolute Error (MAE). Its mathematical expression is as follows:
$$ MAE = \frac{1}{n}\sum^n_{i=1}\vert f(x_i)-y_i\vert $$where $n$ denotes the number of samples in the training batch, $f(x_i)$ denotes the value predicted by the model at sample $x_i$, and $y_i$ is the ground truth value corresponding to sample $x_i$.
L2-loss
The L2 loss function, also known as Mean Square Error (MSE). Its mathematical expression is as follows:
$$ MSE = \frac{1}{n}\sum^n_{i=1}(f(x_i)-y_i)^2 $$where $n$ denotes the number of samples in the training batch, $f(x_i)$ denotes the value predicted by the model at sample $x_i$, and $y_i$ is the ground truth value corresponding to sample $x_i$.
Smooth L1 loss
The L1 loss function is not sensitive to data outliers, while the L2 loss is overly sensitive. This loss function was proposed to make training smooth. Specifically:
$$ loss(x_i) = \begin{cases} 0.5 * (f(x_i)-y_i)^2, \ &\vert f(x_i)-y_i\vert <1 \\ \vert f(x_i)-y_i\vert - 0.5, \ &\text{otherwise} \end{cases} $$Only the loss computation for sample $x_i$ is shown here, where $f(x_i)$ denotes the value predicted by the model at sample $x_i$ and $y_i$ is the ground truth value corresponding to sample $x_i$.
Cross Entropy loss
The cross entropy loss function (CE) is mainly used in classification algorithms.
For 0-1 classification, we have
$$ loss(x_i) = -y_i * log(p_i)-(1-y_i) * log(1-p_i) $$where $p_i$ denotes the probability that the model predicts sample $x_i$ as positive, and $y_i$ is the ground truth class of sample $x_i$, being 1 for positive samples and 0 for negative samples.
For multi-class classification, we have
$$ loss(x) = -\sum^c_{j=1}y_i * log(p_i) \\ y_i = \begin{cases} 1, \ &\text{the label of x is i} \\ 0, \ &\text{otherwise} \end{cases} $$where $p_i$ is the probability that the model predicts the sample as class $i$; the model generally applies softmax first and then computes the cross entropy loss.
Focal loss
This loss function is meant to deal with the imbalance between positive and negative samples, mainly addressing the skewed ratio of positive to negative samples among the anchors in single-stage object detection algorithms.
Let
$$ p_t = \begin{cases} p, \ & y=1 \\ 1-p, \ & y=0 \end{cases} $$then the cross entropy loss function can be written uniformly as:
$$ loss = -log(p_t) $$To adjust the ratio of positive to negative samples, a natural idea is to weight with a coefficient, as follows:
$$ loss = -\alpha_t * log(p_t) \\ \alpha_t = \begin{cases} \alpha, \ & y=1 \\ 1-\alpha, \ & y=0 \end{cases} $$But this does not distinguish easy samples from hard samples, and normally the number of easy samples is far larger than the number of hard samples. Switching finally to an adaptive coefficient gives:
$$ loss = -(1-p_t)^\gamma * log(p_t) $$The authors found experimentally that adjusting the adaptive coefficient gives a slight improvement in model performance, so the final loss function is as follows:
$$ loss = -\alpha(1-p_t)^\gamma * log(p_t) $$where both $\alpha$ and $\gamma$ are hyperparameters. In the authors’ own experiments on the model, $\alpha=0.25$ and $\gamma=2$ worked best. This function is only used when computing the classification loss; see the loss function of the Retina model for details.
Reference
Object detection: RetinaNet: Focal Loss for Object Detection