I have not updated the blog for a long time, having paused for about half a year, and I have also read half a year’s worth of books on machine learning and deep learning; now it is time to write some output

Linear regression

Given a dataset $D=\\{(\mathbf{x_1},y_1),(\mathbf{x_2},y_2),...,(\mathbf{x_m},y_m)\\}$, where $\mathbf{x_i}=(x_{i1};x_{i2};...;x_{id}), \ y_{i} \in R$. Linear regression tries to learn a linear model

$$f(x_i)=wx_i+b$$

such that $f(x_i)\simeq y_i$.

Derivation

Consider using MSE (mean squared error) to measure how good the model is; then the loss function $L(w,b)$ is:

$$ \begin{aligned} L(w,b) & =\sum_{i=1}^m (f(x_i)-y_i)^2 \\ & =\sum_{i-1}^m (y_i-wx_i-b)^2 \end{aligned} $$

The objective function is:

$$ \begin{aligned} (w^*,b^*) & =argmin_{(w,b)} L(w,b) \\ & =argmin_{(w,b)} \sum_{i=1}^m (y_i-w_i-b)^2 \end{aligned} $$

Taking the partial derivatives of $L(w,b)$ with respect to $w$ and $b$ respectively, setting the derivatives to zero and solving the resulting system gives

$$ \begin{cases} \nabla_w L(w,b) = 2(w\sum_{i=1}^m x_i^2 -\sum_{i=1}^m (y_i-b)x_i )=0 \\ \nabla_n L(w,b) = a(mb-\sum_{i=1}^m (y_i-wx_i))=0 \end{cases} $$

Solving gives the optimal closed-form solution for $w$ and $b$:

$$w=\frac{\sum_{i=1}^m y_i(x_i-\overline{x})}{\sum_{i=1}^m x_i^2 - \frac{1}{m}(\sum_{i=1}^m x_i)^2}$$$$b=\frac{1}{m}\sum_{i=1}^m (y_i-wx_i)$$

where $\overline{x}=\frac{1}{m} \sum_{i=1}^m x_i$ is the mean of $x$

Use in Scikit-learn

See the official Scikit-learn site for the concrete API: sklearn.linear_model.LinearRegression

Logistic regression

This is a classification method

Logistic distribution:

Let X be a continuous random variable; $\mathbf{X}$ follows a logistic distribution means that $\mathbf{X}$ has the following distribution function and density function:

$$ \begin{cases} F(x)=P(X \leq x)=\frac{1}{1+e^{-(x-\mu)/\gamma}} \\ f(x)=F\prime(x)=\frac{e^{-(x-\mu)/\gamma}}{\gamma(1+e^(x-\mu)/\gamma)^2} \end{cases} $$

where $\mu$ is the location parameter and $\gamma > 0$ is the shape parameter

The binomial logistic regression model gives the following conditional probabilities:

$$P(Y=1|x)=\frac{exp(w\cdot x+b)}{1+exp(w\cdot x+b)}$$$$P(Y=0|x)=\frac{1}{1+exp(w\cdot x+b)}$$

Here $x\in R^n$ is the input, $Y\in \\{0,1\\}$ is the output, $w\in R^n$ is the weight vector, $b\in R$ is the bias and $w\cdot x$ is the inner product.

For convenience of expression, merging w and b gives:

$$P(Y=1|x)=\frac{exp(w\cdot x)}{1+exp(w\cdot x)}$$$$P(Y=0|x)=\frac{1}{1+exp(w\cdot x)}$$

From the assumptions and conclusions above, the logistic regression model returns the probability that the sample is positive, that is, $P(Y=1\vert x)$

Derivation

From the maximum likelihood function we obtain the loss function $L(w)$:

$$L(w)=\prod_{i=1}^N P(Y=1|x_i)^{y_i}[1-P(Y=0|x_i)]^{1-y_i}$$

To prevent overflow caused by the chained multiplication, take the logarithm:

$$L(w)=\sum_{i=1}^N [y_i\log{P(Y=1|x_i)}+(1-y_i)\log{1-P(Y=0|x_i)}]$$

which simplifies to:

$$L(w)=\sum_{i=1}^N [y_i(w\cdot x_i)-\log{(1+exp(w\cdot x_i))}]$$

Maximizing $L(w)$ gives the estimate of w. Gradient descent and quasi-Newton methods are usually used to solve it

Use in Scikit-learn

sklearn.linear_model.LogistRegression

Reference

Zhou Zhihua’s Watermelon Book and Li Hang’s Statistical Learning Methods