Machine Learning on Simplestory

LightGBM Model

Sun, 24 Mar 2019 00:00:00 +0000

A Highly Efficient Gradient Boosting Decision Tree

Xgboost Model

Sun, 17 Mar 2019 00:00:00 +0000

A Scalabe Tree Boosting Systeam

How the FFM Model Works

Sun, 28 Oct 2018 00:00:00 +0000

CTR (click-through rate) prediction plays an important role in content recommendation and delivery, and the FFM model is good at handling this kind of data. The FFM model is a variant of the FM model, or rather the FM model is a special case of the FFM model

Rough pipeline

Convert the dataset format (using the LIBSVM or FFMs style)
Compute the loss function (using the exponential loss function)
Optimize the parameters (using the AdaGrad + SG strategy)

Converting the data format

LIBSVM format (does not store zero-valued features):

Decision Tree

Wed, 29 Aug 2018 00:00:00 +0000

Decision tree, a common machine learning method, is somewhat like a flowchart. A sample is divided into a set according to the value it takes on some feature, and this is done recursively

Basic algorithm

Input: training set $D=\\{(\mathbf{x_1},y_1),(\mathbf{x_2},y_2),...,(\mathbf{x_m},y_m)\\}$; attribute set $A=\\{a_1,a_2,...,a_d\\}$

Output: a decision tree with node as the root node

Simple process:

Using some splitting criterion, the optimal feature is selected to split the samples, and this is done recursively, finally yielding the branch nodes of the decision tree

A Scikit-learn Cheat Sheet Collection

Sun, 26 Aug 2018 00:00:00 +0000

This time I am putting together the commonly used scikit-learn packages as a memo

K-nearest neighbors:

from sklearn.neighbors import KNeighborsClassifier

Linear regression:

from sklearn.linear_model import LinearRegression

Logistic regression:

from sklearn.linear_model import LogisticRgression

Decision tree:

from sklearn.tree import DecisionTreeClassifier

Naive Bayes:

Gaussian distribution: from sklearn.neive_bayes import GaussianNB
Multinomial distribution: from sklearn.neive_baybes import MultinomnalNB
Bernoulli distribution: from sklearn.neive_baybes import BernoulliNB

Support vector machine:

Classification: from sklearn.svm import SVC
Regression: from sklearn.svm import SVR

Ensemble algorithms (sklearn.ensemble):

bagging:

Classification: from sklearn.ensemble import BaggingClassifier

Bayesian Classification

Sun, 19 Aug 2018 00:00:00 +0000

Bayesian decision theory considers how to choose the optimal class label based on the relevant probabilities and misclassification losses

Suppose there are N possible class labels, that is $\mathbf{y}=\\{c_1,c_2,...,c_N\\}$, and $\lambda_{ij}$ is the loss incurred by misclassifying a sample whose true label is $c_j$ as $c_i$. Based on the posterior probability $P(c_i\vert\mathbf{x})$, we can obtain the expected loss incurred by classifying sample $\mathbf{x}$ as $c_i$, that is, the “conditional risk” on sample $\mathbf{x}$:

SVM Support Vector Machine

Thu, 16 Aug 2018 00:00:00 +0000

SVM is a very important model in machine learning, proposed by Vladimir Vapnik, the father of statistical learning theory

This post takes binary classification with SVM as the example

Derivation

Given the training dataset $D=\\{(\mathbf{x_1},y_1),(\mathbf{x_2},y_2),...,(\mathbf{x_m},y_m)\\}, y_i \in \\{-1, +1\\}$, the most basic idea of a classifier is to find in the sample space a dividing space based on the training set D that separates the samples of different classes. The dividing hyperplane can be described by the following linear equation:

Linear Regression and Logistic Regression

Mon, 13 Aug 2018 00:00:00 +0000

I have not updated the blog for a long time, having paused for about half a year, and I have also read half a year’s worth of books on machine learning and deep learning; now it is time to write some output

Linear regression

Given a dataset $D=\\{(\mathbf{x_1},y_1),(\mathbf{x_2},y_2),...,(\mathbf{x_m},y_m)\\}$, where $\mathbf{x_i}=(x_{i1};x_{i2};...;x_{id}), \ y_{i} \in R$. Linear regression tries to learn a linear model

$$f(x_i)=wx_i+b$$

such that $f(x_i)\simeq y_i$.

Derivation

Consider using MSE (mean squared error) to measure how good the model is; then the loss function $L(w,b)$ is:

Detecting Traffic Violations with Opencv

Sun, 05 Nov 2017 00:00:00 +0000

Opencv contains more than 2500 functions, and almost any mature algorithm you can think of can be implemented by calling Opencv’s functions, which is super convenient. This post combines Python and Opencv to detect cars in surveillance video and check whether they commit a violation (crossing the double yellow line).

System environment: Ubuntu16.04LTS+OpenCV3.3.0+Python2.7.12

Background extraction

For surveillance the position is relatively fixed, so the background will not change considerably over a long period of time. We can therefore consider using background subtraction to roughly extract the background.