2018 on Simplestory

Deep Learning Optimization

Sat, 03 Nov 2018 00:00:00 +0000

Optimization is a very difficult problem; usually one has to carefully design an objective function and related constraints, and also make sure the problem to be optimized is a convex function, but most real-world problems are non-convex. For non-convex optimization problems there are some basic optimization methods: SGD, Momentum, AdaGrad, RMSProp, Adam and so on

Stochastic Gradient Descent (SGD)

// stochastic gradient descent

Given the dataset $X=\\{x^{(1)},x^{(2)},...,x^{(n)}\\}$ and the dataset labels $Y=\\{y^{(1)}, y^{(2)},...,y^{(n)}\\}$

How the FFM Model Works

Sun, 28 Oct 2018 00:00:00 +0000

CTR (click-through rate) prediction plays an important role in content recommendation and delivery, and the FFM model is good at handling this kind of data. The FFM model is a variant of the FM model, or rather the FM model is a special case of the FFM model

Rough pipeline

Convert the dataset format (using the LIBSVM or FFMs style)
Compute the loss function (using the exponential loss function)
Optimize the parameters (using the AdaGrad + SG strategy)

Converting the data format

LIBSVM format (does not store zero-valued features):

Decision Tree

Wed, 29 Aug 2018 00:00:00 +0000

Decision tree, a common machine learning method, is somewhat like a flowchart. A sample is divided into a set according to the value it takes on some feature, and this is done recursively

Basic algorithm

Input: training set $D=\\{(\mathbf{x_1},y_1),(\mathbf{x_2},y_2),...,(\mathbf{x_m},y_m)\\}$; attribute set $A=\\{a_1,a_2,...,a_d\\}$

Output: a decision tree with node as the root node

Simple process:

Using some splitting criterion, the optimal feature is selected to split the samples, and this is done recursively, finally yielding the branch nodes of the decision tree

A Scikit-learn Cheat Sheet Collection

Sun, 26 Aug 2018 00:00:00 +0000

This time I am putting together the commonly used scikit-learn packages as a memo

K-nearest neighbors:

from sklearn.neighbors import KNeighborsClassifier

Linear regression:

from sklearn.linear_model import LinearRegression

Logistic regression:

from sklearn.linear_model import LogisticRgression

Decision tree:

from sklearn.tree import DecisionTreeClassifier

Naive Bayes:

Gaussian distribution: from sklearn.neive_bayes import GaussianNB
Multinomial distribution: from sklearn.neive_baybes import MultinomnalNB
Bernoulli distribution: from sklearn.neive_baybes import BernoulliNB

Support vector machine:

Classification: from sklearn.svm import SVC
Regression: from sklearn.svm import SVR

Ensemble algorithms (sklearn.ensemble):

bagging:

Classification: from sklearn.ensemble import BaggingClassifier

Bayesian Classification

Sun, 19 Aug 2018 00:00:00 +0000

Bayesian decision theory considers how to choose the optimal class label based on the relevant probabilities and misclassification losses

Suppose there are N possible class labels, that is $\mathbf{y}=\\{c_1,c_2,...,c_N\\}$, and $\lambda_{ij}$ is the loss incurred by misclassifying a sample whose true label is $c_j$ as $c_i$. Based on the posterior probability $P(c_i\vert\mathbf{x})$, we can obtain the expected loss incurred by classifying sample $\mathbf{x}$ as $c_i$, that is, the “conditional risk” on sample $\mathbf{x}$:

SVM Support Vector Machine

Thu, 16 Aug 2018 00:00:00 +0000

SVM is a very important model in machine learning, proposed by Vladimir Vapnik, the father of statistical learning theory

This post takes binary classification with SVM as the example

Derivation

Given the training dataset $D=\\{(\mathbf{x_1},y_1),(\mathbf{x_2},y_2),...,(\mathbf{x_m},y_m)\\}, y_i \in \\{-1, +1\\}$, the most basic idea of a classifier is to find in the sample space a dividing space based on the training set D that separates the samples of different classes. The dividing hyperplane can be described by the following linear equation:

Linear Regression and Logistic Regression

Mon, 13 Aug 2018 00:00:00 +0000

I have not updated the blog for a long time, having paused for about half a year, and I have also read half a year’s worth of books on machine learning and deep learning; now it is time to write some output

Linear regression

Given a dataset $D=\\{(\mathbf{x_1},y_1),(\mathbf{x_2},y_2),...,(\mathbf{x_m},y_m)\\}$, where $\mathbf{x_i}=(x_{i1};x_{i2};...;x_{id}), \ y_{i} \in R$. Linear regression tries to learn a linear model

$$f(x_i)=wx_i+b$$

such that $f(x_i)\simeq y_i$.

Derivation

Consider using MSE (mean squared error) to measure how good the model is; then the loss function $L(w,b)$ is:

Using g++ and gdb

Fri, 02 Mar 2018 00:00:00 +0000

G++ is the C++ compiler under Unix/Linux, and gdb is a powerful debugging tool under Unix/Linux. Most of the time, the C/C++ code we write does not run on Windows but on a Linux system. Here I record some usage of g++ and gdb in case they are needed.

Using g++

When g++ performs compilation work, it needs 4 steps in total:

Preprocessing, generating a .i file, mainly expanding header files and macro definitions: g++ -E test.cpp -o test.i
Compilation, converting the preprocessed file into assembly language and generating a .s file: g++ -S test.i -o test.s
Assembly, turning the assembly into object code (machine code) and generating a .o file, mainly converting the .s file into a binary file: g++ -c test.s -o test.o
Linking, linking the object code to generate an executable program: g++ test.o -o test

The steps above can also be done with a single command:

Configuring Django on Ubuntu

Mon, 19 Feb 2018 00:00:00 +0000

At the time of the 2018 Spring Festival, heaven knows what I went through, posting a blog article on the fourth day of the new year. Django was actually configured before the New Year; I had just learned some knowledge about developing web applications with python at that time, so I configured it along the way.

Environment used for the configuration

System: Ubuntu16.04 (LTS)
python version: python3.6 (using Anaconda to create a virtual environment)

Preparation

Uninstalling an old version of Django

If you are upgrading from an older version of Django, you need to deal with the old version a little.

Configuring Tensorflow on Linux

Sun, 04 Feb 2018 00:00:00 +0000

Borrowing a line from the TensorFlow Chinese community: TensorFlow, an open-source magic tool for artificial intelligence.

Environment used for the configuration

System: Ubuntu16.04 (LTS)
python version: python3.6 (using Anaconda to create a virtual environment)
Nvidia card: Geforce 920M (if there is no Nvidia card, only the CPU version can be installed)

Installing Tensorflow

tensorflow has two versions, CPU and GPU; I created two python virtual environments and installed these two versions separately. Below I record this painful installation process

Linux Penetration Testing Commands

Sat, 20 Jan 2018 00:00:00 +0000

Below is a penetration testing memo for Linux machines; I came across it on the internet by chance and copied it over here.

Penetration testing a Linux machine

Command	Description
netstat -tulpn	Show network ports in Linux together with the corresponding ID (PID)
watch ss -stplu	Observe TCP and UDP ports in real time through sockets
lsof -i	Show established connections
macchanger -m MACADDR INTR	Change the MAC address on KALI Linux
ifconfig eth0 192.168.2.1/24	Set the IP address in Linux
ifconfig eth0:192.168.2.3/24	Add an IP address to an existing network interface in Linux
ifconfig eth0 hw ether MAC ADDR	Change the address in Linux using ifconfig
ifconfig eth0 mtu 1500	Change the MTU size with ifconfig; change 1500 to the MTU you want
dig -x 192.168.1.1	Perform a reverse lookup on an IP address
host 192.168.1.1	Perform a reverse lookup on an IP address; useful when dig is not installed
dig @192.168.2.2 domain.com -t AXFR	Perform a DNS zone transfer with dig
host -l domain.com nameserver	Perform a DNS zone transfer with host
nbtstat -A x.x.x.x	Obtain the domain name corresponding to an IP address
ip addr add 192.168.2.22/24 dev eth0	Add a hidden IP address that does not show up when running ifconfig
tcpkill -9 host goole.com	Block access to goole.com from the host
echo "1">/proc/sys/net/ipv4/ip_forward	Enable IP forwarding, turning the Linux box into a router — which makes it convenient to control the traffic routed through this box
echo "8.8.8.8">/etc/resolv.conf	Use Goole's DNS

System information commands

Very helpful for local enumeration