The previous post mentioned an efficient tree ensemble algorithm, the Xgboost model, but its efficiency is still low when dealing with data that has a large feature dimension and a large number of samples. Compared with models such as GBDT and Xgboost, the Lightgbm model open-sourced by Microsoft uses a variety of techniques to reduce the amount of data and the number of features, which greatly speeds up training while keeping accuracy. ...
Today it occurred to me that I have not updated my blog for a long time; lately I have been busy with written tests and interviews, and with my graduation project. Now that things have come to a pause today, I might as well organize the Xgboost model I reviewed a few days ago and post an article. Tree ensemble Ensemble algorithms are a broad and very important class of algorithms in machine learning, generally divided into Bagging and Boosting. Ensembling decision trees is of course a customary technique. For tree ensembles, the representative of the Bagging method is Random Forest, which randomly groups attributes and data for decision tree learning and finally aggregates the results; the representative of the Boosting method is AdaBoost and GBDT (gradient boosted decision trees). These algorithms are all quite important and cannot be summed up in a sentence or two, so I will not expand on them. Within tree Boosting, Tianqi Chen proposed a new model, Xgboost. This model makes use of the second derivative of the objective function and also adds a regularization term to prevent the model from overfitting. Below is an interpretation of the Xgboost model. ...
CTR (click-through rate) prediction plays an important role in content recommendation and delivery, and the FFM model is good at handling this kind of data. The FFM model is a variant of the FM model, or rather the FM model is a special case of the FFM model Rough pipeline Convert the dataset format (using the LIBSVM or FFMs style) Compute the loss function (using the exponential loss function) Optimize the parameters (using the AdaGrad + SG strategy) Converting the data format LIBSVM format (does not store zero-valued features): ...
Decision tree, a common machine learning method, is somewhat like a flowchart. A sample is divided into a set according to the value it takes on some feature, and this is done recursively Basic algorithm Input: training set $D=\\{(\mathbf{x_1},y_1),(\mathbf{x_2},y_2),...,(\mathbf{x_m},y_m)\\}$; attribute set $A=\\{a_1,a_2,...,a_d\\}$ Output: a decision tree with node as the root node Simple process: Using some splitting criterion, the optimal feature is selected to split the samples, and this is done recursively, finally yielding the branch nodes of the decision tree ...