Shulou(Shulou.com)05/31 Report--

This article mainly introduces the relevant knowledge of the evaluation case analysis of the python model, the content is detailed and easy to understand, the operation is simple and fast, and has a certain reference value. I believe you will gain something after reading this python model evaluation case analysis article. Let's take a look at it.

In addition to using estimator's score function to evaluate the quality of the model simply and roughly

The sklearn.metrics module provides a variety of evaluation indicators for different types of problems and can create user-defined evaluation indicators.

The cross-validation method in the model_selection module can be used to evaluate the generalization ability of the model and effectively avoid over-fitting.

First, an overview of metrics evaluation indicators

There are two types of evaluation metrics in sklearn.metrics: those ending with _ score are some kind of score, and the bigger the better.

Ending with _ error or _ loss is some kind of deviation, as small as possible.

Commonly used classification evaluation indicators include: accuracy_score,f1_score

Precision_score,recall_score, wait.

The commonly used regression evaluation indicators include: R2 score variation score and so on.

Commonly used clustering evaluation indicators include: adjusted_rand_score,adjusted_mutual_info_score and so on.

Second, the evaluation of classification model

All the information of the model classification effect:

Confusion_matrix confusion matrix, error matrix.

The overall classification effect of the model:

Accuracy correct rate. General classification evaluation index.

The effect of the model on the classification of a category:

Precision accuracy, also known as precision. The ability of the model not to mark the positive sample incorrectly. "do not wronge a good man".

Recall recall rate, also known as recall rate. The ability of the model to identify all positive samples. "never let go of a bad guy."

F1_score F1 score. The harmonic average of accuracy and recall.

The overall classification effect is obtained by summing and averaging the classification effect in different ways:

Macro_averaged: macro average. The effect of each category of prediction is equally important.

Micro_averaged: micro-average. The effect of each classification prediction is equally important.

Weighted_averaged: weighted average. The prediction effect of each category is proportional to the frequency of occurrence of samples in that category.

Sampled_averaged: average of samples. It only applies to multi-label classification problems. The evaluation index is calculated according to the predicted value and true value of multiple labels in each sample. Then average the sample.

The evaluation method which is only applicable to the probability model and the problem is a binary classification problem:

ROC curve

Auc_score

Third, the evaluation of regression model

The most commonly used evaluation indicators of regression models are:

R2_score (r square, goodness of fit, determinable coefficient)

Explained_variance_score (explain Variance score)

Fourth, use a virtual estimator to generate a benchmark score

For supervised learning (classification and regression), the scores of some simple estimation strategies based on experience (virtual estimation) can be used as reference benchmarks.

DummyClassifier implements several simple classification strategies:

Stratified generates random predictions in terms of class distribution of training sets.

Most_frequent always predicts the most common tag in the training set.

Prior is similar to most_frequenct, but has a precit_proba method

Uniform randomly generates predictions.

Constant always predicts the constant tags provided by the user.

DummyRegressor implements four simple rules of thumb for regression:

Mean always predicts the average of training goals.

Median always predicts the median of training goals.

Quantile always predicts the quantile (quantile) of the training target provided by the user.

Constant always predicts constant values provided by the user.

Fifth, underfitting, over-fitting and cross-validation

In machine learning problems, it often occurs that the model scores high on the training data.

But in the case of poor performance on the new data, this is called overfitting overfitting, also known as high variance high variance.

If the score on the training data is very low, this is called underfitting underfitting, also known as high deviation high bias.

Set aside method

In order to solve the over-fitting problem, the common method divides the data into the training set and the test set, uses the training set to train the parameters of the model, and uses the test set to test the performance of the model after training. Sometimes for some models with hyperparameters (for example, parameters C and kernel of svm.SVC are hyperparameters), it is necessary to draw some data from the training set to verify the validity of the hyperparameters.

Cross verification method

When the amount of data is limited, dividing the data into three parts according to the retention method will seriously affect the effect of model training. In order to make effective use of limited data, cross-validation cross_validation method can be used.

The basic idea of cross-validation is to divide the data set into training set and test set many times in different ways, train and test respectively, and then synthesize the final test scores. Each data belongs to the training set in some partitions and the test set in others.

Simple 80% discount cross-validation: the data set is divided into two groups averagely: group An is used to train the group B test, and then group B is used to train the group A test, so it is called cross-validation.

The commonly used cross-validation methods are K-fold (KFold), left-one cross-validation (LeaveOneOut,LOO), P-left cross-validation (LeavePOut,LPO), repeated K-fold cross-validation (RepeatedKFold), random arrangement cross-validation (ShuffleSplit).

In addition, in order to ensure that the distribution of each label category data in the training set is consistent with that in the complete data set, the hierarchical cross-validation method (StratifiedKFold,StratifiedShuffleSplit) can be used.

When the source of the data set has different groups, the independent same distribution hypothesis (independent identical distributed:i.i.d) will be broken, and the group cross-validation method can be used to ensure that the proportion of the data of the training set from each group is consistent with the complete data set. (GroupKFold,LeaveOneGroupOut,LeavePGroupsOut,GroupShuffleSplit)

For time series data, a very important feature is the correlation (autocorrelation) between time adjacent observations, so it is very important to train with past data and test with future data. TimeSeriesSplit can achieve this kind of segmentation.

Sixth, cross-validation comprehensive score

Call the cross_val_score function to calculate the score of the model on each cross-validation dataset.

You can specify a scoring function in metrics or a cross-validation iterator.

The cross_validate function is similar to the cross_val_score function, but more powerful, it allows you to specify multiple metrics for evaluation, and in addition to returning the specified metrics, it also returns a fit_time and score_time, that is, training time and scoring time.

Using cross_val_predict, you can return the prediction results of each sample as a test set in CV by the corresponding model.

This requires that the CV strategy used can ensure that each sample has a chance to be used as test data, otherwise an exception will be reported.

This is the end of the article on "Evaluation case Analysis of python Model". Thank you for reading! I believe you all have a certain understanding of the knowledge of "case Analysis of python Model Evaluation". If you want to learn more knowledge, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.