Shulou(Shulou.com)06/01 Report--

Today, I would like to share with you the difference between Pytorch model.train and model.eval and how to use the relevant knowledge points, the content is detailed, the logic is clear, I believe most people still know too much about this knowledge, so share this article for your reference, I hope you can get something after reading this article, let's take a look at it.

When training the model, it will be preceded by: model.train () used in front of the test model: model.eval ()

At the same time, it is found that the two programs can be run without writing them, because the two methods are aimed at different ways of network training and testing, such as Batch Normalization and Dropout.

Training is for each min-batch, but in tests it is often for a single picture, that is, there is no concept of min-batch.

Because the parameters are fixed after the network training, so the mean and variance of each batch are constant, so the mean and variance of all batch are settled directly.

All Batch Normalization training and testing operations are different.

In the training, the neurons in each hidden layer multiply the probability P first, and then activate them. In the test, all the neurons are activated first, and then the output of each hidden layer neuron is multiplied by P.

Add: Pytorch trampling record-model.eval ()

Recently, I encountered a problem when I was writing code. The originally trained model was loaded into the inference and the accuracy rate dropped directly by 5 points. Nima, this is simply unbearable. The chicken subconsciously sensed that I must have written bug somewhere. ~ ~ so I began to search everywhere, from model load to data load, and finally found the problem in the corner of a sealed module, so by the way, I summarized it here. Avoid doing it again.

There are two common reasons why the loading accuracy of the trained model is not consistent with the original one.

1) data

2) model.state_dict ()

1) data

On the data side, check whether the data loaded before and after the two times has changed. First of all, check whether the mean and variance used by transforms.Normalize are the same as those used during training; in addition, check whether the data has been changed in the form of storage in this process, which may lead to the change of data accuracy and lead to the loss of certain information.

For example, one of the datasets I used originally stored the image as a vector, but it corresponds to the data in "png" format (the corresponding description was later found in the original file. ), and I did a data-to-img operation to convert the vector to "jpg" form, and the loading caused the drop point.

2) model.state_dict ()

The drop point caused by the first aspect is generally not too serious, and the drop point caused by the second aspect is more serious. Once the parameters of the model are loaded wrong, the error is large.

It is easier to find if the parameters are not loaded correctly, and the accuracy is very low, which is almost tantamount to a wild guess.

The situation I encountered this time is that the accuracy is not particularly low. Only a few points have been dropped and checked many times, all of which show that the model parameters have been loaded successfully. Later, after a closer look, it was found that when one of the calls to the model for inference, forgot to write 'model.eval ()', resulting in a change in the parameters of the model, and a drop occurred when called again.

So I reviewed the specific role of model.eval () and model.train (). As follows:

Model.train () and model.eval () usually add these two sentences in model training and evaluation, mainly because model Batch during training and evaluation

The Normalization and Dropout method modes are different:

A) model.eval (), BatchNormalization and Dropout are not enabled. At this point, pytorch will automatically fix the BN and DropOut, not taking the average, but using the trained value. Otherwise, once the batch_size of test is too small, it is easy to lose a lot of model performance due to the BN layer.

B) model.train (): enable BatchNormalization and Dropout. In the model testing phase, use model.train () to turn model into training mode, when the operation of dropout and batch normalization plays a role in training Q to prevent the network from overfitting.

Therefore, it is important to remember to specify the train/eval of the instantiated model when using PyTorch for training and testing.

Model.eval () vs torch.no_grad ()

Although both are used for eval, they do not serve the same purpose:

Model.eval () is responsible for changing the way batchnorm and dropout work. For example, in eval () mode, dropout does not work. See the code below:

Import torch import torch.nn as nn drop = nn.Dropout () x = torch.ones (10) # Train mode drop.train () print (drop (x)) # tensor ([2.2,2.0.,2.2,2.2.2.0.0.0.2.]) # Eval mode drop.eval () print (drop (x)) # tensor ([1.1,1.1.1.1.1.1.1. 1.])

Torch.no_grad () is responsible for turning off the gradient calculation to save eval time.

When only doing inference, model.eval () must be used, otherwise it will affect the accuracy of the results. While torch.no_grad () is not mandatory, it only affects the efficiency of the operation.

These are all the contents of this article entitled "the difference between Pytorch model.train and model.eval and how to use it". Thank you for reading! I believe you will gain a lot after reading this article. The editor will update different knowledge for you every day. If you want to learn more knowledge, please pay attention to the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.