Shulou(Shulou.com)06/03 Report--

This article will explain in detail how to use the pytorch neural network Dropout in Python. The editor thinks it is very practical, so I share it for you as a reference. I hope you can get something after reading this article.

Robustness of disturbance

As we saw earlier when we discussed weight decay (L2 regularization), the norm of the parameter also represents a useful measure of simplicity. Another useful angle of simplicity is smoothness, that is, functions should not be sensitive to changes in the smile of their input. For example, when we classify images, we expect that adding some random noise to pixels should be basically unaffected.

In the process of forward propagation, dropout calculates that noise is injected into each internal layer at the same time, which has become a standard technique for training neural networks. This method is called dropout because on the surface we dropout some neurons during training. In each iteration of the entire training process, dropout includes zeroing some nodes in the current layer before calculating the next layer.

So the key challenge is how to inject this noise, one idea is to inject noise in an unbiased way. In this way, when other layers are fixed, the expected value of each layer is equal to that when there is no noise.

Dropout in practice

The previous multi-layer perceptron has a hidden layer and 5 hidden units. When we apply dropout to the hidden layer, the hidden unit is set to 00:00 with the probability of p, and the result can be regarded as a network containing only a subset of primitive neurons. In the figure, H3 and H6 are deleted. Therefore, the calculation of the output is no longer dependent on h3 or h6, and their respective gradients disappear when backpropagation is performed. In this way, the calculation of the output layer cannot be excessively dependent on H2, … , any element of H6.

Usually, we only use dropout when testing. Given a trained model and a new sample, we will not discard any nodes, so there is no need for standardization.

Concise implementation

For advanced API, all we need to do is add a Dropout layer after each full connection layer, passing the discard probability as the only parameter to its constructor. During the training, the Dropout layer will randomly discard the output of the upper layer (equivalent to the input of the next layer) according to the specified discard probability. When not in training mode, the dropout layer only passes data during testing.

Net = nn.Sequential (nn.Flatten (), nn.Linear (784,256), nn.ReLU (), # add a dropout layer nn.Dropout (dropout1) after the first fully connected layer Nn.Linear (256,256), nn.ReLU (), # add a dropout layer nn.Dropout (dropout2) after the second fully connected layer Nn.Linear (256,10) def init_weights (m): if type (m) = = nn.Linear: nn.init.normal_ (m.weight, std=0.01) net.apply (init_weights)

Next, we train and test the model.

Trainer = torch.optim.SGD (net.parameters (), lr=lr) d2l.train_ch4 (net, train_iter, test_iter, loss, num_epochs, trainer)

This is the end of this article on "how to use pytorch neural network Dropout in Python". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.