Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to compare pytorch's ReLU and custom class GuidedBackpropReLU, the content of the article is of high quality, so the editor will share it with you for reference. I hope you will have some understanding of the relevant knowledge after reading this article.

Conclusion: the difference between GuidedBackpropReLU and ReLU is obvious. In back propagation, only the positive gradient received from the upper layer is propagated, and the negative gradient is set to zero directly. On the other hand, ReLU accepts the gradient of the upper layer, regardless of whether the gradient is positive or negative.

The experimental code is shown (in the experiment, on lines 58 and 59, setting coefficient to-1 and + 1 will have different effects):

Import torchfrom torch.autograd import Functionclass GuidedBackpropReLU (Function):''special ReLU, the difference is that only inputs greater than zero and gradients greater than zero are considered in backpropagation' @ staticmethod# def forward (ctx, input_img): # torch.Size ([1,64,112,112]) # positive_mask = (input_img > 0). Type_as (input_img) # torch.Size ([1,64,112) Output = torch.addcmul (torch.zeros (input_img.size ()). Type_as (input_img), input_img, positive_mask) # output = input_img * positive_mask # this line of code has the same function as the previous line # ctx.save_for_backward (input_img, output) # return output # torch.Size ([1,64,112) Staticmethoddef forward (ctx, input_img): # torch.Size ([1,64,112,112]) output = torch.clamp (input_img, min=0.0) # print ('input tensor in the function requires_grad',input_img.requires_grad) ctx.save_for_backward (input_img, output) return output # torch.Size ([1,64,112,112]) @ staticmethoddef backward (ctx) Grad_output): # torch.Size ([1, 2048, 7, 7]) input_img, output = ctx.saved_tensors # torch.Size ([1, 2048, 7]) torch.Size ([1, 2048, 7]) # grad_input = None # this line of code does not work positive_mask_1 = (input_img > 0). Type_as (grad_output) # torch.Size ([1, 2048, 7) 7]) the input feature is greater than zero positive _ mask_2 = (grad_output > 0). Type_as (grad_output) # torch.Size ([1, 2048, 7]) gradient is greater than zero # grad_input = torch.addcmul (# torch.zeros (input_img.size ()). Type_as (input_img) # torch.addcmul (# torch.zeros (input_img.size ()). Type_as (input_img), # grad_output # positive_mask_1#) # positive_mask_2#) grad_input = grad_output * positive_mask_1 * positive_mask_2# the function of this line of code is the same as the previous line of code return grad_inputtorch.manual_seed (seed=20200910) size = (3p5) input_data_1 = input = torch.randn (* size, requires_grad=True) torch.manual_seed (seed=20200910) input_data_2 = input = torch.randn (* size Requires_grad=True) torch.manual_seed (seed=20200910) input_data_3 = input = torch.randn (* size, requires_grad=True) print ('the dimensions of the three input data are:', input_data_1.shape, input_data_2.shape Input_data_3.shape) # print (input_data_1) # print (input_data_2) # print (input_data_3) coefficient =-1.The coefficient = 1.0loss_1 = coefficient * torch.sum (torch.nn.ReLU () (input_data_1)) loss_2 = coefficient * torch.sum (torch.nn.functional.relu (input_data_2)) loss_3 = coefficient * torch.sum (GuidedBackpropReLU.apply (input_data_3) loss_1 Print () loss_2.backward () loss_3.backward () print (loss_1 Loss_2, loss_3) print (loss_1.item (), loss_2.item (), loss_3.item ()) print ('whether the three loss values are equal' Loss_1.item () = loss_2.item () = loss_3.item () print ('print three gradient information briefly...') print (input_data_1.grad) print (input_data_2.grad) print (input_data_3.grad) print ('the dimensions of these three gradients are:', input_data_1.grad.shape, input_data_2.grad.shape. Input_data_3.grad.shape) print ('check whether the three gradients are equal...') print (torch.equal (input_data_1.grad, input_data_2.grad)) print (torch.equal (input_data_1.grad, input_data_3.grad)) print (torch.equal (input_data_2.grad, input_data_3.grad))

Console output (# 58 coefficient =-1.0):

Copyright (C) Windows PowerShell Microsoft Corporation. All rights reserved. It took 842ms to try a new cross-platform PowerShell https://aka.ms/pscore6 to load personal and system profiles. (base) PS C:\ Users\ chenxuqi\ Desktop\ News4cxq\ test4cxq > conda activate ssd4pytorch2_2_0 (ssd4pytorch2_2_0) PS C:\ Users\ chenxuqi\ Desktop\ News4cxq\ test4cxq > &'D:\ Anaconda3\ envs\ ssd4pytorch2_2_0\ python.exe''c:\ Users\ chenxuqi\ .vscode\ extensions\ ms-python.python-2021.1.502429796\ pythonFiles\ lib\ python\ debugpy\ launcher' '62123' -'c:\ Users\ chenxuqi\ Desktop\ The dimensions of the three input data News4cxq\ test4cxq\ testReLU.py' are: torch.Size ([3] 5]) torch.Size ([3,5]) torch.Size ([3,5]) tensor (- 7.1553, grad_fn=)-7.155285835266113-7.155285835266113-7.155285835266113-7.155285835266113 True briefly prints three gradient information. Tensor ([- 1.0.0.0.0.0.], [- 1. -1., 0.,-1.], [- 1,-1, 0, 0.]) tensor ([- 1, 0,-1, 0.], [- 1,-1, 0,-1.], [- 1,-1, 0, 0. 0.]) tensor ([[- 0,-0,-0,-0.], [- 0,-0,-0,-0.], [- 0,-0,-0,-0,-0]) The dimensions of the three gradients are: torch.Size ([3,5]) check whether the three gradients are equal... TrueFalseFalse (ssd4pytorch2_2_0) PS C:\ Users\ chenxuqi\ Desktop\ News4cxq\ test4cxq >

Console output (# 59 coefficient = 1.0):

Copyright (C) Windows PowerShell Microsoft Corporation. All rights reserved. It took 846ms to try a new cross-platform PowerShell https://aka.ms/pscore6 to load personal and system profiles. (base) PS C:\ Users\ chenxuqi\ Desktop\ News4cxq\ test4cxq > &'D:\ Anaconda3\ envs\ ssd4pytorch2_2_0\ python.exe''c:\ Users\ chenxuqi\ .vscode\ extensions\ ms-python.python-2021.1.502429796\ pythonFiles\ lib\ python\ debugpy\ launcher' '62135' -' c:\ Users\ chenxuqi\ Desktop\ News4cxq\ test4cxq\ testReLU.py' the dimensions of the three input data are torch.Size ([3,5]) torch.Size ([3]) 5]) torch.Size ([3,5]) tensor (7.1553, grad_fn=) 7.155285835266113 7.155285835266113 7.155285835266113 whether the three loss values are equal True briefly prints three gradient information. Tensor ([1, 0, 1, 0.], [1, 1, 0, 1.], [1, 1, 0. 0.]) tensor ([[1, 0, 1, 0, 0.], [1, 1, 0, 0, 1.], [1, 1, 0, 0.]) tensor ([[1, 0, 1, 0, 0.], [1, 1, 0, 1.], [1, 1, 0, 0.]. 0.]]) The dimensions of the three gradients are: torch.Size ([3,5]) check whether the three gradients are equal... TrueTrueTrue (base) PS C:\ Users\ chenxuqi\ Desktop\ News4cxq\ test4cxq > conda activate ssd4pytorch2_2_0 (ssd4pytorch2_2_0) PS C:\ Users\ chenxuqi\ Desktop\ News4cxq\ test4cxq > about how to compare pytorch's ReLU and custom class GuidedBackpropReLU. I hope the above content can be of some help to you and learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.