Shulou(Shulou.com)06/01 Report--

This article is about how pytorch forbids / allows the calculation of local gradients. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

The calculation of local gradients is prohibited

Torch.autogard.no_grad: disables the context manager for gradient calculation.

Setting the prohibition of calculating gradients reduces memory consumption when it is determined that Tensor.backward () will not be called to calculate gradients. If you need to calculate the gradient setting Tensor.requires_grad=True

There are two ways to disable:

Put the variables that do not need to calculate the gradient in with torch.no_grad ()

> x = torch.tensor ([1.], requires_grad=True) > with torch.no_grad ():. Y = x * 2 > > y.requires_gradOut [12]: False

Functions decorated with the decorator @ torch.no_gard () are not allowed to calculate gradients when called

@ torch.no_grad (). Def doubler (x):... Return x * 2 > z = doubler (x) > z.requires_gradOut [13]: false II. Local gradient is allowed to be calculated after prohibition.

Torch.autogard.enable_grad: context manager that allows gradient to be calculated

Enable gradient calculation in a no_grad context. This context manager has no effect outside of no_grad.

The usage is similar to the above:

Use with torch.enable_grad () to allow the gradient to be calculated

> x = torch.tensor ([1.], requires_grad=True) > with torch.no_grad ():. With torch.enable_grad ():... Y = x * 2 > y.requires_gradOut [14]: True > y.backward () # calculate gradient > x.gradOut [15]: tensor ([2.])

When it is forbidden to calculate the gradient, the function that is allowed to calculate the gradient can be used to calculate the gradient.

@ torch.enable_grad (). Def doubler (x):... Return x * 2 > > with torch.no_grad ():... Z = doubler (x) > > z.requires_grad Out [16]: True 3. Whether to calculate the gradient torch.autograd.set_grad_enable ()

Can be used as a function:

> x = torch.tensor ([1.], requires_grad=True) > is_train = False > with torch.set_grad_enabled (is_train):. Y = x * 2 > y.requires_gradOut [17]: False > torch.set_grad_enabled (True) > y = x * 2 > y.requires_gradOut [18]: True > torch.set_grad_enabled (False) > y = x * 2 > y.requires_gradOut [19]: False summary:

There is nothing to use these three functions alone, but if they are nested, follow the principle of proximity.

X = torch.tensor ([1.], requires_grad=True) with torch.enable_grad (): torch.set_grad_enabled (False) y = x * 2 print (y.requires_grad) Out [20]: False torch.set_grad_enabled (True) with torch.no_grad (): Z = x * 2 print (z.requires_grad) Out [21]: False

Add: gradient calculation is disabled locally in pytorch, and examples of no_grad, enable_grad and set_grad_enabled are used.

The original and translated Locally disabling gradient computation turn off (disable) the calculation of gradients in the local area "context managers torch.no_grad (), torch.enable_grad (), and torch.set_grad_enabled () are helpful for locally disabling and enabling gradient computation." See Locally disabling gradient computation for more details on their usage. These context managers are thread local, so they won't work if you send work to another thread using the threading module, etc. Context managers torch.no_grad (), torch.enable_grad (), and torch.set_grad_enabled () can be used to enable or disable gradient computation locally. The use of locally disabled gradient calculation is introduced in detail in the Locally disabling gradient computation chapter. These context managers are thread localized, so if you use the threading module to send the workload to another thread These context managers will not work. No _ grad Context-manager that disabled gradient calculation.no_grad is used to disable the context manager for gradient calculation. Enable _ grad Context-manager that enables gradient calculation.enable_grad is used to enable the context manager for gradient calculation. Set _ grad_enabled Context-manager that sets gradient calculation to on or off.set_grad_enabled is used to set the context manager for gradient calculation on or off state. Example 1Microsoft Windows [version 10.0.18363.1440] (c) 2019 Microsoft Corporation. All rights reserved. C:Userschenxuqi > conda activate pytorch_1.7.1_cu102 (pytorch_1.7.1_cu102) C:Userschenxuqi > pythonPython 3.7.9 (default, Aug 31 2020, 17:10:11) [MSC v.1916 64 bit (AMD64)]:: Anaconda, Inc. On win32Type "help", "copyright", "credits" or "license" for more information. > import torch > > torch.manual_seed (seed=20200910) > > a = torch.randn > > atensor ([0.2824,-0.3715, 0.9088) -1.7601], [- 0.1806, 2.0937, 1.0406,-1.7651], [1.1216, 0.8440, 0.1783, 0.6859], requires_grad=True) > > b = a * 2 > > btensor ([[0.5648,-0.7430,1.8176,-3.5202], [- 0.3612,4.1874,2.0812,-3.5303], [2.2433] 1.6879, 0.3567, 1.3718], grad_fn=) > > b.requires_gradTrue > b.grad__main__:1: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its. Grad attribute won't be populated during autograd.backward (). If you indeed want the gradient for a non-leaf Tensor, use. Retain _ grad () on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. > > print (b.grad) None > > a.requires_gradTrue > > a.grad > print (a.grad) None > with torch.no_grad ():. C = a * 2. > > ctensor ([[0.5648,-0.7430, 1.8176,-3.5202], [- 0.3612, 4.1874, 2.0812,-3.5303], [2.2433, 1.6879, 0.3567] 1.3718]) > c.requires_gradFalse > print (c.grad) None > > a.grad > print (a.grad) None > c.sum () tensor (6.1559) > c.sum (). Backward () Traceback (most recent call last): File ", line 1, in File" D:Anaconda3envspytorch_1.7.1_cu102libsite-packages orch ensor.py ", line 221, in backward torch.autograd.backward (self, gradient, retain_graph) Create_graph) File "D:Anaconda3envspytorch_1.7.1_cu102libsite-packages orchautograd\ _ init__.py", line 132, in backward allow_unreachable=True) # allow_unreachable flagRuntimeError: element 0 of tensors does not require grad and does not have a grad_fn > b.sum () tensor (6.1559, grad_fn=) > b.sum (). Backward () > a.gradtensor ([[2.2,2.2,2.2,2.] [2.2,2.2.2.2.]) > a.requires_gradTrue > example 2Microsoft Windows [version 10.0.18363.1440] (c) 2019 Microsoft Corporation. All rights reserved. C:Userschenxuqi > conda activate pytorch_1.7.1_cu102 (pytorch_1.7.1_cu102) C:Userschenxuqi > pythonPython 3.7.9 (default, Aug 31 2020, 17:10:11) [MSC v.1916 64 bit (AMD64)]:: Anaconda, Inc. On win32Type "help", "copyright", "credits" or "license" for more information. > import torch > > torch.manual_seed (seed=20200910) > a = torch.randn > > atensor ([0.2824,-0.3715, 0.9088) -1.7601], [- 0.1806, 2.0937, 1.0406,-1.7651], [1.1216, 0.8440, 0.1783, 0.6859], requires_grad=True) > > a.requires_gradTrue > with torch.set_grad_enabled (False):. B = a * 2... > > btensor ([[0.5648,-0.7430, 1.8176,-3.5202], [- 0.3612, 4.1874, 2.0812,-3.5303], [2.2433, 1.6879, 0.3567, 1.3718]]) > > b.requires_gradFalse > with torch.set_grad_enabled (True):. C = a * 3. > > ctensor ([[0.8472,-1.1145, 2.7263,-5.2804], [- 0.5418, 6.2810, 3.1219,-5.2954], [3.3649, 2.5319, 0.5350, 2.0576]] Grad_fn=) > > c.requires_gradTrue > d = a * 4 > d.requires_gradTrue > torch.set_grad_enabled (True) # this can also be used as a function > # use > e = a * 5 > etensor ([[1.4119,-1.8574, 4.5439,-8.8006], [- 0.9030, 10.4684, 5.2031,-8.8257], [5.6082, 4.2198, 0.8917] 3.4294], grad_fn=) > > e.requires_gradTrue > dtensor ([[1.1296,-1.4859, 3.6351,-7.0405], [- 0.7224, 8.3747, 4.1625,-7.0606], [4.4866, 3.3759,0.7133, 2.7435]] Grad_fn=) > torch.set_grad_enabled (False) # use > f = a * 6 > ftensor ([[1.6943,-2.2289, 5.4527,-10.5607], [- 1.0836, 12.5621, 6.2437,-10.5908], [6.7298, 5.0638,1.0700,4.1153]) > > f.requires_gradFalse > Thank you for reading! This is the end of this article on "how pytorch forbids / allows the operation of calculating local gradients". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it out for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.