Shulou(Shulou.com)06/01 Report--

This article mainly introduces the example analysis of tf.GradientTape gradient solving tool, which is very detailed and has certain reference value. Friends who are interested must finish it!

Tf.GradientTape is defined in the tensorflow/python/eager/backprop.py file, and it can be roughly seen from the file path that GradientTape is used to calculate gradients in eager mode, while eager mode (see the link at the end of the article for a specific introduction to eager mode) is the default mode of TensorFlow 2.0, so tf.GradientTape is the official recommended usage. The following is a specific introduction to the principle and use of GradientTape.

Tape in English is tape, the meaning of tape, which is used here because of the influence of eager mode. In the era of TensorFlow 1.x static diagrams, we know that every static graph has two parts, one is a forward graph and the other is a reverse graph. The reverse graph is used to calculate the gradient and is used in the whole training process. TensorFlow 2.0 defaults to eager mode, where each line of code is executed sequentially, without the process of building the diagram (and eliminating the use of control_dependency). But you can't calculate the gradient on every line, can you? The amount of calculation is too large and it is not necessary. Therefore, a context manager (context manager) is needed to connect functions and variables that need to calculate gradients to facilitate solution and improve efficiency.

For example: calculate the derivative of y = x ^ 2 at x = 3:

X = tf.constant (3.0)

With tf.GradientTape () as g:

G.watch (x)

Y = x * x

# y'= 2cm x = 2m 3 = 6

Dy_dx = g.gradient (y, x)

The watch function in the example adds the variable x that needs to calculate the gradient. By default, GradientTape only monitors variables for the traiable=True attribute (default) created by tf.Variable. The x in the above example is constant, so calculating the gradient requires the addition of the g.watch (x) function. Of course, you can also set not automatically monitor trainable variables, completely specified by yourself, set watch_accessed_variables=False on the line (generally not needed).

GradientTape can also nest multiple layers to calculate higher-order derivatives, such as:

X = tf.constant (3.0)

With tf.GradientTape () as g:

G.watch (x)

With tf.GradientTape () as gg:

Gg.watch (x)

Y = x * x

# y'= 2cm x = 2m 3 = 6

Dy_dx = gg.gradient (y, x)

# yearly'= 2

D2y_dx2 = g.gradient (dy_dx, x)

In addition, by default, the resources of GradientTape are released after calling the gradient function, and cannot be calculated if called again. So if you need to calculate the gradient multiple times, you need to turn on the persistent=True attribute, for example:

X = tf.constant (3.0)

With tf.GradientTape (persistent=True) as g:

G.watch (x)

Y = x * x

Z = y * y

# z = y ^ 2 = x ^ 4, z' = 4* x ^ 3 = 4* 3 ^ 3

Dz_dx = g.gradient (z, x)

# y'= 2cm x = 2m 3 = 6

Dy_dx = g.gradient (y, x)

Del g # remove this contextual tape

Finally, when generally used in a network, there is no need to explicitly call the watch function. Using the default settings, GradientTape monitors trainable variables, such as:

With tf.GradientTape () as tape:

Predictions = model (images)

Loss = loss_object (labels, predictions)

Gradients = tape.gradient (loss, model.trainable_variables)

In this way, the gradients of all trainable variables can be calculated and then updated in the next step. For TensorFlow 2.0, it is recommended that you use this method to calculate gradients, and you can view the specific gradient values in eager mode.

The above is all the contents of the article "example Analysis of tf.GradientTape gradient solving tool". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.