Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to solve the problem that the weight in API can not be updated due to the use of custom layer in trainable_variables. The content of the article is of high quality, so the editor will share it for you as a reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

Gupta, an engineer who works in NLP, discovered a serious bug that exists in TensorFlow:

Everyone who uses Keras functional API in the custom layer should pay attention! Weights created with Keras's Functional API may be lost.

This topic is hotly discussed on the Reddit machine learning section, which resonates with many TensorFlow users.

Specifically, using a custom layer in API will cause the weights in trainable_variables to not be updated. And these weights are not put into the non_trainable_variables.

In other words, the weight that originally needed training is now frozen.

To the dissatisfaction of the engineer, he reported the bug to Google in GitHub about a month ago, and Google hasn't officially fixed it yet.

Solution.

How do you check your code to see if there are similar problems? Call model.trainable_variables to test your model:

For I, var in enumerate (model.trainable_variables):

Print (model.trainable_ variables [I] .name)

Check to see if all your variable weights are correct. If the weights are missing or haven't changed, you've been hit, too.

The bug of Gupta's own model created by the Transformer library has been reproduced in the Colab notebook, which can be watched by interested readers.

Https://colab.research.google.com/gist/Santosh-Gupta/40c54e5b76e3f522fa78da6a248b6826/missingtrainablevarsinference_var.ipynb

One solution to this problem given by Gupta is to create a model using Keras subclasses instead. When you switch to this method, all weights will appear in trainable_variables.

To absolutely ensure that the models created with functional API and subclass methods are exactly the same, Gupta reasoned them at the bottom of each Colab notebook with the same input, and the output of the model is exactly the same.

However, training with a functional API model treats many weights as frozen, and these weights do not appear in the non_trainable_variables, so they cannot be unfrozen.

To check whether Google recently fixed the vulnerability, Gupta also installed a Nightly version of TF 2.3.0-rc1 to keep the framework up-to-date, but bug still exists.

On the use of custom layers in API caused by the weight in trainable_variables can not be updated how to share here, I hope the above content can be of some help to you, you can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.