Shulou(Shulou.com)06/03 Report--

This article mainly introduces the example analysis of TensorFlow neural network optimization strategy, the article is very detailed, has a certain reference value, interested friends must read it!

In the process of neural network model optimization, many problems will be encountered, such as how to set the learning rate, we can make the model approach the optimal solution quickly at the initial stage of training by exponential attenuation, and steadily enter the optimal solution region in the later stage of training; for the problem of over-fitting, we can deal with it by regularization; the moving average model can make the final model more robust on unknown data.

I. the setting of the learning rate

The learning rate setting should be neither too large nor too small. TensorFlow provides a more flexible learning rate setting method-exponential decay method. This method realizes the exponential decay learning rate, first uses a larger learning rate to quickly get a better solution, and then gradually reduces the learning rate with the continuation of the iteration, which makes the model more stable in the later stage of training and reaches the optimal value slowly and smoothly.

Tf.train.exponential_decay (learning_rate, global_step, decay_steps, decay_rate,staircase=False, name=None)

This function will reduce the learning rate exponentially, and realize the learning rate decayed_learning_rate = learning_rate * decay_rate ^ (global_step / decay_steps) after each round of actual optimization. Learning_rate is the set accident learning rate, decay_rate is the attenuation coefficient, and decay_steps is the attenuation speed. As shown in the figure below, when the parameter staircase=False is used, the change trend of the learning rate is light; when staircase=True is dark, the learning rate is changed into a ladder function (staircase function). The common application scenario of this setting is that the learning rate is reduced every time the training data is completed.

Use example: learning_rate = tf.train.exponential_decay (starter_learning_rate, global_step, 100000, 0.96).

Second, the problem of overfitting

1. Over-fitting problem and its solution

The so-called over-fitting problem means that when a model is too complex, it can well remember the random noise part of each training data and forget to learn the general trend in the training data.

In order to avoid the over-fitting problem, the commonly used method is Regularization. The idea is to add an index to the loss function to describe the complexity of the model, and define the optimization objective as J (θ) + λ R (w), where R (w) describes the complexity of the model, including that the weight term w does not include the offset term b, and λ represents the proportion of the model complex loss in the total loss. Generally speaking, the complexity of the model is only determined by the weight w. There are two commonly used functions R (w) to describe the complexity of the model, one is L1 regularization:

The other is L2 regularization:

No matter which regularization method, the basic idea is to limit the weight, so that the model can not arbitrarily fit the random noise in the training data. Difference: L1 regularization will make the parameters more sparse, while L2 will not. The so-called parameters become more sparse means that more parameters will become 0, which can achieve the function of similar feature selection. In practice, L1 regularization and L2 regularization can also be used at the same time:

two。 TensorFlow solution to over-fitting problem

Loss = tf.reduce_mean (tf.square (yy) + tf.contrib.layers.l2_regularizer (lambda) (w)

The above is a loss function with L2 regularization term. The first part is the mean square error loss function, and the second part is the regularization term. The lambda parameter represents the weight of the regularization term, that is, λ in J (θ) + λ R (w), and w is the parameter that needs to calculate the regularization loss. The tf.contrib.layers.l2_regularize () function can calculate the L2 regularization term of a given parameter. Similarly, tf.contrib.layers.l1_regularizer () can be the L1 regularization term of that given parameter.

# compare the effects of L1 regularization and L2 regularization function w = tf.constant ([[1.0,2.0,3.0]) 4]]) with tf.Session () as sess: # 0.5* (| 1 | + |-2 | + | 3 | + | 4 | = 5.0) print (sess.run (tf.contrib.layers.l1_regularizer (0.5) (w) # 5.0 # 0.5 * [(1x 4x 9x 16) / 2] = 7.5 TensorFlow divides the L2 regularization term by 2 to make the derivation more concise print (sess.run (tf.contrib.layers.l2_regularizer (0.5) (w) # 7.5

When the parameters of the neural network increase, the above way of defining the loss function will lead to the long definition of loss and poor readability. In addition, when the network structure is complex, the part of defining the network structure and the part of calculating the loss function may not be in the same function, so it is not convenient to calculate the loss function by variable. To solve this problem, you can use the collections provided in TensorFlow (collection). For the specific implementation, see the code section.

Tf.add_to_collection () adds variables to the specified collection; tf.get_collection () returns a list that stores the elements in the collection.

3. Moving average model

The other makes the model more robust on test data (robust) the moving average model. When using the random gradient descent algorithm to train the neural network, the moving average model can improve the performance of the final model on the test data in many applications, and both GradientDescent and Momentum training can benefit from the ExponentialMovingAverage method.

The tf.train.ExponentialMovingAverage provided in TensorFlow is a class class to implement the moving average model. When initializing a tf.train.ExponentialMovingAverage class object, you specify the decay rate decay and the parameter num_updates, which is used to dynamically control the decay rate. Tf.train.ExponentialMovingAverage maintains a shadow variable (shadow variable) for each variable, and the initial value of the shadow variable is the initial value of the corresponding variable. Each time the variable is updated, shadow_variable = decay * shadow_variable + (1-decay) * variable. It can be seen from the formula that decay determines the speed of model updating, and the larger the decay is, the more stable the model is. In practical application, decay is generally set to close to 1. Num_updates defaults to None. If set, the decay rate is calculated as min (decay, (1 + num_updates) / (10 + num_updates)).

The apply method of the tf.train.ExponentialMovingAverage object returns an operation to update the moving average of var_list. The var_list must be the Variable or Tensor of list, which updates the shadow variable shadowvariable of var_list. The average method can obtain the value of the variable after the moving average.

Fourth, code rendering

1. Weight L2 regularization method for complex Neural Network structure

Import tensorflow as tf'# compares the effects of L1 regularization and L2 regularization function w = tf.constant ([[1.0,2.0,3.0]) 4]]) with tf.Session () as sess: # 0.5* (| 1 | + |-2 | + |-3 | + | 4 | = 5.0) print (sess.run (tf.contrib.layers.l1_regularizer (0.5) (w) # 5.0 # 0.5 * [(1x 4x 9x 16) / 2] = 7.5 TensorFlow divides the L2 regularization term by 2 to make the derivation more concise print (sess.run (tf.contrib.layers.l2_regularizer (0.5)) (W)) # 7.5'') complex neural network structure weight L2 regularization method # defines the weight of each layer And add the L2 regularization term of the weight to the set named 'losses' (def get_weight (shape, lambda1): var = tf.Variable (tf.random_normal (shape), dtype=tf.float32) tf.add_to_collection (' losses', tf.contrib.layers.l2_regularizer (lambda1) (var)) return var x = tf.placeholder (tf.float32, (None, 2)) y = tf.placeholder (tf.float32, (None, 1)) layer_dimension = [2lop5 3] # defines the number of nodes in each layer of the neural network n_layers = len (layer_dimension) current_layer = x # sets the current layer to the input layer in_dimension = layer_dimension [0] # to generate a five-layer fully connected neural network structure for i in range by loop: out_dimension = layer_ dimensions [I] weight = get_weight ([in_dimension,out_dimension]) 0.003) bias = tf.Variable (tf.constant, shape= [out _ dimension]) current_layer = tf.nn.relu (tf.matmul (current_layer, weight) + bias) in_dimension = layer_ dimensionality [I] mse_loss = tf.reduce_mean (tf.square (y _ current_layer)) tf.add_to_collection ('losses' Mse_loss) loss = tf.add_n (tf.get_collection ('losses')) # loss function containing all parameter regularization terms

2. Sample use of tf.train.ExponentialMovingAverage

Import tensorflow as tf # tf.train.ExponentialMovingAverage uses sample v1 = tf.Variable (0, dtype=tf.float32) step = tf.Variable (0, trainable=False) # where step simulates the number of rounds of neural network iteration # to define a moving average class object to initialize the decay rate decay=0.99 The parameter num_updates ema = tf.train.ExponentialMovingAverage (0.99, num_updates=step) # apply for dynamically controlling the decay rate returns an operation to update the moving average of the var_list Var_list must be Variable or Tensor of list # this operation updates the shadow variable shadow variable maintain_averages_op = ema.apply (var_list= [v1]) with tf.Session () as sess: init_op = tf.global_variables_initializer () sess.run (init_op) # average method to get the value of the variable after moving average print (sess.run ([v1, ema.average (v1)]) # [0.0) 0.0] sess.run (tf.assign (v1,5)) # min {0.99, (1+step) (10+step) = 0.1} = 0.1 # the moving average of update v1 is 0.1 "0.09" 0.9 "5" 4.5 sess.run (maintain_averages_op) print (sess.run ([v1, ema.average (v1)]) # [5.0,4.5] sess.run (tf.assign (step) 10000)) sess.run (tf.assign (v1,10)) # min {0.99, (1+step) (10+step) = 0.999} = 0.99 # the moving average of update v1 is 0.99 (4.5) 0.01 (10) 4.555 sess.run (maintain_averages_op) print (sess.run ([v1, ema.average (v1)])) # [10.0 4.5549998] # the moving average of update v1 is 0.99, 4.555, 0.01, 10, 4.60945 sess.run (maintain_averages_op) print (sess.run ([v1, ema.average (v1)]) # [4.6094499] is all the contents of the article "example Analysis of TensorFlow Neural Network Optimization Strategy" Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.