Shulou(Shulou.com)06/01 Report--

This article mainly explains "how to use ReduceLROnPlateau". Interested friends may wish to have a look at it. The method introduced in this paper is simple, fast and practical. Now let the editor take you to learn how to use ReduceLROnPlateau.

ReduceLROnPlateau is the callback included by default in Keras. The learning rate of neural network determines the scale factor of the gradient, so too high learning rate will cause the optimizer to exceed the optimal value, while too low learning rate will lead to too long training time. It is difficult to find a static, effective and constant learning rate.

As the name implies, "reducing the learning rate at high altitude" is to reduce the learning rate when the loss index stops improving or reaches stability. The general learning rate is reduced by 2 to 10 times, which helps to hone the optimal value of the parameters.

To use ReduceLROnPlateau, you must first create a callback object. Four parameters are important:

Monitor, which is used to monitor metrics

Factor, which is the factor that the new learning rate will be reduced (multiplied by)

Persistence, the number of stagnant epoch waiting before callback activation

Min_lr, which can be reduced to the minimum learning rate. This can prevent unnecessary and unbeneficial reductions.

From keras.callbacks import ReduceLROnPlateaureduce_lr = ReduceLROnPlateau (monitor='val_loss', factor=0.2, patience=5, min_lr=0.001) model.fit (X_train, Y_train, callbacks= [reduce _ lr])

When using model.fit, you can specify callback parameters. Note that this accepts a list, so multiple callbacks can be scheduled.

LearningRateScheduler is another option for ReduceLROnPlateau, which allows users to arrange learning rates according to epoch. If you know (possibly from previous research or experiments) that the learning rate of the network from epochs 1-10:00 should be x, and in epochs 10-20 it should be yling learning Scheduler that can help implement these changes. The numbers of the above epoch can be changed at will.

Creating a learning rate scheduler requires a user-defined function that takes epoch and learning rate as parameters. The return object should be the new learning rate.

From keras.callbacks import LearningRateSchedulerdef scheduler (epoch, lr): # define callback schedule if lr < 0.001: return lr * 1.5 # if lr is too small, add lr elif epoch < 5: return lr # the first five epoch, do not change lr elif epoch < 10: return lr * tf.math.exp (- 0.1) # fifth to tenth epoch, reduce lr else: return lr * tf.math.exp (- 0.05i) # after 10th epoch Reduce callback = LearningRateScheduler (scheduler) # create callback object model.fit (X_train, y_train, epochs=15, callbacks= [callback])

Then, after converting it to a Keras callback, it can be used for model training. These schedulers are useful and allow control of the network, but it is recommended that you use ReduceLROnPlateau when training the network for the first time because it is more adaptable. You can then visualize the model to see if you can provide ideas on how to construct an appropriate LR scheduler.

In addition, you can use both ReduceLROnPlateau and LearningRateScheduler, for example, using a scheduler to hard-code some learning rates (for example, no change in the first 10 epoch), while taking advantage of adaptive ability to reduce learning rates on the plateau to improve performance.

EarlyStopping can be very helpful in preventing additional redundant operations when training the model. Redundant operation can lead to high computational costs. When the network is not improved within a given period of time, the network completes the training and stops using computing resources. Like ReduceLROnPlateau, EarlyStopping requires monitor.

From keras.callbacks import EarlyStopping callback = EarlyStopping (monitor='loss', patience=5) model.fit (X_train, y_train, epochs=15, callbacks= [callback])

TerminateOnNaN helps prevent gradient explosion problems during training because entering NaN causes other parts of the network to explode. If you do not use TerminateOnNaN,Keras, it does not prevent the training of the network. In addition, nan can lead to an increase in the demand for computing power. To prevent this from happening, adding TerminateOnNaN is a good security check.

Rom keras.callbacks import TerminateOnNaNmodel.fit (X_train, y_train, epochs=15, callbacks = [TerminateOnNaN ()])

For many reasons, ModelCheckpoint can save the weights of a model at some frequency (perhaps every 10 or so epoch), so it is very useful.

If the model is suddenly interrupted during training, there is no need to completely retrain the model.

If, for example, at the 30th epoch, the model begins to show signs of overfitting or other problems, such as gradient explosions, we can reload the model with recently saved weights (such as at 25th epoch) and adjust the parameters to avoid the problem without most of the retraining.

The ability to extract the weight of an epoch and reload it into another model is beneficial to migration learning.

In the following scenario, ModelCheckpoint is used to store the weights of the model with the best performance. At each epoch, if the model performs better than the epoch of other records, its weight is stored in a file (overwriting the weight of the previous one). At the end of the training, we use model.load_weights to load.

From keras.callbacks import ModelCheckpointcallback = ModelCheckpoint (# create callback filepath='/filepath/checkpoint', # tell callback where the filepath to store weights is save_weights_only=True, # keep weights only (more efficient), not the entire model monitor='val_acc', # Metric mode='max', # find model weights that maximize metrics save_best_only=True # retain only the weights of the best model (more efficient) Instead of all weights) model.fit (X_train, y_train, epochs=15, callbacks= [callback]) model.load_weights (checkpoint_filepath) # loads the optimal weights into the model.

Or, if you need frequency-based saving (every 5 epoch saves), set save_freq to 5

Writing custom callbacks is one of the best features included in Keras, allowing highly specific operations to be performed. Note, however, that it is much more complex to construct than using the default callback.

Our custom callback will take the form of a class. Similar to building neural networks in PyTorch, we can inherit the keras.callbacks.Callback callback, which is a base class.

Our class can have many functions that must have the given names listed below and when they will run. For example, the on_epoch_begin function will be run at the beginning of each epoch. Here are all the functions that Keras will read from the custom callback, but you can add other "helper" functions.

Class CustomCallback (keras.callbacks.Callback): # inherits keras's base class def on_train_begin (self, logs=None): # Log is a dictionary of certain metrics, for example, the key can be ['loss',' mean_absolute_error'] def on_train_end (self, logs=None):. Def on_epoch_begin (self, epoch, logs=None):... Def on_epoch_end (self, epoch, logs=None):... Def on_test_begin (self, logs=None):... Def on_test_end (self, logs=None):... Def on_predict_begin (self, logs=None):... Def on_predict_end (self, logs=None):... Def on_train_batch_begin (self, batch, logs=None):... Def on_train_batch_end (self, batch, logs=None):... Def on_test_batch_begin (self, batch, logs=None):... Def on_test_batch_end (self, batch, logs=None):... Def on_predict_batch_begin (self, batch, logs=None):... Def on_predict_batch_end (self, batch, logs=None):...

Depending on the function, you can access different variables. For example, in the function on_epoch_begin, the function can access both the epoch number and the dictionary of the current metrics and logs. If you need other information, such as the learning rate, you can use keras.backend.get_value.

Then, you can treat your custom callback function like any other callback function.

Model.fit (X_train, y_train, epochs=15, callbacks= [CustomCallback ()]) so far, I believe you have a deeper understanding of "how to use ReduceLROnPlateau". You might as well do it in practice! Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.