Shulou(Shulou.com)06/01 Report--

Today, I would like to share with you how to customize the relevant knowledge points of Keras-based expansibility. The content is detailed and the logic is clear. I believe most people still know too much about this knowledge, so share this article for your reference. I hope you can get something after reading this article. Let's take a look.

Brief introduction

Keras is a framework for building neural network models on python, and the syntax is similar to torch. I personally think that the biggest feature of Keras is good packaging, some of the methods to be output in the training process and commonly used optimization functions, objective functions have been built-in, very suitable for writing large assignments. The philosophy of Keras and python is somewhat similar, that is, try not to build your own wheels.

But recently, when I visited Zhihu, I saw an answer saying that Keras can only be used to build some networks that are already popular in the world, which is relatively small compared with other frameworks. In other words, Keras is not scalable. As a person who tried theano, tensorflow, torch, caffe and other frameworks, and finally settled in Keras, I don't quite agree with this. In fact, Keras has good extensibility, on the one hand, because it has a good interface at design time, and on the other hand, because of the clear code structure, you can have a lot of custom space. So here are a few examples of how to customize layers and various methods in Keras.

0 、 backend

If you want to customize various layers and functions in Keras, you are sure to use backend. The general method of import is

From keras import backend as K

This is because Keras can have two kinds of background, namely theano and tensorflow, so some functions that manipulate the tensor may vary from background to background.

By introducing this backend, you can let Keras handle compatibility.

For example, the average of x is K.mean (x). The backend file itself is in the keras/backend folder, and you can read the code to see what operations backend supports. There are many functions in backend, which are generally enough.

1. Lambda layer

If you just want to make a transformation of the data flowing through that layer, and the transformation itself does not have any parameters to learn, then it is most appropriate to use Lambda Layer directly.

The method of import is

From keras.layers.core import Lambda

The Lambda function takes two parameters, the first is the mapping function of the input tensor to the output tensor, and the second is the mapping function of the input shape to the output shape. For example, if you want to build a layer where the data flowing through that layer will be subtracted from the average, you can define it as follows:

Def sub_mean (x): X-= K.mean (return xmodel.add (Lambda (sub_mean,output_shape=lambda input_shape:input_shape))

Because the output shape is the same as the input shape, the second parameter directly uses the identity mapping.

To build a complete model:

Def get_submean_model (): model = Sequential () model.add def sub_mean (x): X-= K.mean (x-= K.mean) return x model.add (Lambda (sub_mean,output_shape=lambda input_shape:input_shape)) model.compile (optimizer='rmsprop',loss='mse') return modelmodel = get_submean_model () res=model.predict (np.random.random

The average value of the res obtained is [5.96046448e-08-5.96046448e-08 0.00000000e+00], which shows that the function of subtracting the mean has been realized.

2. Custom non-recursive layer

If you want to define a layer that has variables to learn, you can't use the lambda layer. You need to write one yourself.

For example, if I want to define a layer whose effect is to multiply the tensor by a positive diagonal matrix (in other words, the input vector is multiplied by an element-by-element vector to be learned), you can write:

The first step is to import the base class

From keras.engine.topology import Layer

Then the MyLaber is defined as follows:

Class MyLayer (Layer): def _ init__ (self,output_dim,**kw): self.output_dim = output_dim super (MyLayer,self). _ init__ (* * kw) def build (self,input_shape): input_dim = input_shape [1] assert (input_dim = = self.output_dim) inital_SCALER = np.ones ((input_dim) ) * 1000 self.SCALER = K.variable (inital_SCALER) self.trainable_weights = [self.SCALER] super (MyLayer,self) .build (input_shape) def call (self,x,mask=None): # return x-K.mean

Mainly refer to the Keras built-in layer writing, such as Dense in keras/layers/core.py, to put the parameters that can be learned in self.trainable_weights. The initial value is set to 1000 to make the effect of this layer more significant. Then write all the models to test them.

Def get_mylayer_model (): model = Sequential () model.add (Dense (5)) model.compile (optimizer='rmsprop',loss='mse') return modelmodel = get_mylayer_model () res=model.predict (np.random.random ((3) print res

Res is as follows:

[271.2746582-1053.31506348 147.17185974-1120.33740234 609.54876709]

[- 263.69671631-390.41921997 291.17721558-594.58721924 615.97369385]

[- 46.58752823-733.11328125-21.9815979-570.79351807 649.44158936]

These are large numbers, and each value is generally no more than +-2 without MyLayer, so this layer does work.

Call model.get_weights () before fit, see that the weight of this layer is 1000, randomly come out a test set, fit thousands of epoch only, loss becomes very small, the weight of MyLayer becomes about 997, while the weight of the previous layer Dense is of the order of 10 ^-4, indicating that the parameters in MyLayer are indeed learnable.

3. Custom loss function

The built-in loss functions of Keras are all in keras/objectives.py. For example, mse is defined as:

Def mean_squared_error (y_true, y_pred): return K.mean (K.square (y_pred-y_true), axis=-1)

You can define your own loss function in the same format. For example, we want the average of the fourth power of the difference as the loss function:

Def my_object (yearly true): return K.mean (K.square (K.square (y_pred-y_true)), axis=-1)

Write all the models:

Def get_myobj_model (): model = Sequential () model.add (Dense (5) Dense (3)) def my_object: return K.mean (K.square (K.square (y_pred-y_true)), axis=-1) model.compile (optimizer='sgd',loss=my_object) return modelmodel = get_myobj_model ()

Being able to customize the loss function is a very important link, which greatly expands the application of the network. For example, you want to use cnn to train a background segmentation filter, its output pixel in the corresponding foreground position is 1, in the corresponding background position is 0. Not only do you want the mse of the network output to be small, but also you want 0 and 1 to be connected separately, so as not to produce snowflake output. Then the custom loss function can do it, actually putting two loss functions into one loss function.

Other useful loss functions, such as warp-ctc, can be integrated into the model here.

4. Custom recursion layer

The definition of a recursive layer is different from that of a non-recursive layer. According to the way LSTM is written in Keras, it also has a reset_states function and a step function, which is determined by the nature of recursion. The examples are all in keras/layers/recurrent.py.

Looking at the variant of LSTM written by the seniors in lasagne before, it made me want to cry. I might as well copy the LSTM code in Keras to fix it. However, LSTM cannot be copied directly, and several dependencies of import are needed:

Rom keras.layers.recurrent import LSTM,Recurrent,time_distributed_densefrom keras import initializations,regularizers,activationsfrom keras.engine import InputSpec5, custom optimization function

The code of Keras is really good, and the coupling is very low. Keras's built-in optimization function is in keras/optimizers.py, and the base class Optimizer is also in this file. For example, copy its built-in SGD algorithm to its own file, as long as the first from keras.optimizers import Optimizer can be compiled through.

Sometimes in order to get the results of state-of-the-art, it is necessary to use the sgd addition method to fully converge. For example, the learning rate is 0.01to study 100epoch, then halve the learning rate, then learn 100epoch, and so on. If you do not customize the optimization function, you have to call the fit function in stages, modify the learning rate, and possibly re-compile. This is not very beautiful. Other bizarre learning strategies can also be obtained through custom optimization functions.

6. Postscript

Keras is really very powerful, not only can be used to do big homework, but also enough to do some research. Yeah

Add: extensibility of keras: custom keras

1. Custom keras

Keras is a deep learning API that can quickly implement your experiments. Keras also integrates many pre-training models, which can achieve many routine tasks, such as image classification. Since TensorFlow 2. 0, tensorflow itself has become keraseized.

On the other hand, keras shows a high degree of modularization and encapsulation, so some people will think that keras is not easy to expand, such as implementing a new Loss, a new network layer structure; in fact, it can be quickly expanded through the basic modules of keras to achieve updated algorithms.

This article summarizes the customization of layer,model and loss based on the extensibility of keras.

two。 Custom keras layers

Layers is an important part of keras, and every component of the network structure should be represented by layers. Keras provides a lot of regular layer, such as Convolution layers,pooling layers, activation layers, dense layers, etc. We can extend the custom layers by inheriting the base layers.

2.1 base layer

Layer implements the operation classes of input tensor and output tensor. Here are five methods of base layer. Custom layer only needs to override these methods.

Init (): define some properties of a custom layer

Build (self, input_shape): define the weight weights required by layer

Call (self, * args, * * kwargs): the specific operation of layer will be performed automatically when calling custom layer.

Get_config (self): the configuration initialized by layer, which is a dictionary dictionary.

Compute_output_shape (self,input_shape): calculates the shape of the output tensor

2.2 example # Standardization layer class InstanceNormalize (Layer): def _ init__ (self, * * kwargs): super (InstanceNormalize, self). _ init__ (* * kwargs) self.epsilon = 1e-3 def call (self, x, mask=None): mean, var = tf.nn.moments (x, [1,2], keep_dims=True) return tf.div (tf.subtract (x, mean) Tf.sqrt (tf.add (var, self.epsilon)) def compute_output_shape (self,input_shape): return input_shape# calls inputs = keras.Input (shape= (None, None, 3)) x = InstanceNormalize () (inputs) # you can create weights class SimpleDense (Layer) through add_weight (): def _ _ init__ (self) Units=32): super (SimpleDense, self). _ _ init__ () self.units = units def build (self, input_shape): self.w = self.add_weight (shape= (input_shape [- 1], self.units), initializer='random_normal', trainable=True) self.b = self.add_weight (shape= (self.units,) Initializer='random_normal', trainable=True) def call (self, inputs): return tf.matmul (inputs, self.w) + self.b# call inputs = keras.Input (shape= (None, None, 3)) x = SimpleDense (units=64) (inputs) 3. Custom keras model

When we define the network structure, we will put the entire workflow in keras.Model, do compile (), and then train through fit (). When fit () is executed, train_step (self, data) in Model is called when each batch size data is executed.

From keras.models import Sequentialfrom keras.layers import Dense, Activationmodel = Sequential () model.add (Dense (units=64, input_dim=100)) model.add (Activation ("relu")) model.add (Dense (units=10)) model.add (Activation ("softmax") model.compile (loss='categorical_crossentropy', optimizer='sgd', metrics= ['accuracy']) model.fit (x_train, y_train, epochs=5, batch_size=32)

When you need to control the training process yourself, you can override Model's train_step (self, data) method.

Class CustomModel (keras.Model): def train_step (self, data): # Unpack the data. Its structure depends on your model and # on what you pass to `fit () `. X, y = data with tf.GradientTape () as tape: y_pred = self (x, training=True) # Forward pass # Compute the loss value # (the loss function is configured in `compile () `) loss = self.compiled_loss (y, y_pred, regularization_losses=self.losses) # Compute gradients trainable_vars = self.trainable_variables gradients = tape.gradient (loss Trainable_vars) # Update weights self.optimizer.apply_gradients (zip (gradients, trainable_vars)) # Update metrics (includes the metric that tracks the loss) self.compiled_metrics.update_state (y, y_pred) # Return a dict mapping metric names to current value return {m.name: m.result () for m in self.metrics} import numpy as np# Construct and compile an instance of CustomModelinputs = keras.Input (shape= (32 ) outputs = keras.layers.Dense (1) (inputs) model = CustomModel (inputs, outputs) model.compile (optimizer= "adam", loss= "mse", metrics= ["mae"]) # Just use `fitting as usualx = np.random.random ((1000, 32)) y = np.random.random ((1000, 1) model.fit (x, y, epochs=3) 4. Custom keras loss

Keras implements cross-entropy and other common loss, custom loss is relatively common for the use of keras, to achieve a variety of magic changes to loss, such as focal loss.

Let's take a look at the loss implementation in the keras source code

Def categorical_crossentropy (y_true, y_pred): return K.categorical_crossentropy (y_true, y_pred) def mean_squared_error (y_true, y_pred): return K.mean (K.square (y_pred-y_true), axis=-1)

You can see that the input is groud true y_true and the predicted value y_pred, which is returned as the function that calculates loss. Custom loss can refer to this mode.

Def focal_loss (weights=None, alpha=0.25, gamma=2): r "Compute focal loss for predictions. Multi-labels Focal loss formula: FL =-alpha * (zmurp) ^ gamma * log (p)-(1-alpha) * p ^ gamma * log (1Repp), which alpha = 0.25, gamma = 2, p = sigmoid (x), z = target_tensor. # https://github.com/ailias/Focal-Loss-implement-on-Tensorflow/blob/master/focal_loss.py Args: prediction_tensor: A float tensor of shape [batch_size, num_anchors, num_classes] representing the predicted logits for each class target_tensor: A float tensor of shape [batch_size, num_anchors, num_classes] representing one-hot encoded classification targets weights: A float tensor of shape [batch_size Num_anchors] alpha: A scalar tensor for focal loss alpha hyper-parameter gamma: A scalar tensor for focal loss gamma hyper-parameter Returns: loss: a (scalar) tensor representing the value of the loss function "def _ custom_loss (y_true, y_pred): sigmoid_p = tf.nn.sigmoid (y_pred) zeros = array_ops.zeros_like (sigmoid_p, dtype=sigmoid_p.dtype) # For poitive prediction Only need consider front part loss, back part is 0 # target_tensor > zeros zonal 1, so poitive coefficient = z-p. Pos_p_sub = array_ops.where (y_true > zeros, y_true-sigmoid_p, zeros) # For negative prediction, only need consider back part loss, front part is 0; # target_tensor > zeros zodi1, so negative coefficient = 0. Neg_p_sub = array_ops.where (y_true > zeros, zeros, sigmoid_p) per_entry_cross_ent =-alpha * (pos_p_sub * * gamma) * tf.log (tf.clip_by_value (sigmoid_p, 1e-8) )-(1-alpha) * (neg_p_sub * * gamma) * tf.log (tf.clip_by_value (1-sigmoid_p, 1e-8) Return tf.reduce_sum (per_entry_cross_ent) return _ custom_loss above are all the contents of the article "how to customize keras based on the extensibility of Keras" Thank you for reading! I believe you will gain a lot after reading this article. The editor will update different knowledge for you every day. If you want to learn more knowledge, please pay attention to the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.