Shulou(Shulou.com)06/01 Report--

How to implement TensorFlow 2.x Keras-based model construction, in view of this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

Keras introduction

Keras is an advanced neural network API written in Python. It is an independent library that can run with TensorFlow, CNTK or Theano as the back end. TensorFlow has tried to integrate with Keras since version 1.0, and has deeply integrated Keras since the release of version 2.0, and closely relies on tf.keras as its central high-level API. Keras API is also highly recommended to complete the construction of deep models.

Tf.keras has three key advantages:

User-friendly to novice users: Keras has a simple and consistent interface and has clear and feasible suggestions for correcting user-generated errors. Versions prior to TensorFlow 2.0 have been widely criticized for their complex coding, messy API interfaces and poor compatibility between versions. The use of Keras for unification will greatly reduce the workload of developers.

Modular and combinable: Keras models are connected by buildable modules without any restrictions, the model structure is clear, and the code is easy to read.

Easy to extend: when writing a new custom module, it can be easily extended based on existing interfaces.

Keras makes TensorFlow easier to use without losing its flexibility and performance.

Keras model construction

In TensorFlow 2.x, you can build Keras models in three ways: Sequential, Functional API, and custom model (Subclassed). These three construction methods are described below.

Sequential Model

In Keras, multiple layers (layer) are usually assembled to form a model (model). One of the most common ways is layer stacking, which can be easily implemented using tf.keras.Sequential. As an example of the model shown in the above figure, the code implementation is as follows:

Import tensorflow as tffrom tensorflow.keras import layersmodel = tf.keras.Sequential () # Adds a densely-connected layer with 64 units to the model:model.add (layers.Dense (64, activation='relu', input_shape= (16,)) # This is identical to the following:# model.add (layers.Dense (64, activation='relu', input_dim=16)) # model.add (layers.Dense (64, activation='relu', batch_input_shape= (None, 16)) # Add another:model.add (layers.Dense (64) Activation='relu')) # Add an output layer with 10 output units:model.add (layers.Dense (10)) # model.build ((None, 16)) print (model.weights)

Note that for the first layer added by Sequential, you can include an input_shape or input_dim or batch_input_shape parameter to specify the dimension of the input data, as detailed in the comments section. When the parameters such as input_shape are specified, each add new layer is in the process of creating the model continuously, that is to say, the weight matrix of each layer in the model has been initialized, and the weight information of the model can be printed by calling model.weights.

Of course, the first layer may not contain the dimensional information of the input data, which is called the delayed creation pattern, that is, the model has not been really created and the weight matrix does not exist. You can create the model manually by calling the model.build (batch_input_shape) method. If it is not created manually, the model will be created and the weight matrix initialized only when fit or other training and evaluation methods are called, and the model will automatically infer its dimension information based on the input data.

The size of the batch is not specified in input_shape and is set to None because the size of the batch used in training and evaluation may be inconsistent. If it is set to a fixed value, errors will occur during training or evaluation, and after this setting, the batch size can be automatically inferred from the model and calculated, which is more robust.

In addition to this sequential addition (add), you can also build a model by passing layers to Sequential as parameters. The sample code is as follows:

Import tensorflow as tffrom tensorflow.keras import layersmodel = tf.keras.Sequential ([layers.Dense (64, activation='relu', input_shape= (16,)), layers.Dense (64, activation='relu'), layers.Dense (10)]) # model.build ((None, 16)) print (model.weights)

Functional API

Keras's functional API is a more flexible way to create models than Sequential. It can deal with models with nonlinear topology, models with shared layer (layers) and models with multiple inputs and outputs. The model of deep learning is usually a directed acyclic graph composed of layers, and functional API is an effective way to construct such a graph.

Taking the model mentioned in the Sequential Model section as an example, the implementation using functional API is as follows:

From tensorflow import kerasfrom tensorflow.keras import layersinputs = keras.Input (shape= (16,)) dense = layers.Dense (64, activation='relu') x = dense (inputs) x = layers.Dense (64, activation='relu') (x) outputs= layers.Dense (10) (x) model = keras.Model (inputs=inputs, outputs=outputs, name='model') model.summary ()

Different from using the Sequential method to build the model, the functional API specifies the input inputs through keras.Input and generates the output outputs through function calls, and finally uses the keras.Model method to build the whole model.

Why is it called functional API? as you can see from the code, you can use various layers (layers) like function calls. For example, if you define the dense layer, you can directly take inputs as the input of dense and get an output x, and then take x as the input of the next layer, and the final function return value is the output of the entire model.

Functional API can use the same layer (layers) as part of multiple models. The sample code is as follows:

From tensorflow import kerasfrom tensorflow.keras import layersencoder_input = keras.Input (shape= (16,), name='encoder_input') x = layers.Dense (32, activation='relu') (encoder_input) x = layers.Dense (64, activation='relu') (x) encoder_output = layers.Dense (128, activation='relu') (x) encoder = keras.Model (encoder_input, encoder_output, name='encoder') encoder.summary () x = layers.Dense (64, activation='relu') (encoder_output) x = layers.Dense (32) Activation='relu' (x) decoder_output = layers.Dense (16, activation='relu') (x) autoencoder = keras.Model (encoder_input, decoder_output, name='autoencoder') autoencoder.summary ()

The code contains two models, an encoder (encoder) and a self-encoder (autoencoder), and you can see that the two models share the encoder_out layer, and of course all the layers before the encoder_out layer.

All models (models) generated by functional API can be called like layers. Also take the self-encoder (autoencoder) as an example, which is now divided into two parts: the encoder (encoder) and the decoder (decoder), and then generate the autoencoder with encoder and decoder, as follows:

From tensorflow import kerasfrom tensorflow.keras import layersencoder_input = keras.Input (shape= (16,), name='encoder_input') x = layers.Dense (32, activation='relu') (encoder_input) x = layers.Dense (64, activation='relu') (x) encoder_output = layers.Dense (128, activation='relu') (x) encoder = keras.Model (encoder_input, encoder_output, name='encoder') encoder.summary () decoder_input = keras.Input (shape= (128,) Name='decoder_input') x = layers.Dense (64, activation='relu') (decoder_input) x = layers.Dense (32, activation='relu') (x) decoder_output = layers.Dense (16, activation='relu') (x) decoder = keras.Model (decoder_input, decoder_output, name='decoder') decoder.summary () autoencoder_input = keras.Input (shape= (16), name='autoencoder_input') encoded = encoder (autoencoder_input) autoencoder_output = decoder (encoded) autoencoder = keras.Model (autoencoder_input Autoencoder_output, name='autoencoder',) autoencoder.summary ()

Firstly, two models encoder and decoder are generated in the code, and then when generating the autoencoder model, we use the method of model function call to directly take autoencoder_input and encoded as the input of encoder and decoder respectively, and finally get the autoencoder model.

Functional API can easily handle multi-input and multi-output models, which Sequential API cannot achieve. For example, some of the inputs of our model are category features, which generally need to be processed by Embedding, and some are numerical features, which generally do not need special processing. Obviously, it is impossible to merge these two features directly as a single input, so multiple inputs will be used. Sometimes we want the model to return multiple outputs for subsequent calculations, so the multi-output model is used. The sample code for the multi-input and multi-output model is as follows:

From tensorflow import kerasfrom tensorflow.keras import layerscategorical_input = keras.Input (shape= (16,)) numeric_input = keras.Input (shape= (32,)) categorical_features = layers.Embedding (input_dim=100, output_dim=64, input_length=16,) (categorical_input) categorical_features = layers.Reshape ([16 * 64]) (categorical_features) numeric_features = layers.Dense (64, activation='relu') (numeric_input) x = layers.Concatenate (axis=-1) ([categorical_features Numeric_features]) x = layers.Dense (12828, activation='relu') (x) binary_pred = layers.Dense (1, activation='sigmoid') (x) categorical_pred = layers.Dense (3, activation='softmax') (x) model = keras.Model (inputs= [typical _ input, numeric_input], outputs= [binary _ pred, categorical_pred],) model.summary ()

There are two input categorical_input and numeric_input in the code. After different processing layers, the two are combined together through Concatenate. Finally, two output binary_pred and categorical_pred are obtained through different processing layers. The structure of the model is shown in the following figure:

Another good use of functional API is layer sharing of models, that is, layers are reused many times in a model, which learns different characteristics from different inputs. A common sharing layer is the embedded layer (Embedding), which is coded as follows:

From tensorflow import kerasfrom tensorflow.keras import layerscategorical_input_one = keras.Input (shape= (16,)) categorical_input_two = keras.Input (shape= (24,)) shared_embedding = layers.Embedding 64) categorical_features_one = shared_embedding (categorical_input_one) categorical_features_two = shared_embedding (categorical_input_two) categorical_features_one = layers.Reshape ([16 * 64]) (categorical_features_one) categorical_features_two = layers.Reshape ([16 * 64]) (categorical_features_two) x = layers.Concatenate (axis=-1) ([categorical_features_one, categorical_features_two,]) x = layers.Dense (128 Activation='relu') (x) outputs= layers.Dense (1, activation='sigmoid') (x) model = keras.Model (inputs= [typical _ input_one, categorical_input_two], outputs=outputs,) model.summary ()

There are two inputs in the code, categorical_input_one and categorical_input_two, which share an Embedding layer shared_embedding. The structure of the model is shown in the following figure:

Customize Keras layers and models

There are many built-in layers under the tf.keras module, such as Dense, Embedding, Reshape and so on. Sometimes we find that these built-in layers do not meet our needs, and it is easy to create custom layers to extend them. The custom layer is implemented by inheriting the tf.keras.Layer class, which implements the build and call methods of the parent class. For the built-in Dense layer, if you implement it using a custom layer, the code is as follows:

Import tensorflow as tffrom tensorflow import kerasfrom tensorflow.keras import layersclass CustomDense (layers.Layer): def _ init__ (self, units=32): super (). _ _ init__ () self.units = units def build (self, input_shape): self.w = self.add_weight (shape= (input_shape [- 1], self.units), initializer='random_normal', trainable=True ) self.b = self.add_weight (shape= (self.units,), initializer='random_normal', trainable=True,) def call (self, inputs): return tf.matmul (inputs, self.w) + self.b def get_config (self): return {'units': self.units} @ classmethod def from_config (cls Config): return cls (* * config) inputs = keras.Input (4,) layer = CustomDense (10) outputs = layer (inputs) model = keras.Model (inputs) Outputs) model.summary () # layer recreateconfig = layer.get_config () new_layer = CustomDense.from_config (config) new_outputs = new_layer (inputs) print (new_layer.weights) print (new_layer.non_trainable_weights) print (new_layer.trainable_weights) # model recreateconfig = model.get_config () new_model = keras.Model.from_config (config, custom_objects= {'CustomDense': CustomDense},) new_model.summary ()

The _ _ init__ method is used to initialize some basic parameters needed to build the layer, the build method is used to create the weight matrix w and deviation matrix b needed for the layer, and the call method is the real executor of the layer construction, which converts the input into output and returns. In fact, the creation of the weight matrix can also be completed in the _ _ init__ method, but in many cases, we cannot predict the dimension of the input data in advance, and we need to delay the creation of the weight matrix at a certain point in the instantiation layer, so we need to create the weight matrix dynamically according to the dimension information input_shape of the input data in the build method.

The above three methods are called in the order of _ _ init__, build, and call, where _ _ init__ is called when the layer is instantiated, while build and call are called only after the input is determined. In fact, there is a built-in method _ _ call__ in the Layer class, which is called first when the layer is built, and build and call are called inside the method, and build is triggered only when _ _ call__ is called for the first time, that is, variables in build can only be created once, while call can be called multiple times, such as training and evaluation.

If you need to provide serialization support for this layer, you need to implement a get_config method to return the constructor parameters of the instance of this layer in dictionary form. Given the dictionary of the config, you can recreate the layer by calling the classmethod method (from_config) of the layer. The default implementation of from_config is shown in the code, and the re-creation of the layer can be seen in the layer recreate code section. Of course, you can also override the from_config class method to provide a new way of creation. The code to recreate the new model (model) is different from the code rebuilt by layer. It needs to be built with the help of the keras.Model.from_config method, as detailed in the model recreate code section.

Custom layers can be combined recursively, that is, one layer can be used as an attribute of another layer. It is generally recommended to create a sublayer in the _ _ init__ method, because its own build method will be triggered when the outer build is called to perform the task of building the weight matrix. There is no need to display the creation in the parent layer. The model mentioned in the Sequential Model section is also taken as an example, and the code is as follows:

From tensorflow import kerasfrom tensorflow.keras import layersclass MLP (layers.Layer): def _ init__ (self): super (). _ _ init__ () self.dense_1 = layers.Dense (64, activation='relu') self.dense_2 = layers.Dense (64, activation='relu') self.dense_3 = layers.Dense (10) def call (self Inputs): X = self.dense_1 (inputs) x = self.dense_2 (x) x = self.dense_3 (x) return xinputs = keras.Input ((16,)) mlp = MLP () y = mlp (inputs) print ('weights:', len (mlp.weights)) print (' trainable weights:', len (mlp.trainable_weights))

From the code, we can see that we take the three Dense layers as the sub-layers of MLP, and then use them to complete the construction of MLP, which can achieve the same effect as in Sequential Model, and the weight matrix of all sub-layers will exist as the weight matrix of the new layer.

The layers recursively collects the losses (losses) generated during the creation process during the build process. When overriding the call method, you can increase the loss of customization by calling the add_loss method. All the losses of the layer also include the losses of its children, and they can all be obtained through the layer.losses property, which is a list, and it is important to note that the loss of regular items is automatically included. The sample code is as follows:

Import tensorflow as tffrom tensorflow import kerasfrom tensorflow.keras import layersclass CustomLayer (layers.Layer): def _ init__ (self, rate=1e-2, l2_rate=1e-3): super (). _ init__ () self.rate = rate self.l2_rate = l2_rate self.dense = layers.Dense (units=32, kernel_regularizer=keras.regularizers.l2 (self.l2_rate),) def call (self Inputs): self.add_loss (self.rate * tf.reduce_sum (inputs)) return self.dense (inputs) inputs = keras.Input ((16,)) layer = CustomLayer () x = layer (inputs) print (layer.losses)

The call method of the layer or model is preset with a training parameter, which is a variable of type bool, indicating whether it is in the training state or not. It sets the value according to the method called, which is True for training and False for evaluation. Because some layers, such as BatchNormalization and Dropout, are only used in the training process, but not in the process of evaluation and prediction, this parameter can be used to control the different calculation processes performed by the model in different states.

The custom model is similar to the implementation of the custom layer, but the model needs to be inherited from tf.keras.Model. Some API of the Model class are the same as the Layer class. For example, the custom model also implements the _ _ init__, build and call methods. However, there are differences between the two. First, Model has a training, evaluation and prediction interface. Secondly, it can view the information of all built-in layers through model.layers. In addition, the Model class also provides an interface for model storage and serialization. Taking AutoEncoder as an example, the sample code for a complete custom model is as follows:

From tensorflow import kerasfrom tensorflow.keras import layersclass Encoder (layers.Layer): def _ init__ (self, l2_rate=1e-3): super (). _ _ init__ () self.l2_rate = l2_rate def build (self, input_shape): self.dense1 = layers.Dense (units=32, activation='relu', kernel_regularizer=keras.regularizers.l2 (self.l2_rate)) ) self.dense2 = layers.Dense (units=64, activation='relu', kernel_regularizer=keras.regularizers.l2 (self.l2_rate),) self.dense3 = layers.Dense (units=128, activation='relu', kernel_regularizer=keras.regularizers.l2 (self.l2_rate),) def call (self Inputs): X = self.dense1 (inputs) x = self.dense2 (x) x = self.dense3 (x) return xclass Decoder (layers.Layer): def _ init__ (self, l2_rate=1e-3): super (). _ init__ () self.l2_rate = l2_rate def build (self, input_shape): self.dense1 = layers.Dense (units=64) Activation='relu', kernel_regularizer=keras.regularizers.l2 (self.l2_rate),) self.dense2 = layers.Dense (units=32, activation='relu', kernel_regularizer=keras.regularizers.l2 (self.l2_rate),) self.dense3 = layers.Dense (units=16, activation='relu' Kernel_regularizer=keras.regularizers.l2 (self.l2_rate),) def call (self Inputs): X = self.dense1 (inputs) x = self.dense2 (x) x = self.dense3 (x) return xclass AutoEncoder (keras.Model): def _ init__ (self): super (). _ init__ () self.encoder = Encoder () self.decoder = Decoder () def call (self) Inputs): X = self.encoder (inputs) x = self.decoder (x) return xmodel = AutoEncoder () model.build ((None, 16)) model.summary () print (model.layers) print (model.weights)

The above code implements an AutoEncoder Model class, which consists of two layers, Encoder and Decoder, which are also custom. You can view all the weight information of the model by calling model.weights, which of course includes the ownership weight information in the sub-layer.

For a custom layer or model, when calling its summary, weights, variables, trainable_weights, losses and other methods or properties, make sure that the layer or model has been created, otherwise it may report an error or return empty, which should be noted when debugging the model.

Configuration layer (layer)

There are many predefined layers under the tf.keras.layers module, most of which have the same constructor parameters. Here are some commonly used parameters. For the unique parameters of each layer and the meaning of the parameters, you can consult the official documentation when you use it. The documentation is generally explained in detail.

Activation refers to the activation function, which can be set to a string such as relu or activations object tf.keras.activations.relu (), or None by default, indicating a linear relationship.

Kernel_initializer and bias_initializer, which represent the initialization of weight matrix and deviation matrix in the layer, can be set to a string such as Glorotuniform or initializers object tf.keras.initializers.GlorotUniform (), which is initialized by Glorotuniform by default.

Kernel_regularizer and bias_regularizer, which represent the regularization of weight matrix and deviation matrix, as described above, can be L1 or L2 regularization, such as tf.keras.regularizers.l2 (1e-3), which has no regularization by default.

Comparison of model creation methods

When building a relatively simple model, the use of Sequential is of course the most convenient and fast, you can use the existing Layer to complete the process of rapid construction and verification.

If the model is complex, it is best to use a functional API or a custom model.

Functional API can do model verification in advance, because it specifies the input dimension of the model in advance through the Input method, so when the input does not conform to the specification, it will be found earlier, which is helpful for us to debug, while the custom model does not specify the dimension of the input data at the beginning, it is inferred according to the input data during the run.

Using functional API to write code is not modular, it is difficult to read, and through the custom model, you can clearly understand the overall structure of the model, easy to understand.

In practice, we can combine functional API with custom models to meet our various model building needs.

Skills of Keras Model creation

When looking for the required tensorflow method, it is preferred to use this method if there is an implementation under the keras module, and if not, you can find the method under the tf module, which makes the code more compatible and robust.

In the process of model creation, we often use the built-in methods and properties of the model and layer, such as summary, weights, etc., so that we can examine the structure of the model from a global point of view and help to find some potential problems.

Because the TensorFlow 2.x model uses the Eager Execution dynamic graph mechanism by default to run the code, you can print Tensor directly anywhere in the code to view its values and dimensions, which is very helpful when debugging the model.

This is the answer to the question on how to implement TensorFlow 2.x Keras-based model building. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel for more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.