Shulou(Shulou.com)06/01 Report--

What is the principle of BP algorithm in TensorFlow? I believe many inexperienced people don't know what to do about it. Therefore, this paper summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

I. back propagation algorithm

Back propagation algorithm [1] (Backpropagation Algorithm, referred to as BP algorithm) is not only an important ideological basis of deep learning, but also a basic knowledge that must be mastered by beginners. In this section, we will introduce this key knowledge in detail.

We use a neural network as shown in figure 1, which shows a three-layer neural network, two hidden layers and one output layer, and the input layer has two neurons to receive input samples (x1line x2), which is the output of the network.

Fig. 1 A three-layer neural network

Two. the process of feedforward calculation

In order to understand the operation process of the neural network, we need to understand the feedforward calculation, that is, the calculation process of the forward propagation of data along the neural network, taking the network shown in figure 1 as an example:

The samples entered are:

Formula 1

The parameters of the layer 1 network are:

Formula 2

The parameters of the layer 2 network are:

Formula 3

The parameters of the layer 3 network are:

Formula 4

Calculation of the first hidden layer

Figure 2 calculating the first hidden layer

There are three neurons in the first hidden layer: neu1, neu2 and neu3. The input for this layer is:

Formula 5

Taking neu1 neurons as an example, their inputs are:

Formula 6

Similarly, there are:

Formula 7

Formula 8

Suppose we choose the function f (x) as the activation function of this layer (all the activation functions in figure 1 have a subscript. In general, the activation functions of the same layer are the same, and different layers can choose different activation functions), then the output of this layer is as follows:

Calculation of the second hidden layer

Fig. 3 calculate the second hidden layer

There are two neurons in the second hidden layer: neu4 and neu5. The input for this layer is:

Formula 9

That is, the input of the second layer is the output of the first layer multiplied by the weight of the second layer, plus the offset of the second layer. So the inputs to neu4 and neu5 are:

Formula 10

Formula 11

The output of this layer is:

Calculation of output layer

Figure 4 Computing the output layer

There is only one neuron in the output layer: neu6. The input for this layer is:

Formula 12

That is:

Formula 13

Because the network is to solve a binary classification problem, the activation function of the output layer can also use a Sigmoid type function, and the final output of the neural network is:

III. Calculation of back propagation

We have seen the process of forward propagation of data along the neural network in the previous section, and in this section we will introduce the more important calculation process of back propagation. Suppose we use random gradient descent to learn the parameters of the neural network, and the loss function is defined as

Where y is the real class mark of the sample. Using gradient descent to learn parameters, we must calculate the partial derivatives of the loss function with respect to the parameters (weight w and bias b) of each layer in the neural network.

Suppose we want to hide the layer of the k layer

Assume that it represents the input of layer k neurons, that is,

Where n ^ (kMel 1) is the output of the previous layer of neurons, then according to the chain rule:

Formula 14

Formula 15

Therefore, we only need to calculate the partial derivative.

Calculate partial derivative

As mentioned earlier, the input of the layer k neuron is:, so you can get:

Formula 16

Let's take the simple neural network in Section 1.1 as an example, suppose we want to calculate the derivative of the neurons in the first hidden layer with respect to the weight matrix, then there are:

Formula 17

Calculate partial derivative

Because offset b is a constant term, the calculation of the partial derivative is also simple:

Formula 18

Still take the neurons in the first hidden layer as an example, there are:

Formula 19

Calculate partial derivative

According to the forward calculation in the first section, we know that the relationship between the input of layer k + 1 and the output of layer k is:

Formula 20

Formula 21

Formula 22

Formula 23

The following is a back propagation algorithm based on random gradient descent update parameters:

Enter:

After reading the above, have you mastered the principle of BP algorithm in TensorFlow? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.