Shulou(Shulou.com)06/01 Report--

This article shows you how to use code to achieve depth neural network algorithm, the content is concise and easy to understand, can definitely brighten your eyes, through the detailed introduction of this article, I hope you can get something.

Now everyone is talking about deep learning, and it is necessary to maintain the learning spirit. Architects should always be concerned and sensitive to core technologies and key algorithms. If necessary, they should start to write and master them. There is no need to care about when to use them. Whether it is a political issue or not is a technical issue, just as soldiers do not care about whether to fight or not, but how to win.

First, how programmers learn machine learning

Machine learning has a certain threshold for programmers, and this threshold is also its core competitiveness. I believe that when you learn machine learning, English papers full of mathematical formulas must give you a headache and back down. But in fact, the machine learning algorithm landing program is not difficult to write. The following is the inverse multi-layer (BP) neural network algorithm implemented in 70 lines of code, that is, the legendary deep learning (of course, this concept may be very large). After the author's research, it is not only the neural network. In fact, most machine learning algorithms such as logical regression, decision tree c45, id3, random forest, Bayesian, collaborative filtering, graph computing, kmeans, pagerank and so on can be implemented in 100 lines of stand-alone programs (consider sharing later). The real difficulty of machine learning lies in why it is calculated in this way, what is the mathematical principle behind it, and how to derive the formula. Most of the materials on the Internet introduce this theoretical knowledge, but rarely tell you what the calculation process and program landing of the algorithm is like. For programmers, what you need to do is only engineering applications, and do not need to prove a new mathematical calculation method. In fact, most machine learning engineers use open source packages or tool software written by others to input data and adjust calculation coefficients to train the results, and even rarely implement the algorithm process themselves. However, it is still very important to master the calculation process of each algorithm, so that you can understand what kind of change the algorithm has caused the data, and understand what the purpose of the algorithm is to achieve. Next, we focus on the stand-alone implementation of the reverse neural network. Fourinone provides a very flexible and perfect parallel computing framework for the multi-machine parallelization of the neural network. We only need to understand the implementation of the single computer program, we can conceive and design the distributed parallelization scheme. If we do not understand the algorithm calculation process, all ideas will not be able to be carried out. In addition, there is convolution neural network, which is mainly a kind of dimensionality reduction idea, which is used for image processing, which is not discussed in this paper.

2. explanation of the calculation process of neural network

The neural network structure is shown in the following figure, the leftmost is the input layer, the rightmost is the output layer, and the middle is a number of hidden layers. for each neural node of the hidden layer and the output layer, it is obtained by multiplying the upper layer by its weight, with the circle marked "+ 1" as the intercept term b, and for each node outside the input layer: Y=w0*x0+w1*x1+...+wn*xn+b From this we can know that the neural network is equivalent to a structure of multi-layer logic regression.

The calculation process of the algorithm: the input layer starts, calculates from left to right, and moves forward layer by layer until the output layer produces the result. If there is a gap between the result value and the target value, then calculate from right to left, calculate the error of each node layer by layer, and adjust the ownership weight of each node, reverse to the input layer, then re-calculate and repeat the above steps until the ownership weight parameter converges to a reasonable value. Because the computer program is different from the mathematical method in solving equation parameters, it is generally necessary to randomly select the parameters, and then constantly adjust the parameters to reduce errors until approaching the correct value, so most machine learning is constantly iterative training. let's take a detailed look at the implementation of the process.

Third, the explanation of the algorithm program of neural network.

It is divided into three processes: initialization, forward calculation results, and reverse modification of weights.

1. Initialization process: because it is an n-layer neural network, we use a two-dimensional array layer to record node values, the first dimension is the number of layers, the second dimension is the location of nodes in this layer, and the values of the array are node values; similarly, the node error value layerErr is also recorded in a similar way. The weight of each node is recorded by three-dimensional array layer_weight. The first dimension is the number of layers, the second dimension is the position of the node in this layer, and the third dimension is the position of the lower node. The value of the array is the weight value of a node to a node in the lower layer, and the initial value is a random number between 0 and 1. In order to optimize the convergence speed, the momentum method is used to adjust the weight, and the last weight adjustment needs to be recorded with the three-dimensional array layer_weight_delta, and the intercept item is processed: in the program, the value of the intercept is set to 1, so you only need to calculate its weight.

2, forward calculation results: using S function 1 / (1+Math.exp (- z)) to unify the value of each node to 0-1, and then calculate layer by layer until the output layer, for the output layer, in fact, we no longer need to use the S function, we regard the output results as a probability value between 0 and 1, so we also use the S function, which is also conducive to the unity of the program.

3. Reverse modify the weight

How the neural network calculates the error, the square error function E is generally adopted, as follows:

It doesn't matter if we can't deduce it, we just need to use the result formula. In our program, we use layerErr to record the minimum error of E-pair weight derivation, and then adjust the weight according to the minimum error.

Note that the momentum method is used here to take into account the experience of the last adjustment to avoid falling into the local minimum. The following k represents the number of iterations, mobp is the momentum term, and rate is the learning step:

Δ w (k) = mobp* Δ w (k) + rate*Err*Layer

There are also a lot of people who use the following formula, and the effect is not too different:

Δ w (k) = mobp* Δ w (k) + (1-mobp) rate*Err*Layer

In order to improve performance, note that the implementation of the program calculates the error and adjusts the weight at the same time in a while, first locates the position to the penultimate layer (that is, the last hidden layer), then adjusts it layer by layer in reverse, adjusts the weight of layer L according to the calculated error of layer 1, and calculates the error of layer L, which can be used to calculate the weight when cycling to layer 1 next time. This cycle continues until the end of the penultimate layer (input layer).

Conclusion: in the whole process of calculation, we can see that the value of the node is changing every time, it is dynamic and does not need to be saved, the weight parameters and error parameters need to be saved and need to provide support for the next iteration. Therefore, if we conceive a distributed multi-machine parallel computing scheme, we can understand why there is a concept of Parameter Server in other frameworks.

4. Complete program realization of multi-layer neural network.

The following implementation program BpDeep.java can be directly used, and it can also be easily modified to be implemented in any other language, such as cdirection, censor, python, etc., because they are all basic statements used, and no other java libraries are used (except the random function). The following is the original program. Please indicate the author and source when reprinting the reference.

Import java.util.Random;public class BpDeep {public double [] [] layer;// Neural Network Node public double [] [] layerErr;// Neural Network Node error public double [] layer_weight;// Node weight public double [] layer_weight_delta;// Node weight momentum coefficient public double rate / / Learning factor public BpDeep (int [] layernum, double rate, double mobp) {this.mobp = mobp; this.rate = rate; layer = new double [layernum.length] []; layerErr = new double [layernum.length] []; layer_weight = new double [layernum.length] [] [] Layer_weight_delta = new double [layernum.length] [] []; Random random = new Random (); for (int lumbago)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.