Shulou(Shulou.com)06/01 Report--

This article shows you how to achieve deep learning algorithm through TensorFlow and apply it to enterprise practice. The content is concise and easy to understand, which will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

What is deep learning?

The term deep learning has been heard many times, what it is, and the technology behind it actually originated from the neural network. The neural network was first inspired by the working principle of the human brain. We know that the human brain is a very complex structure. It can be divided into many regions, such as the auditory center and the visual center. When I was reading the research center, there was a computer vision lab for video, language for language, speech for speech, and different functions have been separated in the division of disciplines. This has something to do with our human understanding of the brain. Scientists later discovered that the human brain is a general computational model.

In such an experiment, the scientists cut off the nerve and ear pathway in the mouse's auditory center, and the visual input was connected to the auditory center, and after a few months, the mice could process visual signals through the auditory center. This means that the human brain works the same way, and neurons work the same way, but need constant training. Based on this assumption, neuroscientists have made such an attempt, hoping to give blind people the hope of seeing the world again. They are equivalent to attaching electrodes to their tongues and transmitting different pixels to their tongues through cameras, making it possible for blind people to see the world through their tongues. With a further understanding of the working principle of human nerves, we can see that deep learning is expected to become a general learning model.

The above figure shows the approximate structure of the neural network. In the picture, there are human neurons on the left and neural network neurons on the right. The neurons of neural networks were first inspired by the structure of human neurons, and tried to model the working mode of human neurons. The specific technology is not discussed too deeply here. The bottom side of the picture above shows the comparison between the human neural network and the artificial neural network (Artificial Neural Network). In the computer neural network, we need to clearly define the input layer and the output layer. Reasonable use of the input and output of artificial neural network can help us to solve practical problems.

The core working principle of neural network is to convert a given input signal into an output signal, so that the output signal can solve the problems that need to be solved. For example, when we complete the problem of text classification, we need to divide the article into sports or art. Then we can provide the words in the article as input to the neural network, and the output nodes represent different types. Which category should the article belong to, then we want the output value of the corresponding output node to be 1 and the other output value to be 0. By reasonably setting the structure of the neural network and training the parameters of the neural network, the trained neural network model can help us to judge which category an article should belong to.

Application of Deep Learning in Image recognition

Deep learning, its initial application, lies in image recognition. The most classic application is the Imagenet dataset.

ImageNet is a very large dataset with 15 million images in it. The following figure shows a sample picture in the dataset.

Before the application of deep learning algorithm, the ability of traditional machine learning method to image processing is limited. By 2012, the best machine learning algorithms can achieve an error rate of 25%, and it will be difficult to make new breakthroughs. In 2012, deep learning was first applied to ImageNet datasets, directly reducing the error rate to 16%. In the following years, with the improvement of deep learning algorithm, the error rate has been reduced to 3.5% in 2016. In the ImageNet dataset, the error rate of human classification is about 5.1%. We can see that the error rate of machines is lower than that of human beings, which is a technological breakthrough brought about by deep learning.

What is TensorFlow?

TensorFlow is a deep learning framework that Google opened up last November. We mentioned AlphaGo at the beginning, and its development team, DeepMind, has announced that all subsequent systems will be implemented based on TensorFlow. TensorFlow is a very powerful open source deep learning open source tool. It can support mobile, CPU, GPU and distributed clusters. TensorFlow is widely used in both academia and industry. In industry, TensorFlow-based Google Translation, Google RankBrain and other systems have been launched. In academic circles, many of my classmates in CMU and Peking University said that TensorFlow is the preferred tool for them to implement deep learning algorithms.

The above ppt gives a simple example of a TensorFlow program that implements the function of vector addition. TensorFlow provides API for Python and C++, but Python's API is more comprehensive, so most TensorFlow programs are implemented through Python. In the first line of the above program, we load the TensorFlow through import. All the data in TensorFlow is stored in the way of Tensor. To calculate the specific value of the data in the tensor, we need to use a session (session).

The second line in the above code shows how to generate a session. The session manages the computing resources needed to run a TensorFlow program. A special tensor in TensorFlow is variable (tf.Variable). Before using variables, we need to explicitly call the process of variable initialization. In the last line of the code above, we can see that to get the value of the result tensor output, we need to explicitly call the process of calculating the tensor value.

It is very simple to realize neural network through TensorFlow. The problem of handwritten digit recognition in MNIST can be realized within 10 lines through TFLearn or TensorFlow-Slim. The above ppt shows TensorFlow's support for different neural network structures, and you can see that TensorFlow can support a variety of major neural network structures in a very short amount of code.

Although TensorFlow can quickly realize the function of neural network, the stand-alone version of TensorFlow is difficult to train large-scale deep neural network.

This picture shows the Inception-v3 model proposed by Google in 2015. This model can achieve 95% accuracy on ImageNet data sets. However, there are 25 million parameters in this model, and classifying an image requires 5 billion additions or multiplications. Even if only using such a large-scale neural network already requires a very large amount of computation, if we need to train the deep neural network, then we need a larger amount of computation. The optimization of neural network is complex, there is no direct mathematical method to solve it, and it needs to be iterated repeatedly. It will take more than 5 months to train the Inception-v3 model to 78% accuracy on a single machine. It will take years to train to 95% accuracy. This is completely intolerable for the actual production environment.

TensorFlow on Kubernetes

As we mentioned above, it is impossible to train large neural networks in a stand-alone environment. Inside Google, Google Brain and TensorFlow are running on Google's internal cluster management system, Borg. When I was at Google e-commerce, the commodity classification algorithm we used ran on more than 1,000 servers. Outside of Google, we can run TensorFlow on Kubernetes. Before introducing how to run TensorFlow on Kubernetes, let's introduce how to parallelize the model of training deep learning.

There are two kinds of distributed training methods commonly used in deep learning model. One is synchronous update, the other is asynchronous update. As shown in the ppt above, in synchronous update mode, all servers read the values of the parameters uniformly, calculate the parameter gradient, and then update them uniformly. In asynchronous update mode, different servers read parameters, calculate gradients, and update parameters without synchronizing with other servers. The biggest problem with synchronous updates is that different servers need to complete all operations synchronously, so fast servers need to wait for slow servers, and resource utilization will be relatively low. On the other hand, asynchronous mode may use old gradient update parameters to affect the effect of training. Different update models have their own advantages and disadvantages, it is difficult to say which is better, it needs specific analysis of specific problems.

No matter which update method is used, using the distributed TensorFlow training deep learning model requires two types of servers, one is the parameter server, and the other is the computing server. The parameter server manages and saves the values of the neural network parameters, and the calculation server is responsible for calculating the gradient of the parameters.

There are also two modes to start the training task of distributed deep learning model in TensorFlow. One is In-graph replication. In this mode, the parameters of the neural network will be saved in the same TensorFlow graph, and only the calculations will be assigned to different computing servers. The other is Between-graph replication, where all computing servers also create parameters, but the parameters are assigned to the parameter servers in a uniform manner. Because In-graph replication has a slightly weaker ability to handle large amounts of data, Between-graph replication is a more common pattern.

Last question, we just mentioned that TensorFlow supports running in a distributed cluster, so why do you need Kubernetes? If we make a simple analogy between TensorFlow and Hadoop system, we can clearly explain this problem. As we all know, Hadoop systems can be divided into Yarn, HDFS and mapreduce computing frameworks, so TensorFlow is only part of the Mapreduce computing framework in Hadoop systems.

TensorFlow does not have a scheduling system like Yarn, nor a storage system like HDFS. This is the part that Kubernetes needs to solve. Kubernetes can provide functions such as task scheduling, monitoring, failure restart and so on. Without these features, it is difficult to manually start the TensorFlow server on each machine and monitor the running status of the task from time to time. In addition, distributed TensorFlow currently does not support lifecycle management, and the finished training process is not automatically shut down, which requires additional processing.

The above content is how to implement the deep learning algorithm through TensorFlow and apply it to the enterprise practice. Have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.