Shulou(Shulou.com)06/02 Report--

Introduction: AlphaGo became Master in early 2017, sweeping Chinese, Japanese and South Korean go masters on platforms such as Yicheng and Wild Fox, achieving a 60-game winning streak without losing. The mysterious driving force behind AlphaGo is the machine learning and deep learning framework that TensorFlow--Google opened up in November 2015.

TensorFlow received a great deal of attention as soon as it appeared at the end of 2015, gaining more than 10, 000 stars on GitHub in one month. It currently ranks first among all machine learning and deep learning programs, and even among all Python projects. This article will give us a brief look at TensorFlow and compare it with other mainstream deep learning frameworks.

TensorFlow

TensorFlow is a relatively high-level machine learning library, which users can easily use to design neural network structures without having to write C++ or CUDA code for efficient implementation. Like Theano, it supports automatic derivation, and users no longer need to solve the gradient through back propagation. Its core code, like Caffe, is written in C++, which simplifies the complexity of online deployment and allows devices like mobile phones, which are short of memory and CPU resources, to run complex models (Python is more resource-consuming and inefficient). In addition to the C++ interface of the core code, TensorFlow also has official Python, Go and Java interfaces, which are implemented through SWIG (Simplified Wrapper and Interface Generator), so that users can experiment with Python on a machine with better hardware configuration, and deploy the model with C++ in embedded environments where resources are tight or in environments that need low latency. SWIG supports providing interfaces in a variety of languages to Cmax Candle + code, so interfaces for other scripting languages can be easily added through SWIG in the future. However, a problem that affects the efficiency when using Python is that every mini-batch has to feed from the Python to the network. When the amount of data in the mini-batch is very small or the computing time is very short, it may cause a relatively large delay. Now TensorFlow also has unofficial Julia, Node.js, R interface support.

TensorFlow also has built-in upper-layer components such as TF.Learn and TF.Slim, which can help to quickly design new networks, and are compatible with Scikit-learn estimator interfaces, and can easily achieve evaluate, grid search, cross validation and other functions. At the same time, TensorFlow is not only limited to neural networks, its data flow graph supports very free algorithm expression, of course, it can also easily implement machine learning algorithms other than deep learning. In fact, TensorFlow can be used as long as the calculation can be represented in the form of a computation graph. Users can write inner loop code to control the calculation of the branch of the calculation graph, and TensorFlow will automatically turn the related branches into subgraphs and perform iterative operations. TensorFlow can also assign each node in the calculation diagram to different devices for execution, making full use of hardware resources. To define a new node, you only need to write a Python function. If there is no corresponding underlying operation core, you may need to write C++ or CUDA code to implement the operation.

In the data parallel mode, TensorFlow is very similar to Parameter Server, but TensorFlow has a separate Variable node, unlike other frameworks that have a globally unified parameter server, so parameter synchronization is more free. The core of TensorFlow and Spark is a flow graph of data computing. Spark is oriented to large-scale data and supports SQL and other operations, while TensorFlow is mainly oriented to the environment with sufficient memory to load model parameters, which can maximize computational efficiency.

Another important feature of TensorFlow is its flexible portability, which makes it easy to deploy the same code to a PC, server, or mobile device with any number of CPU or GPU without modification. Another advantage over Theano,TensorFlow is its extremely fast compilation speed. Theano usually takes a long time to compile when defining a new network structure, so it takes a lot of cost to try a new model, but TensorFlow has no this problem at all. TensorFlow also has a powerful visualization component TensorBoard, which can visualize the network structure and training process, which is very helpful for observing complex network structure and monitoring long-time and large-scale training. TensorFlow is highly optimized for the production environment, and its product-level high-quality code and design can ensure stable operation in the production environment. At the same time, once TensorFlow is widely used by industry, it will produce a virtuous circle and become the de facto standard in the field of deep learning.

In addition to supporting common network structures such as convolution neural network (Convolutional Neural Network,CNN) and cyclic neural network (Recurent Neural Network,RNN), TensorFlow also supports deep reinforcement learning and even other computing-intensive scientific computing (such as solving partial differential equations, etc.). TensorFlow previously does not support symbolic loop and needs to use Python loops for graph compilation and optimization, but recently the newly added XLA has begun to support JIT and AOT. In addition, it can also implement cyclic neural networks more efficiently using bucketing trick. One weakness of TensorFlow may be that computation diagrams must be built as static diagrams, which makes many calculations difficult to implement, especially beam search, which is often used in sequence prediction.

TensorFlow users can easily deploy the trained model to a variety of hardware and operating system platforms, CPU that supports Intel and AMD, GPU of NVIDIA through CUDA (recently started to support GPU of AMD through OpenCL, but not as mature as CUDA), Linux and Mac, and recently began to support Windows in version 0.12. In the industrial production environment, some hardware devices are up-to-date, some are old models that have been used for several years, and the sources may be complicated. The heterogeneity of TensorFlow enables it to fully support a variety of hardware and operating systems. At the same time, its matrix operation library on CPU uses Eigen instead of BLAS library, and can be compiled and optimized based on ARM architecture, so it performs well on mobile devices (Android and iOS).

TensorFlow initially supported stand-alone, only CUDA 6.5 and cuDNN v2, and there was no official comparison with other deep learning frameworks. At the end of 2015, many other frameworks made a variety of performance comparisons, each time TensorFlow appeared as a poor control group. At that time, TensorFlow was really unhappy, and its performance was only comparable to that of Theano, which is generally considered to be very slow, and it can be regarded as the bottom of all frameworks. However, with the strong development strength of Google, the new version of cuDNN (currently supporting cuDNN V5.1) was soon supported, and the performance on a single GPU caught up with other frameworks. The following figure shows the performance evaluation of a single GPU on AlexNet for each framework given by https://github.com/soumith/convnet-benchmarks.

At present, under the condition of single GPU, most deep learning frameworks rely on cuDNN, so as long as there is little difference in hardware computing power or memory allocation, the final training speed will not be too different. However, for large-scale deep learning, the huge amount of data makes it difficult for a single machine to complete the training in a limited time. At this time, distributed computing is needed to make GPU cluster and even TPU cluster parallel computing to train a model together, so the distributed performance of the framework is very important. TensorFlow opened up the distributed version in April 2016. it can achieve 15 times the speed of a single GPU with 16 GPU and 40 times faster when 50 GPU. The distributed efficiency is very high. At present, there are not many distributed deep learning frameworks that are natively supported, only TensorFlow, CNTK, DeepLearning4J, MXNet and so on. However, at present, the design of TensorFlow is not very good for the communication between different devices, its stand-alone reduction can only be processed by CPU, and the distributed communication uses socket-based RPC instead of faster RDMA, so its distributed performance may not be optimal.

Google opened up TensorFlow Serving19 in February 2016. this component can export TensorFlow-trained models and deploy them as RESTful interfaces that can provide prediction services, as shown in figure 2-2. With this component, TensorFlow can implement the whole process of applying machine learning: from training model, debugging parameters, to packaging model, and finally deploying services, which is really a framework with a complete pipeline from research to production. Here quote the description of TensorFlow internal developers: "TensorFlow Serving is a high-performance machine learning service system designed for the production environment. It can run multiple large-scale deep learning models at the same time, support model life cycle management, algorithm experiments, and can make efficient use of GPU resources, so that the TensorFlow-trained model can be put into the actual production environment more quickly and easily." Other frameworks except TensorFlow lack the consideration of deployment for production environment, and Google, as a giant that widely applies deep learning in real products, may also be aware of this opportunity, so it has developed a platform for deploying services. TensorFlow Serving can be said to be a trump card that will make a great contribution to TensorFlow becoming an industry standard.

TensorFlow Serving architecture

TensorBoard is a set of Web applications of TensorFlow to monitor the running process of TensorFlow, or to visualize Computation Graph. TensorBoard currently supports five kinds of visualization: scalar (scalars), picture (images), audio (audio), histogram (histograms) and computational graph (Computation Graph). TensorBoard's Events Dashboard can be used to continuously monitor key runtime indicators, such as loss, learning rate (learning rate), or accuracy on the verification set (accuracy); Image Dashboard can display the saved images set by users during training, such as those drawn with Matplotlib (plot) for a training intermediate result; and Graph Explorer can fully display a calculation chart of TensorFlow and support zooming and dragging and viewing node attributes. The visualization of TensorBoard is shown in figures 2-3 and 2-4.

Visualization of loss Scalar in TensorBoard

Model structure Visualization of TensorBoard

Comparison of mainstream deep learning frameworks

TensorFlow has production-level high-quality code, strong development and maintenance capabilities of Google, and excellent overall architecture design. It is more mature and sophisticated than its established Python-based rival Theano,TensorFlow, and many of the major developers of Theano have gone to Google to develop TensorFlow (such as Ian Goodfellow, author of book Deep Learning, who later went to OpenAI). As a giant company, Google has much more resources devoted to the research and development of TensorFlow than universities or individual developers. It can be predicted that the future development of TensorFlow will be rapid, which may leave the deep learning framework maintained by universities or individuals far behind.

The upsurge of deep learning research continues, and a variety of open source deep learning frameworks emerge in endlessly, including TensorFlow, Caffe, Keras, CNTK, Torch7, MXNet, Leaf, Theano, DeepLearning4, Lasagne, Neon, and so on. However, TensorFlow is out of the tight siege, in the attention and the number of users occupy an absolute advantage, there is a tendency to unify the world.

The figure above shows the data statistics of each open source framework on GitHub (statistics on January 3, 2017). You can see that TensorFlow outperforms its competitors in terms of the number of star, fork and contributor. The main reason is that Google's appeal in the industry is indeed strong, and there are many successful open source projects before, as well as the strong artificial intelligence research and development level of Google, all of which make people confident in Google's deep learning framework, so that TensorFlow has accumulated 10000 + star in the first month of open source in November 2015. Secondly, TensorFlow does have excellent performance in many aspects, such as the simplicity of the code for designing neural network structure, the execution efficiency of distributed deep learning algorithms, and the convenience of deployment. If you keep an eye on the progress of TensorFlow development, you will find that basically every week TensorFlow will have more than 10, 000 lines of code updates, as many as tens of thousands of lines. The excellent quality of the product itself, rapid iterative updates, active community and positive feedback have formed a virtuous circle, and it can be expected that TensorFlow will continue to take the lead in various deep learning frameworks in the future.

Observation shows that giants such as Google, Microsoft and Facebook are involved in this deep learning framework war, as well as Caffe led by Jia Yangqing, a graduate of the University of Berkeley, Theano developed by the Lisa Lab team of the University of Montreal, and frameworks contributed by other individuals or business organizations. In addition, we can see that all the major frameworks basically support Python. At present, Python can be said to take the lead in the field of scientific computing and data mining. Although there is competitive pressure from R, Julia and other languages, various libraries of Python are too perfect. Web development, data visualization, data preprocessing, database connections, crawlers and so on are omnipotent, and there is a perfect ecological environment. Only in the data mining tool chain, Python has NumPy, SciPy, Pandas, Scikit-learn, XGBoost and other components, so it is very convenient to do data acquisition and pre-processing, and the subsequent model training phase can be perfectly connected with TensorFlow and other Python-based deep learning frameworks.

Below, the mainstream deep learning frameworks TensorFlow, Caffe, CNTK, Theano and Torch are scored in each dimension.

This article is selected from "TensorFlow practice". Click this link to view the book on the official website of the blog.

For more wonderful articles in time, search for "blog viewpoints" on Wechat or scan the QR code below and follow.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.