Shulou(Shulou.com)05/31 Report--

This article shows you how to learn Spark and TensorFlow deeply. The content is concise and easy to understand. It will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

In the past few years, the field of neural network has developed rapidly, and it is now the strongest in the field of image recognition and automatic translation. TensorFlow is a new framework for numerical computing and neural networks released by Google. In this article, we will demonstrate how to use TensorFlow and Spark to train and apply a deep learning model.

You may wonder: what is the use of Spark in the current era of the highest performance deep learning implementation or single node? To answer this question, we will demonstrate two examples and explain how to use Spark and machine clusters with TensorFlow to increase the number of channels for deep learning.

Super parameter adjustment

A typical application of deep machine learning (ML) technology is artificial neural network. They take a complex input, such as image or audio recording, and then apply complex mathematical transformations to these signals. The output of this transformation is a digital vector that is easier to process by other ML algorithms. Artificial neural networks perform this conversion by mimicking neurons in the visual cortex of the human brain (in a fairly simplified form).

As humans learn to interpret what they see, artificial neural networks need to be trained to identify "interesting" specific patterns. It can be a simple pattern, such as an edge, a circle, but it can also be a more complex pattern. Here, we will use the classic data set provided by NIST to train the neural network to recognize these numbers:

The TensorFlow library will automatically create neural network training algorithms of various shapes and sizes. The actual process of building a neural network is much more complex than simply running some algorithms on a data set. There are usually some very important hyperparameters (generally speaking, parameter configuration) that need to be set, which will affect how the model is trained. Choosing the right parameters can lead to superior performance, while bad parameters will lead to long training and poor performance. In practice, machine learning practitioners will repeatedly run the same model with different hyperparameters in order to find the best set. This is a classic technique called hyperparameter adjustment.

When building a neural network, there are many important hyperparameters that need to be carefully selected. For example:

Number of neurons per layer: too few neurons will reduce the expressive ability of the network, but too many will greatly increase the running time and return fuzzy noise.

Learning speed: if it is too high, the neural network will only focus on seeing a few examples of the past and ignore the previous experience. If it is too low, it will take a long time to reach a good state.

What is interesting here is that even if the TensorFlow itself is not distributed, the superparameter adjustment process is "embarrassingly parallel" and can be allocated using Spark. In this case, we can use Spark to broadcast common elements, such as data and model descriptions, and then arrange individuals in the machine cluster to perform independent repetitive calculations in a way that allows errors.

How to use Spark to improve accuracy? Set the precision to 99.2% with the default hyperparameter. Our best result on the test set is 99.47% accuracy, which reduces the test error by 34%. Distributed computing time is linearly related to the number of nodes added to the cluster: using a cluster with 13 nodes, we can train 13 models in parallel, which is 7 times faster than training one by one on the same machine. Here is a graph of the computing time (in seconds) relative to the number of machines on the cluster:

Most importantly, we analyzed the sensitivity of a large number of hyperparameters in the training process. For example, we draw the final test performance graph relative to the learning rate of different numbers of neurons:

This shows a typical neural network tradeoff curve:

Learning speed is critical: if it is too low, the neural network does not learn anything (high test error). If it is too high, random oscillations may occur in the training process or even divergence in some configurations.

The number of neurons is less important for achieving good performance, and the learning rate of networks with more neurons is more sensitive. This is the Occam razor principle: simple models are often "good enough" for most goals. Unless you invest a lot of time and resources in training and find the right hyperparameters to remove the missing 1% test error, this will be different.

By using parameter sparse samples, we can achieve zero error rate under the optimal parameter set.

How should I use it?

Although TensorFlow can use all the cores on each worker, each worker can only run one task at a time, and we can package them to limit competition. The TensorFlow library can be installed as a normal Python library on the Spark cluster as directed on [instructions on the TensorFlow website] (https://www.tensorflow.org/get_started/os_setup.html). The following notes show how the user installs the TensorFlow library and repeats the article's experiment:

Using TensorFlow for distributed Image processing

Testing distributed Image processing with TensorFlow

Large-scale deployment

The TensorFlow model can embed complex identification tasks on data sets directly in the pipeline. As an example, we will show how we can label a set of pictures using a trained stock neural network model.

First, the model is distributed to the worker in the cluster using Spark's built-in broadcast mechanism:

With gfile.FastGFile ('classify_image_graph_def.pb',' rb') as f: model_data = f.read () model_data_bc = sc.broadcast (model_data)

After that, the model is loaded on each node and applied to the picture. This is the code framework that each node runs:

Def apply_batch (image_url): # Creates a new TensorFlow graph of computation and imports the model with tf.Graph (). As_default () as g: graph_def = tf.GraphDef () graph_def.ParseFromString (model_data_bc.value) tf.import_graph_def (graph_def, name='') # Loads the image data from the URL: image_data = urllib.request.urlopen (img_url Timeout=1.0) .read () # Runs a tensor flow session that loads the with tf.Session () as sess: softmax_tensor = sess.graph.get_tensor_by_name ('softmax:0') predictions = sess.run (softmax_tensor, {' DecodeJpeg/contents:0': image_data}) return predictions

By packaging the pictures together, this code can run faster.

Here is an example of the picture:

This is the neural network's interpretation of this picture, which is quite accurate:

('coral reef', 0.88503921), (' scuba diver', 0.025853464), ('brain coral', 0.0090828091), (' snorkel', 0.0036010914), ('promontory, headland, head, foreland', 0.0022605944)]) the above is how to learn Spark and TensorFlow deeply. Have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.