Shulou(Shulou.com)06/02 Report--

1. Brief introduction

Neural network has achieved great success in a variety of data processing, including images, text, time series and so on. However, a problem faced by academia or industry is that the process of its work can not be understood in any detail, and its effect can only be tested by experiments, but can not make a reasonable explanation. The related problem is that there is often a certain over-fitting phenomenon for specific data sets, which leads to the possibility of antagonistic behavior. For these reasons, it is well worth trying to develop methods to develop an understanding of the internal state of neural networks. Because of the large number of neurons in the network, it is difficult to analyze the data, especially for unsupervised data analysis.

In this article, we will discuss how to use topological data analysis to gain an in-depth understanding of the working process of convolution neural networks (CNN). The examples given in this article come entirely from networks that train image data sets, but we are sure that topology modeling can easily explain the working process of convolution networks in many other areas.

First of all, for neural networks, it is generally composed of nodes and directed edges. Some nodes are designated as input nodes, others are designated as output nodes, and other nodes are designated as internal nodes. The input node is a feature of the dataset. For example, when processing an image, the input node will be pixels of a specific image format. In text analysis, it may also be a word. Given a dataset and a classification problem, such as a handwritten numeral MNIST dataset, try to classify each image into one of the numbers 0 through 9. Each node of the network corresponds to a variable value (activation value). Therefore, each data point generates a value for each internal and output node in the neural network. The value of each node in the network is determined by the weight system assigned to each edge. The value on node Z is determined by the activation function of node A, B, and C, D, which is connected to it.

Fig. 1 example of neural network node

The activation value of the rightmost node Z is calculated based on the weight values assigned to the four edges. One possible form of function is

Where wA,wB,wC and wD are the weights with the edge AZ,BZ,CZ and DZ, and xA,xB,xC and xD are the activation values at C and D, respectively. The range of values is usually between 0 and 1, and is usually monotonous. The selection of weight is optimized by the output function, given the specific output function of the input (cost function or loss function), and then use the optimization process to select the ownership weight in order to best fit the given output function. Readers who are interested in this area can refer to the gradient descent algorithm and back propagation algorithm.

two。 Understand the weight of the training network

There is a kind of neural network that has achieved good results in the field of image processing, that is, convolution neural network. In this case, the input node is arranged in a square grid corresponding to the pixel matrix for making up the data of the image. The network model consists of a series of layers, and there is a connection between each layer, that is, there is a connection between the node in layer I and the node in layer (I + 1). Different from the general neural network, the typical convolution neural network consists of convolution layer (convolutional layers), sampling layer (Pooling layer) and full connection layer (fully-connected), in which the convolution layer is mainly used to extract image features, the sampling layer is used to reduce the dimension of features, and the full connection layer is used for the final classification purpose. With the processing between layers, the feature graph becomes smaller and smaller, and this phenomenon can also be seen in the graph.

Fig. 2 typical structure of convolution neural network

In order to understand the potential behavior of convolution neural networks, it is necessary to understand the weight matrix. Suppose a dataset in which each data point is a weight matrix associated with neurons in the hidden layer. Collect data from all grids in the fixed layer and train the same data set many times in the same network model. Finally, the topological data of the weight matrix is analyzed.

By performing TDA on the weight matrix, we first understand the behavior of the convolution neural network and independently prove that the convolution neural network completely represents the potential distribution of occurrence in the natural image. How can this be accomplished?

First, you need to find a useful structure from a topological point of view. In order to achieve this goal, only points with high enough density are considered. First look at the first convolution layer in the two-layer convolution neural network, which produces the topology model shown in figure 3.

Fig. 3 TDA Mapper model colored according to filter density

As you can see from the figure, the model is circular. The barcodes shown on the right are persistent homologous barcodes, which are topological signatures, indicating that the dataset actually has this shape, and that it is not built using Mapper to build the model. By marking part of the model with the average value of the corresponding weight matrix, the explanation of the shape is also shown in the image. In addition, what is interesting about this model is that the content found in the study of statistical 3 × 3 image blocks in grayscale natural images is exactly the same as that found in the so-called primary visual cortex.

More simply, the topological model describes CNN in such a way that it can independently confirm that it matches the way humans view the world and matches the density analysis of natural images.

The analysis in figure 3 was done on the MNIST dataset, and the correlation analysis performed on the CIFAR 10 dataset yielded the following figure:

Figure 4 the additional complexity of the CIFAR 10 dataset is shown on horizontal and vertical lines

The above image is an analysis of the first convolution layer, which contains lines in the middle and edges of the area. It is observed that the neurons of these line blocks also exist in the primary visual cortex of mammals, which provides us with a quantitative perspective of vision-related quality.

3. Understand the change of weight in the process of training

From the above findings, we can see that using TDA, convolution neural networks can simulate the distribution of data sets in natural images, so we can turn our attention to what happens in the learning process. Fig. 5 is obtained by calculating the topological models in the first and second layers of the convolution neural network on the CIFAR10 data set, and then displaying the first and second layers of the model in different learning iterations.

Fig. 5 topological model of each stage of neural network model

The model is colored to get the information processed by the model, and the color reflects the number of data points in the node, so the red part can be regarded as the actual model, and the rest contains the weight matrix which does not appear frequently.

The first line of the image reflects the information of the first layer, and it is observed that the circular model is quickly discovered by the optimization algorithm after 400 and 500 iterations. However, with the iteration process, the circle becomes more complex, including patches corresponding to horizontal and vertical patches, and after 1000 iterations, the model center becomes more complex. On the other hand, for the second layer, you can see that there is only one weak pattern in the first round of iterations, but after 2000 iterations, there seems to be a well-defined circular model. We assume that the second layer has "taken over" the first layer, and the first layer has moved to capture more complex patches, which is an area of potential future research. It also demonstrates the ability to use topological data analysis to monitor and provide insight into the learning process of neural networks.

4. Higher-level weight matrix

This approach also applies to deeper networks, which are organized in a way similar to those of visual pathways in humans or primates. It should be understood that this pathway has many components, including the retina, as well as various higher components. The primary visual cortex acts as an edge and line detector, and higher components are used to detect more complex shapes and more abstract. The following figure shows the results of the research on the layers of the trained VGG 16 network. The topological data analysis of the second to thirteenth convolution layers is shown here, and the corresponding topology model is given.

Fig. 6 A 13-layer neural network represented by topological data analysis

Notice that the second and third layers are obviously very similar to the circular model obtained from the model trained on the MNIST dataset. The fourth floor has a circular model, but it also contains some lines in the background. At a higher level, however, very interesting patterns have been developed, including line crossover and "bull's eye".

From these topological models, it can be found that convolution neural network can not only simulate the distribution of real-world data sets, but also simulate the development of mammalian visual cortex.

Although convolutional neural networks are as difficult to understand as a black box, topological data analysis provides a way to understand how calculations are performed in neural networks on a macro scale. Although this work is applicable to image data sets, the use of topological data analysis to explain the calculation of neural networks in other fields is also applicable.

By compressing a large number of states into smaller and more understandable models, topological data analysis can be used to understand the behavior and functions of wide-range neural networks.

Author: Gunnar Carlsson, machine learning researcher

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.