Shulou(Shulou.com)06/02 Report--

Introduction: NeurIPS, the top conference in AI field, is being held in Montreal, Canada. Aiming at several research hotspots in the laboratory, such as model compression, automatic machine learning, machine learning and optimization algorithms, this paper analyzes and interprets 23 key papers selected in the conference and shares them with you. Enjoy!

NeurIPS (Conference on Neural Information Processing Systems, Neural Information processing Systems Progress Conference) and ICML are called the two top academic conferences in the field of neural computing and machine learning. This year is the 32nd session, which will be held in Montreal, Canada from December 3 to 8. Tencent AI Lab participated in NeurIPS for the third time and a total of 20 papers were selected, ranking in the forefront of domestic enterprises.

Tencent AI Lab interpreted 23 papers related to the three hot topics of the conference-model compression, automatic machine learning, machine learning and optimization algorithms.

Model compression

Model Compression

Model compression is a hot research direction in the field of deep learning recently. By simplifying the network structure or weight representation of the model, the computational complexity and parameter scale of the model can be reduced under the premise of controllable loss of accuracy. in order to solve the problem of low computational efficiency of deep learning model. According to rough statistics, there are 15-20 model compression related papers in this NIPS, covering many aspects such as network pruning, weighting, knowledge distillation, streamlining network structure design and so on.

There are four major characteristics in the research of this conference: first, a variety of weight quantization algorithms are proposed, and the corresponding training algorithms and theoretical analysis are also discussed; second, the combination of transfer learning and knowledge distillation; third, the continuous emergence of new simplified network structures; fourth, the application of model compression is extended to target detection, natural language understanding and other fields. In addition, the Symposium on "streamlining Deep Neural Networks and their Industrial applications" (Workshop) focused on the recent research results of model compression and its application in practical scenarios. We focus on the following articles:

1. Sparse Neural Network training based on sensitivity regularization

Learning Sparse Neural Networks via Sensitivity-Driven Regularization

Https://nips.cc/Conferences/2018/Schedule?showEvent=11386

This paper is completed by the cooperation of Turin University of Technology, Nuance Communications, Paris Institute of higher Telecommunications and Telecom Italia. The massive parameters in the deep neural network make it difficult to apply to the scene with limited memory, and the training method of regularization and pruning can make the network weight become sparse to solve this problem. In this paper, the sensitivity of the final output of the network relative to the network weight is quantitatively analyzed, and a regularization term is introduced to gradually reduce the absolute value of the network weight with low sensitivity. Based on this algorithm, a large number of network weights will gradually converge to zero, which can be safely deleted from the network model. The experimental results show that the algorithm outperforms the existing algorithms in terms of weight sparsity and model prediction accuracy; in some data sets, when the model prediction accuracy is the same, the algorithm can increase the weight sparsity to twice that of the existing algorithms.

two。 A highly scalable 8-bit neural network training algorithm

Scalable Methods for 8-bit Training of Neural Networks

Https://nips.cc/Conferences/2018/Schedule?showEvent=11503

This article is written in cooperation between Intel and Israel Institute of Technology. Fixed-point quantization of the trained network model can improve the computational efficiency of the model in the reasoning process, but there is no final conclusion on how to determine the optimal number of quantized bits and quantization scheme. First of all, through theoretical analysis, this paper points out that in the process of network training, except for some specific operations, most of the operations are not sensitive to the decrease of model weight accuracy. Based on this conclusion, this paper proposes to quantify the model weight, each layer feature map and gradient signal, and maintains two gradient signals with different quantization accuracy to maximize the computational efficiency without losing accuracy. At the same time, because the batch normalization layer requires higher quantization accuracy, this paper proposes a Range BN layer to improve the tolerance of quantization errors. The experimental results show that on the ImageNet-1K data set, the model weight, each layer feature map and gradient signal are quantized with 8 bits, and the quantized model achieves the best prediction accuracy at present.

3. Application of Channel pruning algorithm with maximum Discrimination in depth Neural Network

Discrimination-aware Channel Pruning for Deep Neural Networks

Https://nips.cc/Conferences/2018/Schedule?showEvent=11109

This paper is jointly completed by South China University of Technology, University of Adelaide, University of Texas at Arlington and Tencent AI Lab. Most of the existing channel pruning algorithms are retrained based on sparse constraints, or the reconstruction error of the model after pruning is minimized based on the pre-training model, which has some problems such as training difficulties and ignoring model discrimination. In this paper, a channel pruning algorithm with maximum discrimination is proposed to screen and retain the channels that really contribute to discrimination in the model. In order to achieve this goal, the author introduces an additional discrimination loss term to improve the discrimination of each layer in the middle of the network, and then selects the channel of each layer based on the discrimination loss term and reconstruction error term. The experimental results show that on the ILSVRC-12 dataset, the pruned ResNet-50 model based on this algorithm can achieve 0.39% higher top-1 classification accuracy than the benchmark model when 30% channels are deleted.

4. Interpreting complex Model: network Compression based on Factor transfer Learning

Paraphrasing Complex Network:Network Compression via Factor Transfer

Https://nips.cc/Conferences/2018/Schedule?showEvent=11283

This paper is completed by National Seoul University. In this paper, a new knowledge distillation training algorithm is proposed, which interprets and reorganizes the information of the reference model by introducing additional convolution layers, so as to better guide the training of the target model. This process is accomplished by two convolution modules, which are called paraphraser and translator respectively. The interpreter is trained in an unsupervised way to extract the effective information in the reference model, while the translator is located in the target model and is used to transform the corresponding information in the target model to fit the information output by the interpreter of the reference model. The experimental results show that the training effect of the target model based on this algorithm is better than that of the existing knowledge distillation algorithms in image classification and target detection tasks on multiple data sets.

5. Moonshine: knowledge distillation based on reduced convolution operation

Moonshine:Distilling with Cheap Convolutions

Https://nips.cc/Conferences/2018/Schedule?showEvent=11295

This article is written by the University of Edinburgh. In this paper, a structured model distillation algorithm is proposed to reduce the memory overhead during the running of the model. Specifically, based on the network structure of the predefined reference model, after a simple structure transformation, the network structure of the target model can be obtained, thus avoiding redesigning the network structure of the target model. At the same time, the hyperparameter settings of the reference model can be directly reused. Taking the residual model as an example, the computational complexity of the model can be significantly reduced by transforming the two 3x3 convolution layers in the residual unit and replacing the structure of grouped convolution and cross-group mixed convolution. The target model obtained after transformation can be trained based on knowledge distillation and attention transfer learning algorithm. The experimental results show that the algorithm can greatly reduce the memory overhead with very low precision loss; at the same time, compared with the direct training target model, the introduction of knowledge distillation training mechanism can significantly improve the prediction accuracy of the target model.

6. Knowledge Distillation based on online self-Integration

Knowledge Distillation by On-the-Fly Native Ensemble

Https://nips.cc/Conferences/2018/Schedule?showEvent=11723

This article is written in cooperation between Queen Mary University of London and Vision Semantics. Knowledge distillation is often used to train models with low complexity and high generalization ability, but off-line knowledge distillation algorithms often need a high-precision pre-training model as a reference, and the training process needs to be divided into two stages. on-line knowledge distillation algorithm can not effectively construct a high-precision reference model. In this paper, an on-line self-integrated knowledge distillation algorithm is proposed to realize single-stage on-line distillation. Specifically, the algorithm trains a multi-branch network and constructs a reference model online to improve the prediction accuracy of the target model. The experimental results show that the algorithm achieves consistent performance improvement for all kinds of network structures on multiple data sets, and the training efficiency is higher.

7. ChannelNet: an efficient convolution neural network based on channel-by-channel convolution

ChannelNets:Compact and Efficient Convolutional Neural Networks via Channel-Wise Convolutions

Https://nips.cc/Conferences/2018/Schedule?showEvent=11508

This article is written by Texas Austin M University. In this paper, a channel-by-channel convolution operation is proposed to replace the dense connection between feature graphs in CNN model. Based on this channel-by-channel convolution operation, the author proposes ChannelNet, a lightweight network structure. Three types of channel-by-channel convolution operations are used in ChannelNet: packet-by-channel convolution, depth-divisible channel-by-channel convolution and full convolution integral layer. Compared with the previous CNN models (including SqueezeNet, MobileNet and ShuffleNet) commonly used in mobile terminals, ChannelNet significantly reduces the number of parameters and computational overhead of the model without losing performance. In the following figure, (a) is the depth separable convolution, (b) is replacing the 1x1 convolution in the depth separable convolution with the grouped 1x1 convolution, (c) is the grouping channel-by-channel convolution proposed in this paper (to fuse each group of information), and (d) is the depth-separable channel-by-channel convolution proposed in this paper.

8. Pelee: a real-time target detection system based on mobile devices

Pelee:A Real-Time Object Detection System on Mobile Devices

Https://nips.cc/Conferences/2018/Schedule?showEvent=11208

This paper is completed by the University of Western Ontario. In order to realize the deployment of deep learning model on mobile devices, researchers have proposed a series of simplified network structures, including MobileNet, ShuffleNet and NASNet-A, but these models rely heavily on deep separable convolution, and this convolution operation is not implemented efficiently in most deep learning frameworks. In order to solve this problem, this paper constructs an efficient network structure of PeleeNet based on the traditional convolution operation. PeleeNet can be seen as a variant of DenseNet, using a similar connection model and design ideas. Specifically, double convolution channels and an initial convolution unit called stem block are adopted in PeleeNet, and the number of channels in the bottleneck layer is dynamically adjusted to save computational overhead. The network structure of SSD model is optimized and combined with PeleeNet to realize real-time target detection on mobile devices.

Automatic machine learning

Auto Machine Learning

Automatic machine learning (AutoML) is an important branch of machine learning. It mainly refers to the technology of automating the application process of machine learning in practical problems. In a typical machine learning application, practitioners must manually do feature preprocessing, algorithm selection, network structure design (if deep learning algorithm is selected), and super-parameter adjustment in order to maximize the performance of the model. Many of these steps are beyond the capabilities of non-algorithmic experts, so automatic machine learning is proposed to automate these steps. Automatic machine learning makes machine learning easier in the application of practical problems, and it is also easier to get some new models that can exceed the performance of manual design models. According to the preliminary statistics, there are more than 20 papers related to automatic machine learning in this NIPS, covering Bayesian optimization, network structure search, transfer learning and so on.

The research of this conference embodies three major characteristics: first, the research begins to solve the network structure search of more complex tasks (for example, semantic segmentation); second, the integration of small fields has begun. For example, there is a technical research on using Bayesian optimization (traditionally used for hyperparameter optimization) to solve the network structure search. Third, other technologies in the field of non-automatic machine learning are becoming more and more applied, such as transfer learning.

1. Efficient multi-scale structure for predictive search of dense images

Searching for Efficient Multi-Scale Architectures for Dense Image Prediction

Https://nips.cc/Conferences/2018/Schedule?showEvent=11831

Neural network structure design is the key component of the most advanced performance of many machine learning systems in many tasks. A lot of work is devoted to the automatic design and construction of neural network structure by combining cleverly constructed search space with simple learning algorithms. Recent progress shows that this meta-learning method can achieve better performance than the human-designed neural network in image classification. An open question is what kind of performance these methods can achieve in other areas. In this work, we focus on the construction of meta-learning techniques for dense image prediction, focusing on the tasks of scene analysis, human body segmentation and semantic image segmentation. Due to the multi-scale of visual information and the high resolution of the image, it is very challenging to search the appropriate network structure in the dense image prediction task. Based on the investigation of dense image prediction techniques, we construct a recursive search space and prove that even with effective random search, we can find a network structure that is better than human design and achieve the most advanced performance tasks on three dense predictions: including 82.7% accuracy on the Cityscapes (Street View parsing) dataset. The accuracy is 71.3% on PASCAL-Person-Part (human body segmentation) and 87.9% on PASCAL VOC 2012 (semantic image segmentation). In addition, the network structure obtained by our model is more efficient in calculation, which requires only half the parameters and half the computational power compared with the existing technology.

two。 Neural structure optimization

Neural Architecture Optimization

Https://nips.cc/Conferences/2018/Schedule?showEvent=11750

Automatic neural network structure design has shown its potential in discovering powerful neural network architecture. Whether based on reinforcement learning or evolutionary algorithm (EA), the existing technology is to search in discrete space. This is very inefficient. In this paper, we propose a simple and effective automatic neural network structure design method based on continuous optimization. We call this new method neural structure optimization (NAO). Our proposed method has three key components: (1) the encoder maps the neural network architecture to the continuous space (2) the predictor takes the continuous representation of the network as input and predicts its accuracy (3) the decoder maps the continuous representation of the network back to its architecture. Performance predictors and encoders enable us to perform gradient-based optimization in continuous space to find expressions of new architectures with higher accuracy. The decoder then decodes this better embedding into the network. The experimental results show that the architecture found by our method has strong competitiveness for image classification tasks on CIFAR-10 and language modeling tasks on PTB. It is better than or equal to the best results of previous network structure search methods and the computing resources are significantly reduced. Specifically, our error rate is 2.07% on the test set of CIFAR-10 image classification task and 55.9% on the test set of PTB modeling task. The best architecture found on these two tasks can be migrated to other data sets such as CIFAR-100 and WikiText-2. In addition, by using the weight sharing mechanism proposed recently, our model can achieve good results on CIFAR-10 (error rate is 3.53%) and PTB (test set perplexity is 56.3) under the condition of limited computing resources (10 hours of single GPU training).

3. Neural structure search using Bayesian optimization and optimal transportation

Neural Architecture Search with Bayesian Optimisation and Optimal Transport

Https://nips.cc/Conferences/2018/Schedule?showEvent=11214

Bayesian optimization is a kind of method for global optimization of black box function f, in which we can only get the evaluation value of f at some points. This method is usually applied to the evaluation of f when it is very expensive, one of the application cases in machine learning is model selection. Because the generalization performance of the model is difficult to analyze, we generally use noisy and expensive training and verification procedures to select the best model. The traditional Bayesian optimization method focuses on the field of Euclidean and category data, so it can only be used to adjust the hyperparameters of machine learning models in model selection. However, with the surge of interest in deep learning, the need to adjust the network structure is also growing rapidly. In this paper, we design an algorithm called NASBOT, a Bayesian optimization framework for network structure search based on Gaussian process. In order to achieve this goal, we design a metric which can be calculated efficiently through the optimal transportation program for the neural network structure space. This metric may also be useful for deep learning communities other than Bayesian optimization. On several cross-validation-based multilayer perceptron and convolution neural network model selection problems, we show that NASBOT can surpass other network structure search alternatives.

4. Efficient and High-dimensional Bayesian Optimization using Fourier Features

Efficient High Dimensional Bayesian Optimization with Additivity and Quadrature Fourier Features

Https://nips.cc/Conferences/2018/Schedule?showEvent=11859

We design an efficient Bayesian optimization algorithm for high-dimensional black box function optimization. We assume a generalized additive model of overlapping variables. When there is no overlap between variable groups, we can provide the first provable polynomial time algorithm for high-dimensional Bayesian optimization. In order to make the optimization efficient and feasible, we introduce a new deterministic Fourier feature approximation based on numerical integration, and provide a detailed analysis for the square exponential kernel. The error of this approximation decays exponentially with the number of features, and the posterior mean and variance can be accurately approximated. In addition, the computational complexity of the inverse of kernel matrix is reduced from the cube of the number of data points to linear.

5. Regret upper bound of meta-Bayesian optimization with an unknown Gaussian process priori

Regret bounds for meta Bayesian optimization with an unknown Gaussian process prior

Https://nips.cc/Conferences/2018/Schedule?showEvent=11991

Bayesian optimization usually assumes that a Bayesian prior is given in advance. However, due to the unknown parameters in a priori, the strong theoretical guarantee in Bayesian optimization is often impossible to achieve in practice. In this paper, we use a variant of empirical Bayes and show that by using the offline data extracted from the same a priori to estimate a priori Gaussian process and construct a posteriori unbiased estimator, variants based on confidence upper bound and lifting probability can reach a regret upper bound close to zero. With the increase of the number of offline data and online evaluations, the upper bound of regret can be reduced to a constant proportional to the observed noise. In experiments, we successfully verify the proposed method in the challenging problem processing and motion planning of simulated robots.

6. Maximize the function for Bayesian optimization

Maximizing acquisition functions for Bayesian optimization

Https://nips.cc/Conferences/2018/Schedule?showEvent=11938

Bayesian optimization is an efficient sampling method for global optimization, which mainly depends on the acquisition function to guide the search process. The complete maximization of acquisition functions leads to Bayesian decision rules, but because these acquisition functions are usually difficult to optimize, this ideal is difficult to achieve. Especially when evaluating several queries in parallel, because the acquisition function is non-convex, high-dimensional and tricky, it is more difficult to maximize the acquisition function. For this reason, we propose two new methods to maximize the acquisition of functions by using the differentiability of Monte Carlo integrals and the submodules of parallel queries.

7. Probability Matrix decomposition for automatic Machine Learning

Probabilistic Matrix Factorization for Automated Machine Learning

Https://nips.cc/Conferences/2018/Schedule?showEvent=11337

In order to achieve the top performance, modern machine learning technology requires careful data preprocessing and super-parameter adjustment. In addition, due to the growth of the number of machine learning models developed, model selection is becoming more and more important. The selection and adjustment of automated machine learning has always been one of the research goals of the machine learning community. In this paper, we propose to solve this meta-learning task by combining collaborative filtering and Bayesian optimization. Specifically, we use probability matrix decomposition to make use of experiments on hundreds of different data, and then use the obtained function to guide the exploration of possible machine learning process space. In our experiments, we have shown that our approach can quickly identify high-performance processes on a lot of data and can significantly exceed current top methods.

8. Scalable hyperparametric transfer learning

Scalable Hyperparameter Transfer Learning

Https://nips.cc/Conferences/2018/Schedule?showEvent=11660

Bayesian optimization (BO) is a model-based method for gradient-free black box function optimization problems such as hyperparametric optimization. Generally speaking, Bayesian optimization depends on the traditional Gaussian process regression, but the algorithm complexity of the traditional Gaussian process is the cubic level of evaluating the number of tests. Therefore, Bayesian optimization based on Gaussian process can not use a large number of past evaluation and testing experience as a hot start. In this paper, we propose a multi-task adaptive Bayesian linear regression model to solve this problem. It is worth noting that the model only evaluates the linear complexity of the number of tests. In this model, each black box function optimization problem (task) is modeled by a Bayesian linear regression model, and multiple models are coupled by sharing a deep neural network to realize transfer learning. Experiments show that the neural network can learn a characteristic representation suitable for the hot start black box optimization problem, and Bayesian optimization can be faster when the target black box function is learned together with other related signals (such as training loss). The model is proved to be at least one order of magnitude faster than the recently published black box optimization literature.

9. Transfer Learning using Neural automatic Machine Learning

Transfer Learning with Neural AutoML

Https://nips.cc/Conferences/2018/Schedule?showEvent=11799

In this paper, we reduce the computational cost of neural network AutoML through transfer learning. AutoML reduces human intervention by automating machine learning algorithms. Although neural network AutoML is very popular recently because it can automatically design the structure of deep learning network, its computational cost is huge. In order to solve this problem, we propose a neural network AutoML based on transfer learning to make full use of the knowledge learned on previous tasks to speed up the design of network structure. We extend the network structure search method based on reinforcement learning to support parallel training on multiple tasks, and transfer the search strategy to the new task. In the task of natural language and image classification, this method reduces the convergence time of single-task training by at least one order of magnitude.

Machine Learning and Optimization algorithm

Optimization For Learning

As we all know, the relationship between machine learning and optimization algorithms is inseparable. In the process of machine learning modeling, there will be various minimum optimization models. In the process of learning parameters, a variety of optimization algorithms are used. Machine learning algorithms and optimization algorithms and their crossover have been the focus of research over the years. In the articles included in this nips, there are also a lot of work on the modeling of machine learning and the design of model training algorithms.

1. Adaptive Methods for Nonconvex Optimization

Https://papers.nips.cc/paper/8186-adaptive-methods-for-nonconvex-optimization.pdf

This article is written in cooperation between google research and Carnegie Mellon University. In this paper, it is proved that the global convergence can be guaranteed by adding batch-size,ADAM and RMSProp. Based on this, a new adaptive random gradient algorithm YOGI for training depth neural networks is also proposed in this paper. By adding batch-size, the global convergence of YOGI is also proved, and the experimental results show that YOGI has faster training speed and higher testing accuracy than ADAM. It is noted that the author of this paper has pointed out that ADAM and RMSProp are divergent at the theoretical level, and thus obtained ICLR2018 Best Paper Award.

Figure1. Comparison of algorithm Framework between ADAM and YOGI

Figure2. Comparison of testing accuracy of ResNet on CIFAR10 dataset

Figure3. Comparison of training loss and test loss between YOGI and ADAM on the deep automatic codec model

2. Spider: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator

Https://papers.nips.cc/paper/7349-spider-near-optimal-non-convex-optimization-via-stochastic-path-integrated-differential-estimator.pdf

This article is completed in cooperation between Peking University and Tencent AI Lab. The main contribution of this paper is mainly at the theoretical level. An algorithm SPIDER is proposed to find the first-order and second-order stable points of nonconvex stochastic optimization. In this paper, it is proved that the computational complexity of SPIDER algorithm is the best in solving non-convex stochastic optimization algorithms. In addition, a new analysis technique of computational complexity is used in this paper, which has strong portability and can be applied to the analysis of many other random optimization algorithms.

Figure1. SPIDER algorithm framework

Figure2. Comparison of computational complexity of different algorithms

3. Escaping Saddle Points in Constrained Optimization

Https://papers.nips.cc/paper/7621-escaping-saddle-points-in-constrained-optimization.pdf

This article is completed independently by the Massachusetts Institute of Technology. In this paper, a general algorithm framework is proposed for constrained non-convex smooth optimization problems, and it is proved for the first time that the algorithm framework can escape the saddle point of constrained non-convex optimization problems. It is worth explaining that this paper is the first to propose an algorithm that can escape the saddle point of constrained non-convex optimization problems.

4. Online Adaptive Methods, Universality and Acceleration

Https://papers.nips.cc/paper/7885-online-adaptive-methods-universality-and-acceleration.pdf

This paper is written in cooperation between the Federal Institute of Technology Zurich and the Federal Institute of Technology Lobsang. In this paper, a new class of adaptive stochastic gradient algorithms is proposed to solve unconstrained stochastic convex optimization problems by AcceleGrad. Novel weighted adaptive learning rate estimation strategy and Nesterov acceleration strategy are adopted to improve the efficiency of AcceleGrad algorithm. The author theoretically proves that the algorithm can achieve the optimal computational complexity. In addition, the author illustrates the superiority of AcceleGrad algorithm from experiments.

Figure 1. AcceleGrad algorithm framework

Figure2. The effect of AcceleGrad algorithm is compared with SGD and AdaGrad.

5. Multi-Task Learning as Multi-Objective Optimization

Https://papers.nips.cc/paper/7334-multi-task-learning-as-multi-objective-optimization.pdf

This paper is independently completed by Intel laboratory. The multitask learning problem is usually modeled as a minimization problem, in which the loss function of the optimization problem is a linear combination of the loss functions of multiple tasks. In this paper, the author creatively models multi-task learning directly into a multi-objective optimization problem.

And a kind of gradient algorithm for solving the above multi-objective optimization problems is proposed. Finally, the author illustrates the superiority of multi-task learning algorithm based on multi-objective optimization through experiments.

Figure1. Gradient algorithm Framework for Multi-objective Optimization problems

Figure2. Comparison of the effects of different algorithms (the smaller the better)

6. Semidefinite relaxations for certifying robustness to adversarial examples

Https://papers.nips.cc/paper/8285-semidefinite-relaxations-for-certifying-robustness-to-adversarial-examples.pdf

This article is completed independently by Stanford University. The attack and defense of neural network is a research hotspot in recent years. In this paper, the defense capability of neural network against attack samples is determined based on the positive semidefinite relaxation technique in optimization. The positive semidefinite relaxation technique proposed in this paper verifies the defensive robustness of arbitrary ReLU neural networks, and theoretically shows that the positive semidefinite relaxation technique adopted in this paper is more accurate than the linear programming relaxation technique, thus obtaining better robustness estimation.

Figurre1. Comparison of relaxation effects of different relaxation techniques

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.