Shulou(Shulou.com)06/02 Report--

Introduction: truly valuable achievements will not be forgotten by time.

Lei Feng website AI Science and Technology Review: recently, NeurIPS 2019 has released the list and ICLR 2020 has been closed. There must be a lot of students who are unhappy about missing the paper and depressed about submitting (or rewriting). However, students who care about real academic contributions need not worry. At the top of the paper, the most important thing is to show that it is in line with the trend of fashion and good luck, just as the best papers at the top of each year are often not papers that really promote progress in the field ten years later.

In fact, not only are the "best papers at the top meeting often not papers that really promote progress in the field", there are many important papers that promote progress that are not submitted, or even submitted and then rejected. Colin Raffel, a Google brain researcher, discussed the matter on Twitter, along with several other scholars to list a number of important papers that have promoted progress in the field but do not belong to any top meeting.

Generating Sequences With Recurrent Neural Networks

Generating sequences with RNN

Https://arxiv.org/abs/1308.0850

This paper was quite amazing when it was published, showing for the first time that it is possible to generate satisfactory text paragraphs or handwritten text directly with RNN (specifically, using LSTM to capture the structure of discrete long sequences, predicting the next element at a time). In addition, there are rudiments of attention mechanism, Adam and other techniques that have been widely used in this paper.

WaveNet: A Generative Model for Raw Audio

WaveNet: a Model for generating original Audio signal

Https://arxiv.org/abs/1609.03499

The famous WaveNet paper from DeepMind can be said to usher in a new era. In the previous speech generation model, the voice code was transformed into a "voice code", and then a separate voice model was used to turn the sound code into an audio waveform signal. WaveNet directly shows that we can now (2016) directly use the depth neural network to generate audio waveform signals, skip the sound code this link, and the generation effect has been greatly improved. Along this direction, later researchers have also made a lot of improvements and new explorations, and parallel WaveNet (Parallel WaveNet, arxiv.org/abs/1711.10433), which greatly improves the speed of voice generation, has soon entered Google's commercial system.

Learning to Generate Reviews and Discovering Sentiment

Learn to generate comments and explore emotions

Https://arxiv.org/abs/1704.01444

A simple and surprising result (thresholding a neuron in an unsupervised LM could classify sentiment accurately) that helped kicked off the transfer learning craze in NLP.

This paper uses a simple unsupervised pre-training method to learn text representation, and then gets a surprising result: the emotion of the text can be accurately judged according to the threshold of a single neuron in the unsupervised language model. This study also contributes to the popularity of transfer learning methods in the field of NLP.

Implicit Autoencoders

Implicit automatic encoder

Https://arxiv.org/abs/1805.09804

The concept of variable automatic encoder (VAE) has been proposed for a long time, but this paper discusses a new form of automatic encoder: the reconstruction term and regularization term in the encoder are expressed by anti-loss, that is, implicit parameterization. Compared with the previous explicit methods, implicit parameterization and implicit data distribution can enable the automatic encoder to learn more expressive prior knowledge and conditional likelihood distribution, thus, the hidden space in the automatic encoder can focus more on capturing abstract and high-dimensional information in the data, while the rest of the low-dimensional information has been included by the implicit conditional likelihood distribution. The models performed well in the experiments of style and content decoupling of the authors.

Learning Dexterous In-Hand Manipulation

Learn flexible manipulator control

Https://arxiv.org/abs/1808.00177

This paper from OpenAI has been a hot topic in the field of robot control since it was published. In fact, the AI technology review of Lei Feng.com has also made a detailed interpretation. OpenAI not only puts forward two or eight difficult problems (manipulator terminal control, manipulator acquisition and playing with objects, which are difficult to solve directly by early reinforcement learning algorithm), but also uses new reinforcement learning to train agents that can accomplish these tasks in a simulated environment. The most amazing thing is that even if it is completely trained in the simulator, the model can be transferred directly to the real manipulator without any fine-tuning, and the task can be accomplished equally gracefully. This is not only a breakthrough in reinforcement learning robot control, but also a reference to their skills when using reinforcement learning in other tasks.

Evolution Strategies as a Scalable Alternative to Reinforcement Learning

Evolutionary strategy is an expandable alternative to reinforcement learning.

Https://arxiv.org/abs/1703.03864

This paper is a pioneering thesis in the research direction of evolutionary strategy Evolution Strategies. The evolutionary strategy was proposed as a supplement to the popular reinforcement learning methods such as Q-learning and policy gradient based on Markov decision process at that time, but in fact, even though the evolutionary strategy was a black box optimization model. It still has many advantages: it is scalable on multi-CPU clusters, insensitive to action frequency and delay feedback, can perform very long-term tasks, and does not require time discounts or value function approximation.

Distilling the Knowledge in a Neural Network

Knowledge in Distillation Neural Network

Https://arxiv.org/abs/1503.02531

First of all, there are Jeff Dean and Geoffrey Hinton among the authors of this paper. It can be said that there must be some valuable insights in this paper when you see these two names. However, Jeff Dean revealed in a Twitter discussion that the paper was submitted to NIPS 2014 and was rejected, with two of the three reviewers saying that "the improvement in this work is so small that it is likely to have no impact." It's depressing, isn't it? as we all know in 2019, as models with millions of parameters emerge one after another, the methods of knowledge distillation and model compression are not only useful. in many cases, it is even an indispensable link in practical application (in order to achieve acceptable delay and power consumption); knowledge distillation has also become a hot research topic in the past two years. Citation data don't lie, and this paper now has about 2000 citations, higher than most top papers.

During the discussion, it was also said that the papers we listed here have been proved to have a lasting impact over time, and their citations are not low, but if they had been voted at the top meeting and were accepted, they might have been much higher.

(as to whether it is necessary to put the paper to the top of the meeting, some people say that it depends on whether the author already has a higher teaching position and whether he has the same strength as "fuck you money". If a person already has a lifetime teaching position, or has reached the requirement of a doctoral thesis, he can just pass it to arXiv, and valuable papers will not be forgotten. In contrast, a considerable part of the papers at the summit are from doctoral students who have not yet reached their goals, and it is not surprising that the academic value is not as good as that of papers that have not been submitted.

Several papers are packaged and downloaded at https://www.yanxishe.com/resourceDetail/1030.

Original post https://twitter.com/colinraffel/status/1174691881114058752

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.