Shulou(Shulou.com)06/02 Report--

In the past two days, there has been a widely watched news in the AI circle. Baidu has released a semantic understanding framework for continuous learning, ERNIE 2.0. on the basis of surpassing BERT in Chinese tasks, this model has made a new breakthrough in English tasks, surpassing BERT and XLNet in a total of 16 Chinese and English tasks, and achieving SOTA results.

Perhaps for most AI technology followers and AI developers, the most direct value of ERNIE 2.0 is that the industry will get another powerful NLP model, and it is another exciting achievement of AI technology in China.

In addition to the achievements, the industrial value behind ERNIE 2.0 can not be ignored. Especially in the current context of international trade and science and technology, the unsupervised pre-training language model represented by ERNIE 2.0 is in a very critical industrial position.

Behind the peak showdown between ERNIE 2.0 and BERT and XLNet, there is a change and confrontation of the global NLP industry pattern.

The speed-up shift era of NLP

Let's go back a little bit and see why BERT and ERNIE, our friends in Sesame Street, have become the stars of the entire AI industry.

NLP, namely natural language processing, is a very important technology path in the field of AI. It is related to how agents understand human language and words, and human-computer intelligent interaction on the basis of understanding. The importance of this area is obvious, and what excites the industry is that NLP technology has set off a "speed race" around the world from the end of last year to the present.

Last October, Google released what is considered to be a landmark NLP model, BERT. This model adopts the mode of pre-training on the polar big data set, which refreshes the highest score of AI in a large number of NLP data sets. Even in tasks such as machine reading comprehension, it shows a performance comparable to that of human beings.

BERT, which is called "brutal to leave no way to live" by AI industry insiders, can be said to promote NLP to a new track. The other good news is that BERT has not entered a unified quack model. In March this year, Baidu released ERNIE, a NLP model based on knowledge enhancement technology, which surpassed BERT; in a number of Chinese NLP tasks. In June this year, XLNet, the NLP model proposed by Google and CMU, ranked first among more than 20 tests, fully surpassing BERT; in performance, while the new BERT optimized by Facebook not long ago has regained a lot of its own records.

At this point in the story, the unsupervised pre-training language NLP model has basically entered the era of "three Kingdoms killing". Behind it, there is a technological leap between Google and Baidu, the East and West AI giants.

From the perspective of industrial value, the unsupervised pre-training language model based on large-scale data is catching up with each other and constantly setting new records, which brings new opportunities for the application of NLP technology to various industries. As the basic solution of NLP problem, these models bring machine reading comprehension, emotion recognition, text classification and other tasks to a new level, which will benefit a wide range of fields, such as voice assistant, online customer service, intelligent finance, dialogue robot and so on.

Perhaps we can understand the evolution of NLP over the past 10 months as a speed shift. The field of language intelligence has just experienced a upheaval from diesel to gasoline, which is a page-making of the times.

In this vast imaginative age of NLP, ERNIE 2.0 is coming.

The battle of the Tower of Babel changes with the sword of ERNIE 2.0 coming from the east.

As mentioned above, today's NLP race is a game in which top experts refresh each other's AI records, with the taste of Ye Gucheng vs. Ximen blowing snow at the top of the purple ban. And this race takes place at the top of the Tower of Babel of human language intelligence.

The just-released ERNIE 2.0 actually represents such a thing: in this peak battle, Baidu won the initiative of the race.

In just a few months since the release of ERNIE 1.0 in March, Baidu has completed the re-upgrade of ERNIE and released ERNIE 2.0, a semantic understanding framework for continuous learning, and a framework-based ERNIE 2.0 pre-training model.

Overall, ERNIE 2.0 not only completed a record-breaking performance, but also solved the problem of "miracles" that BERT has long been criticized by the industry. There are fewer requirements for computing power and the amount of data, which makes the unsupervised preprocessing language model closer to the industrial reality. Overall, ERNIE 2.0 shines on the NLP stage with three hardcore capabilities:

1. Better results: ERNIE made a new breakthrough in English tasks, surpassing BERT and XLNet in a total of 16 Chinese and English tasks, and achieved SOTA results. In English tasks, ERNIE 2.0 beat BERT and XLNet; in seven tasks of natural language understanding dataset GLUE, while ERNIE 2.0 surpassed BERT and refreshed SOTA in nine different types of datasets, including reading comprehension, emotional analysis, question and answer, and so on. It can be said that among the key issues in the field of NLP, it has promoted the frontier exploration and breakthrough in the industry.

2. Smaller data: the unsupervised pre-training NLP model has always been thought by the industry that the more data, the better the effect, which smacks of a miracle. However, ERNIE 2.0 takes full advantage of Baidu PaddlePaddle (flying propeller) multi-machine distributed training, and uses 7.9 billion tokens data to complete the model training, which is about 1/4 of XLNet data.

3. Less computing power: a major problem with the unsupervised pre-training language model is that the computational power is consumed too much and the training time is too long, which leads to the difficulty of industrialization. ERNIE 2.0 also does well at this point, using only 64 V100s, or about 1/8 XLNet of hardware computing power, and provides a solution for developers to customize their own NLP models.

Behind the excellent data performance, perhaps we should also pay attention to the broader industry problems: ERNIE 2.0 represents the international attention of the NLP race, China Energy has never been absent.

AI is not absent: Chinese Energy on the Road of NLP

In the past, we always thought that China was backward in scientific and technological innovation. China's science and technology industry was good at imitating and shaping applications, but it was often absent from basic research and breakthroughs. However, ERNIE 2.0 has proved that China AI can participate in and contribute to the key technologies and topics on the road of AI at the first time. Chinese science and technology is no longer absent in the AI era.

In the development of ERNIE 2.0, BERT and XLNet, we can see that three layers of Chinese energy is shining in the industry. Baidu AI is bringing surprise to the world's AI industry.

1. Chinese Wisdom: the most criticized thing about BERT is that it smashes into the model with huge data sets, resulting in a simple, rough and effective effect. However, there is a lack of innovation at the algorithm level. In ERNIE 2.0, Baidu realizes the energy of sustainable learning semantic understanding. By supporting the incremental introduction of custom pre-training tasks at three levels: vocabulary (lexical), grammar (syntactic) and semantics (semantic), we can fully capture the lexical, grammatical, semantic and other potential information in the training corpus. These tasks train and update the model through multi-task learning, and whenever a new task is introduced, you can learn the task without forgetting the information learned before.

(ERNIE 2.0: a framework for semantic understanding of sustainable learning)

This means that ERNIE 2.0 can continuously improve the effectiveness of the model through continuous construction training, including lexical, syntactic, semantic and other pre-training tasks. In other words, through the cross-domain combination of linguistics and AI science, ERNIE 2.0 has completed a large number of innovations in algorithm logic and verified the task performance of sustainable learning semantic understanding. This opens a new path for the AI industry and a new development track for the NLP field. This contribution of Chinese wisdom happens to play a role in the cutting-edge exploration focused by the AI world, contributing a unique force to the promotion of global AI technology.

2. Location in China: the industrial application of NLP is developing rapidly. Unsupervised pre-training language model is considered to be the backbone network in all kinds of NLP applications and the bottom layer of the next generation NLP technology. If this position is completely stuck by BERT, or by European and American companies, then it is easy to get stuck in the underlying technology again. On the other hand, models such as BERT are not good at Chinese tasks, and English applications in the field of NLP may be significantly ahead of Chinese applications in the long run, affecting the industrial process. This is what we don't want to see.

ERNIE 2.0 proves that China's position has been established in this low-level technology race. With the all-out efforts of Baidu AI, Chinese science and technology has become a leader and explorer.

3. China's speed: last year, BERT and Tu Network occupied the top two when selecting the most valued new AI technology in the world. It is not difficult to see the importance that the global technology industry attaches to BERT. However, over the past few months, China has developed a NLP model that is comparable to or even ahead of BERT. This is also an embodiment of China's speed in the AI era.

ERNIE 2.0 can be used as a horizontal axis, which proves that basic algorithms, the core area of AI, China has gradually caught up with the speed of the United States, and can surpass the algorithms of the top AI companies in Europe and the United States. Reverse output core algorithm, using Chinese speed to lead the AI. Baidu's AI hard core strength, we can see that it is showing accelerating characteristics. At this year's Google conference, it has been commented as a weakness in algorithm innovation. However, Baidu caught the problem and went up against the trend. The speed of AI in China is shown in the algorithms that Baidu throws to the world.

It must be admitted that in today's AI field, Chinese energy can not be ignored. This road is not paved overnight, but condensed through the efforts of countless Chinese AI people. For example, in the just-concluded ACL, Baidu has a total of 10 papers included in the conference, the research field covers a number of hot topics and cutting-edge directions in the field of NLP, showing the accumulation of international top-level technology.

Behind the continuous academic and industrial breakthroughs, Baidu has accelerated the development of NLP since 2010. When AI technology has not been popular in the world, it has devoted itself to the cutting-edge technology research and industrial application of NLP. Baidu NLP, which has trained a large number of AI talents for the industry and has become a top international research institution, is known as the "Iron Army" in the AI world. At present, Baidu brain language and knowledge technology has the largest multi-semantic knowledge graph and the best Chinese semantic representation model, and can provide 30 + technical tools, including text review, machine translation, language generation and other capabilities. provide diverse, flexible and customizable services and scenario solutions for the industry.

Thus it can be seen that behind today's achievements is the technical exploration and personnel training that has been sharpening the sword for ten years. A sword from the east of Baidu, behind the condensation of ten years of wall map broken wall NLP hard work. In fact, every time China's AI shines on the front line of the world, there is more or less such a story behind it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.