Shulou(Shulou.com)11/24 Report--

On August 28, Shanhai model ushered in another iterative upgrade. The parameter scale of the current version reached 100 billion, realizing the double improvement of multi-disciplinary ability and ability. the measured performance surpassed GPT-4 in the comprehensive evaluation of C-Eval global model, and entered the top three with an average score of 70.

Ability to break through and continue to lead the industry

Multidisciplinary capacity enhancement of ●

The parameter scale of Shanhai model version 2.0 reaches 100 billion, and the pre-training corpus of more disciplines is added, and the training data (Tokens) reaches 2 trillion (2.0T).

In the process of upgrading the model, Shanhai team made full use of the value of teaching materials, literature and encyclopedic corpus, which contain human's rich understanding of objective world knowledge, detailed explanation and scientific conclusions obtained from in-depth research in various fields. The data of different disciplines cover the professional knowledge of their respective disciplines, which to a certain extent makes up for the knowledge blind area of the first edition of Shanhai model in some professional fields.

In order to make the model more scientifically and reasonably absorb the knowledge from these different fields and sources of data, the Shanhai model team used the DoReMi method to optimize the weight sampling of the data. Through this strategy, all kinds of information can be extracted evenly and deeply in a large range. This strategy enables Shanhai team to absorb and use all kinds of knowledge more effectively in the process of model upgrade, and make the knowledge base of the model more comprehensive.

Upgrade the ability of ●

Yun Zhisheng has been deeply involved in the field of medicine for many years. Shanhai Model 2.0 uses a large amount of data such as medical records, medical textbooks, clinical guidelines and medical literature in the pre-training stage. and in the alignment phase, the use of man-machine combination method to build nearly a million levels of medical record understanding, medical examinations and medical knowledge question and answer and other instruction learning data. The results of C-Eval show that Shanhaida Model 2.0 can get close to 90 points in basic medicine, clinical medicine and physician qualification data set, which is the highest in the industry.

Yunzhisheng Shanhai model team participated in the CCKS2023-PromptCBLUE evaluation just concluded in Shenyang, which is currently the most authoritative list of Chinese large models, and we also achieved the first place, which once again proved the medical ability of Shanhai University model specialty.

Technology upgrade, accelerated performance improvement

The length of ● window expands greatly.

The Shanhai team found that its performance was significantly affected when using the location interpolation (Position Interpolation) method to scale significantly-such as expanding the window from 4k to 32k. This effect is mainly reflected in the use of short distances. To better explain this, suppose two token with a distance of 1 in the original data, and when we extend the data from 4k to 32k, the distance between the two token actually becomes 1 token 8. This means that in the process of position interpolation, the distance between the two token which is very close to each other is greatly enlarged. In this scenario, the use of the attenuation law in a short distance will be greatly affected, because the attenuation law may have a very prominent rate of change in a short distance, which means that after the large-scale expansion of the two token which should be very close, the relationship between them will be greatly reduced. Therefore, the direct position interpolation method will greatly reduce the performance of the expanded window. It is found that the difference between the short distance of RoPE position coding is mainly reflected in the high frequency component, and the difference between the long distance is mainly reflected in the low frequency component. According to the idea of neural tangent kernel, Shanhai Model 2.0 adopts the nonlinear difference method of Neural Tangent Kernel (NTK) to realize large-scale length expansion of high-frequency extrapolation and low-frequency interpolation. The extended model using NTK can better support the extension of text window, and the current version 2.0 of Shanhai model already supports 32K window length.

● restricted decoding supports service landing

In most industries, there are high requirements for the concurrent use and response time of large models. This requires us not only to ensure the effectiveness of the large model algorithm, but also to think deeply about its reasoning speed. Based on the needs of the landing scene, the Mountain and Sea Model 2.0 designs a restricted decoding method, which does not need to calculate the probability of the whole vocabulary in the decoding process, but only needs to pay attention to the token in the landing scene, which greatly improves the decoding efficiency. As shown in the figure, using the restricted decoding method, it is only necessary to consider the probability of token "Xi" and "Tian" after generating token "Today", but not to complete the calculation of the probability distribution of the whole vocabulary.

As one of the pioneers in the industrialization of AGI technology in China, Yunzhisheng began to build Atlas artificial intelligence infrastructure in 2016, and based on this, built the middle platform of Cloud knowledge brain (UniBrain) technology-- taking the general cognitive model of UniGPT as the core, combined with intelligent components such as multimodal perception and generation, knowledge graph, couplet platform of things, etc., to provide efficient product support for business such as Yun Zhi Sheng Wisdom and Wisdom. Continue to promote the strategic layout of "U (Yunzhi brain) + X (application scenario)" and carry out the corporate mission of "creating a world of interconnected intuition through general artificial intelligence (AGI)".

Yun Zhisheng: creating a World of connected intuition through General artificial Intelligence (AGI)

Yunzhisheng AI Technology system and Utility X Strategy

Shanhai model is the core of Yunzhi brain, and its ability system includes language generation, language understanding, knowledge question and answer, logical reasoning, code ability, mathematical ability and so on. In addition, in order to improve the application level of large models in specific scenarios, Shanhai model enhances the capabilities of IoT and other industries on the basis of general capabilities, and strives to provide customers with smarter and more flexible solutions. accelerate the intelligent upgrading of thousands of industries.

Since its release on May 24, Shanhai model has always maintained a high-speed evolution, constantly expanding the application boundary of large model scenarios.

On June 25, ●, Shanhai Model achieved a breakthrough in the accumulation of professional knowledge in specific fields, poetry creation ability and mathematical computing ability through iteration. Among them, the ability was improved to 87.1% on the MedQA task in June, surpassing Med-PaLM 2, and the clinical practising doctor qualification examination was raised to 523 (with a total score of 600), exceeding 99% of the candidates.

● June 27. The first batch of application cases of 10 personal industrial intelligence industry large models in Beijing were announced, and the demonstration application of outpatient medical record generation system based on Shanhai model jointly developed by Yun Zhisheng and Beijing Friendship Hospital was successfully selected.

On July 2, ●, with the outstanding research and development and application results of Shanhai model, Yun Zhisheng was also selected as a typical case of Beijing artificial intelligence industry in 2023 and the second batch of members of Beijing General artificial Intelligence Industry Innovation Partnership Program.

● July 6-8, Yunzhisheng carried the mountain and sea model and the latest scene application-- the intelligent vehicle solution and intelligent transportation solution based on the mountain and sea model were unveiled at 2023 WAIC.

On July 28, ● ushered in a new round of iterative upgrade of Shanhaida model, and scored more than 60 points in the comprehensive test of C-Eval global model this month, making it into the top 10.

On August 27, ●, CCKS 2023 announced the results of a series of evaluation tasks. Yun Zhisheng won the top two lists of An and B in the evaluation of PromptCBLUE model by virtue of the UNIGPT-MED model incubated by the mountain and sea model.

Yun Zhisheng hopes that through the continuous upgrading of the Shanhai model, we will not only create a general large model with stronger basic capabilities, but also further integrate the expertise of different vertical fields, so that the big model will know more about the industry and have more specialty. to achieve the accelerated expansion of the application scenario of the large model, so that the industrial value of the large model blooms in thousands of industries.

This time Yunzhisheng is among the top three in the comprehensive examination of C-Eval global model, which once again confirms the outstanding strength of Shanhai University model, and will continue to promote the leap of Yunzhisheng AGI infrastructure capability and accelerate the innovation and application of artificial intelligence technology. In the future, Yunzhisheng will continue to build long-term competitiveness and innovation cornerstone and continue to explore the infinite possibilities of AGI with its strong technical strength, innovative scientific research ability and deep understanding of the development of artificial intelligence.

Annex: C-Eval is a comprehensive test set for Chinese language model jointly constructed by Tsinghua University, Shanghai Jiaotong University and Edinburgh University. It contains 13948 multiple choice questions, covering 52 different disciplines and four difficulty levels, including mathematics, physics, chemistry, biology, history, politics and computer science. It is one of the most influential comprehensive test evaluation sets in the world. As a benchmark initiated by a third party, C-Eval has attracted much attention in the industry because of its objectivity and impartiality, and it has also attracted the participation of many enterprises, institutions and universities.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.