Shulou(Shulou.com)06/02 Report--

Abstract: China's strongest database academic conference is coming! During the NDBC conference, Dharma Institute Database and Storage Laboratory held the first Industrial Database R & D Forum in conjunction with the Database Professional Committee of China Computer Society (CCF). The collision of ideas of production, learning and research witnesses "Ali Power"

From October 12 to October 14, the annual highest academic conference in the field of database in China-NDBC 2018 was held in Dalian as scheduled. At this academic feast, which represents the strongest strength in the domestic database field, Alibaba not only gathered a strong lineup of participants, but also led by Dr. Li Feifei (name: Flying Dagger), vice president of Alibaba Group, scientist of Dharma Institute and head of database and storage technology. In addition, during the NDBC conference, Dharma Institute Database and Storage Laboratory also held the first Industrial Database R & D Forum in conjunction with the Database Professional Committee of China Computer Society (CCF). The forum strongly invites technical leaders of well-known enterprises and academic experts to jointly discuss the industry-university-research ecology of domestic databases and make original contributions to the long-term development of databases.

Relying on the NDBC platform, the Industrial Database R & D Forum is a two-way database technology salon between academia and industry, which is to close the distance between database academic research and emerging Internet business needs, communicate with top database scholars in actual scenarios, seek cutting-edge technology breakthroughs at system level, and build bridges for academic cooperation between enterprises and universities.

▲ The forum was packed

The forum was greatly welcomed by academics and industry people. Many industry celebrities and industry elites gathered together and the scene was packed. Dr. Li Feifei elaborated Ali's thinking on the co-construction of production, learning and research at the meeting, and promoted the healthy development of the whole industry with closer connection with an open attitude, showing Ali's mind and spirit as an industry leading company.

▲ Dr. Li Feifei, Vice President of Alibaba Group, Scientist of Dharma Institute and Head of Database and Storage Technology

Li Feifei said: "The database system is the core infrastructure software facility. After years of development, it has formed a mature and perfect commercial market and ecosystem. The global market has reached 40 billion US dollars. The recent boom in cloud computing and big data markets has brought unprecedented new opportunities to the relatively traditional field of database systems, which need to adapt to the challenges of cloud native and large-scale global distributed deployment.

In addition, the rapid development of various types of new hardware and machine learning/artificial intelligence technologies also requires database systems to further promote the application of emerging technologies such as integrated software and hardware design and intelligent operation and maintenance and kernel optimization. Finally, database security is another important issue. How to implement an efficient encrypted database system and protect data privacy remains to be solved. Solving these challenges and problems requires deep cooperation between academia and industry to discover and overcome core problems from practical applications and system requirements, and to help large-scale engineering implementation and commercial deployment.

This seminar, with the platform of NDBC, provided a rare opportunity for core representatives from the domestic industry and academia to gather together to discuss in depth how to better cooperate and exchange. We have also conducted in-depth exploration and collision on the hot issues facing the current database system, and the results are remarkable. The industry forum will continue to be held at the annual NDBC conference in the future. We believe that this forum will have a long-term and far-reaching impact on the development of China's database industry and scientific research. "

Senior technical leaders from Alibaba, Huawei, Neusoft Group and East China Normal University gave wonderful speeches in turn, sharing their business challenges, technologies and solutions, sharing cutting-edge technologies and development trends of databases, and discussing how industry-university-research cooperation can promote application landing.

Cloud-native database POLARDB: self-developed black technology releases excellent performance

▲ Zhang Youdong, Senior Technical Expert of Alibaba

Alibaba senior technical expert Zhang Youdong (name: Lin Qing) introduced, POLARDB is Alibaba cloud self-developed relational database, using computing and storage separation architecture, computing nodes light weight easy to expand, can complete backup node expansion in seconds; storage platform, support up to 100TB storage. POLARDB adopts a number of black technologies to ensure high performance, including user-mode network protocol stack, user-mode storage protocol stack, high-speed RDMA network, NVMe SSD, Optane new storage hardware, etc. POLARDB meets the needs of users on the cloud for large-capacity, high-performance and Auto Scaling database services.

Alibaba Cloud Database team customized PolarFS, a distributed file system with low latency and high reliability for POLARDB. PolarFS research results have been published in VLDB 2018 "PolarFS: An Ultra-low Latency and Failure Resilient Distributed File System for Shared Storage Cloud Database"; Based on PolarFS shared storage platform, POLARDB redesigned the storage engine, based on shared storage for physical replication, greatly reducing POLARDB IO overhead, read and write performance compared to MySQL has several times the performance improvement; Finally, POLARDB is 100% compatible with MySQL ecosystem and integrated into the existing cloud product ecosystem, so that users do not need to do business transformation, and the cost of use is extremely low. At present, POLARDB has begun to serve hundreds of enterprises on Alibaba Cloud.

In the future, POLARDB will also make more database technology innovations, covering more database application scenarios on the cloud, creating greater value for users.

POLARDB X (Powered By X-DB): Global Global Pursuit of Extreme

▲ He Dengcheng, Senior Technical Expert of Alibaba Database Business Department

As the world's largest online trading platform, Alibaba's current business volume is very large. The data scale that needs to be processed online is on the order of PB. On the day of Double Eleven, the SQL requests that need to be processed even reach 100 million times per second. He Dengcheng (name: Guido), senior technical expert of Alibaba Database Business Department, said: "In order to handle such a large amount of data and transaction processing capabilities, we developed a distributed relational database POLARDB X (Powered By X-DB), which adopts shared-nothing architecture and transparent data sharding mechanism for users. The optimized paxos protocol ensures consistency between data replicas and high performance across geographic deployments. To avoid single point failures, decentralized HLC mechanism is used to implement distributed transactions. "

It is reported that POLARDB X also analyzes the access characteristics of e-commerce business, and specially designs the storage architecture of hot and cold data hierarchy based on access frequency to better utilize the hierarchy of storage hardware itself. In addition, a range of hard-software technologies, such as heterogeneous computing hardware FPGAs, are utilized to handle computationally intensive background tasks. PolarDB X strives to achieve the ultimate in both performance and cost, and will drive the cloud in the near future to provide better database services for cloud users.

SolarDB: the crystallization of the collision of wisdom between academia and industry

▲ Professor and Vice President of School of Data Science and Engineering, East China Normal University

SolarDB is a database system developed jointly by East China Normal University, Northwestern Polytechnical University and Shanghai Communications Bank, and has been applied in industry. SolarDB is based on the open source OB version 0.4. After years of development, SolarDB has been implemented in Bank of Communications and replaced the original DB2 database in many scenarios. It adopts a shared-everything architecture consisting of storage nodes (s-nodes), transaction nodes (t-nodes) and compute nodes (p-nodes), where s-nodes and p-nodes can scale horizontally, while t-nodes are single nodes for transaction processing. This architecture design finds a good balance between scalability and complexity. Zhou Xuan, professor and vice dean of the School of Data Science and Engineering of East China Normal University, said: "It is very difficult to make a database system that can be applied in colleges and universities, but we have been insisting on doing it. There are many challenges ahead, such as quality assurance of distributed systems, compatibility with commercial databases, etc., which are worth thinking about. "

SolarDB is a database system jointly developed by academia and industry. It condenses the thinking of teachers and students of East China Normal University in the development of database technology. After continuous tempering of actual business scenarios, SolarDB has become increasingly mature and stable, and will develop better and better.

Looking at the three OLTP databases above, they differ in architecture, application scenarios, and technical thinking:

Different architectures: POLARDB(shared-disk), POLARDB X(shared-nothing), SolarDB (shared-everything)

Different application scenarios: POLARDB(Cloud Computing), POLARDB X(Large-scale Internet Applications), SolarDB (Financial Services)

Different technical thinking: POLARDB(Cloud Native, Auto Scaling, 100% Open Source Ecosystem Compatible, Cost/Performance, New Hardware), POLARDB X(High Performance, Low Cost, Global Deployment), SolarDB (Balance of Scalability and Complexity)

Although OLTP databases are the most mature and traditional part of the database technology landscape, it is gratifying to see that there is still a lot of thought and progress in OLTP in academia and industry.

▲ Li Sihao, Chief Product Planner of Huawei Gauss Database Team

Li Sihao, chief product planner from Huawei Gauss Database Team, systematically introduced Huawei's investment in database field and industrial layout over the years. Huawei began to invest in the R & D of memory database system in 2005, mainly used to support telecom billing services, which can realize millions of QPS and respond within microseconds at a time. In recent years, Huawei has continuously expanded its memory database product architecture to support distribution and data persistence capabilities, and has developed towards general products with higher scalability.

Huawei also started building distributed analytical database products in 2012, adopting MPP architecture and introducing new compilation technologies such as codegen to achieve industry-leading analytical capabilities, while building key technologies such as online expansion and upgrade to achieve high system availability. It realizes the comprehensive migration of the data warehouse system of the main domestic banks from the traditional manufacturer Teradata database to its products, and also has a certain market scale at home and abroad.

▲ Yu Mingguang, System Architect of Neusoft Group

Yu Mingguang, a system architect from Neusoft Group, shared real-time insights and pattern recognition of machine data. The system proposes specifications for different customer environments, realizes automatic data acquisition, and uses clustering method to make data available. Yu Mingguang pointed out that the correlation between low frequency events and high frequency events are very different in data mining. In the scenario of ad hoc query, Yu Mingguang proposed semi-structured data query, structured query language such as SQL is not the best choice, and proposed domain-specific language DSL. He stressed that these technologies could be extended to a wider range of areas in the future and would be promising.

Technological Change, Evolution and Challenge of OLAP Database

▲ Alibaba Group Database Division researcher Zhan Chaoqun

The explosive growth of enterprise data provides a data base for expanding analytics strategies in enterprise applications, but the challenges are unprecedented. How can we provide real-time online flexible processing and open data analysis for massive amounts of data? Alibaba Group Database Division Researcher Zhan Chaoqun (name: Li Zhe) introduced Alibaba's experience in building a large-scale real-time online analysis system and its future technical trend outlook, focusing on Alibaba Group's next-generation analysis database product AnalyticDB. AnalyticDB is one of Alibaba's earliest self-developed commercial-grade database products and the earliest commercialized self-developed database product, which supports Alibaba Group's e-commerce, advertising, ants, rookie, Entertainment and many other online data analysis services; as well as providing core real-time analysis capabilities for traditional large and medium-sized enterprises, government agencies, and numerous Internet innovation companies on Alibaba Cloud.

AnalyticDB supports more than 2,000 physical nodes, more than 10 petabytes of data storage management, and high concurrency and low latency analysis, helping businesses enter the next generation of analysis and rapid exploration. It adopts hierarchical decoupling architecture, introduces intelligent SQL optimizer, ultra-large-scale distributed MPP+DAG fusion analysis and calculation engine, supports technologies such as row and column mixed storage, intelligent construction of multi-dimensional index and GPU hardware acceleration, and integrates unstructured data analysis support such as text, pictures and faces, greatly expanding technical boundaries and business boundaries.

With the development of online analysis business in these years, AnalyticDB has continuously innovated and developed in timeliness, online, complexity, intelligence, availability and cost, and has made a lot of technical breakthroughs and innovations. AnalyticDB has grown into an industry-leading commercial-grade database product capable of transforming the way users use data by making data analytics online and real-time to help users realize the value of their data.

▲ Round table discussion: current situation and development trend of the combination of production, learning and research

(From left: Zhang Youdong, Zhan Chaoqun, He Dengcheng, Yu Mingguang, Zhou Xuan, Li Sihao)

At the final roundtable, the teachers and students participating in the forum had a warm exchange with the sharing guests to discuss the construction experience and technical difficulties of Alibaba POLARDB X (Powered By X-DB), POLARDB, Analytical Database and Huawei Gauss Database. The guests shared in detail the technical issues such as cloud computing multi-tenant solution, database high availability solution, data row hierarchical storage, and advantages of multi-mode database. At the same time, the forum discussed how colleges and universities can apply the developed system to the core system of enterprises to ensure sustainable development under the normal flow of students. Teacher Zhou Xuan shared in detail the successful construction experience of SolarDB database developed by China Normal University and Bank of Communications.

▲ NDBC 2018--Group photo of Alibaba's lineup

This industrial database R & D forum is not only an important stage to show the scientific research strength and share achievements of enterprises themselves, but also reflects the highest level of R & D and application of China's database industry. At the same time, facing the future, it builds a bridge for cooperation between enterprises and academia, actively promotes the construction of industry-university-research cooperation community, and jointly builds database industry cooperation ecology.

Alibaba has always paid great attention to cooperation with academia, and the needs of users have driven the iteration of technology updates, further improving the relationship between industry and academia. Academia and industry have their own strengths in the field of database. Alibaba will continue to deepen scientific research in the future, organically combine academic research and application practice, actively expand cooperation with academia, promote the integration of production, learning and research, and drive the development of database cause in China with technology.

At the same time, at this NDBC conference, Zhang Rui, Zhan Chaoqun and Li Chao, three senior database technicians from Alibaba, were successfully elected as members of the database professional committee of the new China Computer Society, and will contribute more to the development of database technology on behalf of Alibaba in the future. So far, Alibaba has seven special committees.

With the holding of NDBC Industry Database R & D Forum and more industrial technicians joining CCF China Database Committee, it will further promote the integration of industry, university and research and the development of China's database technology industry, leading database personnel to forge ahead.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.