Shulou(Shulou.com)12/24 Report--

On November 30, 2023, the China Institute of Electronic Technology Standardization and the Storage Industry Technology Innovation Strategic Alliance, together with more than 20 research institutions, enterprises and user representatives, including Huazhong University of Science and Technology, Sun Yat-sen University, Zhijiang Laboratory, Tide Information, Tianyi Yun, etc., the industry's first distributed Fusion Storage Research report (hereinafter referred to as "the report") was officially released. For the first time, the report systematically combs and deeply analyzes the concept, technical architecture and application scenarios of distributed converged storage, which provides reference and guidance for the development of converged storage industry.

The leader of the China Institute of Electronic Technology Standardization said, "data has become an important factor of production, and data storage and data management play a more significant role in information technology change. As a new storage technology, distributed converged storage will play a greater role in the field of data storage."

Xie Changsheng, chairman of the Information Storage and Security Special Committee of the China computer Industry Association and a professor at Huazhong University of Science and Technology, said, "Storage, as one of the bases of digital technology, has achieved unprecedented development as well as great challenges. Distributed converged storage is the key technology to meet this challenge. The report brings together the collective wisdom and experience of academia, industry and users, and describes a panoramic view."

Research report on distributed converged Storage

Distributed converged storage will become the data cornerstone of the intelligent computing era.

Zheng Weimin, academician of the Chinese Academy of Engineering and professor of Tsinghua University, pointed out that "digitization and intelligence have led to explosive growth in the amount of data, and the emergence of distributed fusion storage technology can not only provide the efficiency and security of data storage." it can also achieve dynamic data management and intelligent optimization, providing more comprehensive and reliable data support for various application scenarios. "

The report points out that compared with the traditional distributed storage technology, distributed converged storage realizes that the same storage cluster provides multiple storage services at the same time on the basis of the traditional distributed storage architecture. and the storage technology of protocol fusion and data fusion interworking between unstructured data protocols is realized, which can provide the ability of cross-schema / format data sharing and interworking. And support block, file, object, big data multi-protocol access and multi-protocol association unified management, to meet the multi-application across different protocols to achieve efficient data access.

The report identifies four key technologies of distributed converged storage: unified storage pool, ensuring balanced distribution of data among storage nodes, supporting horizontal expansion of storage pool, and automatic data repair; storage service convergence, providing file, object, big data and block storage services in one storage system, and rich multi-protocol support capabilities. Data management integration, using a unified unstructured data and metadata management architecture to ensure native access to various protocols; storage protocol integration, supporting the integration and interworking of unstructured data protocols such as NFS, CIFS, HDFS, S3, etc., ensuring that various protocols share the same data and metadata. Through the high versatility and fine-grained management of the underlying hardware resources, distributed converged storage can greatly alleviate the shortage of storage resources caused by the rapid growth of data in the intelligent computing era.

Distributed converged storage architecture

Provide strong survival support for five typical scenarios

Distributed fusion storage provides strong survival support in massive multi-source and heterogeneous data processing and the edge scene of "communication, perception, computing" integration, so that it can meet the storage needs of different scenes, and provides efficient, multi-dimensional, consistent and customized production factors for the current digital transformation era, so that data-driven infiltration into the panoramic scene.

The report summarizes five typical application scenarios for distributed converged storage:

● is a deep learning scene represented by autopilot: the amount of data in the autopilot scene is relatively large, and the data type is complex. In the whole data life cycle, different protocols need to be used according to different stages of application. On the other hand, distributed converged storage realizes multi-protocol interaccess and interworking, and has the ultimate aggregation performance, which can meet the requirements of data acquisition, data tagging and model training for storage bandwidth, small file processing, storage IO and so on, making the business process more concise and efficient.

● large model application scenario represented by AI / AIGC: model training is the core of AI development, which requires the storage system to provide high bandwidth and high concurrency performance. Distributed converged storage can avoid data copies caused by different access protocols, and provide a storage infrastructure with massive multi-source data, high performance and low latency, which provides strong storage capacity support for the training and reasoning of AI applications.

● is a data intelligent application scenario represented by the integration of lakes and warehouses: the architecture of integration of lakes and warehouses, a variety of data sources, and a variety of services and storage protocols, so converged storage is an inevitable trend. Distributed converged storage provides higher computing scalability for this scenario, provides storage requirements for various computing engines at the same time, and realizes collaborative cooperation and data sharing.

● risk control scenario: the particularity of risk control scenario means that it puts forward high requirements for storage system availability, data consistency, scalability, data security and performance. Through the design of high availability, high security, high scalability and high performance, distributed converged storage can support real-time data access and flexible expansion of the system to meet the needs of large-scale data storage and analysis.

● HPC high-performance computing scenario: in the field of high-performance computing, distributed converged storage technology plays a more and more important role. By integrating a variety of storage business resources into a whole, it meets the requirements of file, object, block function, performance and capacity, and provides high reliability and scalability, which provides a reliable data basis for scientific computing and data analysis.

Distributed converged storage is becoming the preferred storage base for users.

The report predicts that the emerging technology of distributed converged storage is currently in the early stage of development and is still facing the high-performance challenges of multi-service convergence, the standardization challenges of new storage forms and intelligent operation and maintenance management challenges.

At the same time, distributed converged storage has a broad prospect. first, distributed converged storage contributes to the construction of "survivability", which is in line with the policy direction; secondly, distributed converged storage effectively organizes and manages different storage media, storage protocols and storage services in one storage architecture, which makes it possible that "one storage architecture supports one data center". Based on the AI large model technology, all kinds of new applications emerging in the future need more powerful data infrastructure and data platform capabilities, which brings inestimable market space, which is destined to be a promising future for distributed converged storage.

Li Rengang, secretary-general of the storage industry's technological innovation strategic alliance, proposed that "the rapid development of technologies such as cloud computing, big data and artificial intelligence will promote the further application and promotion of distributed converged storage. It has become the preferred storage base for more and more industry users."

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.