Network Security Internet Technology Development Database Servers Mobile Phone Android Software Apple Software Computer Software News IT Information

In addition to Weibo, there is also WeChat

Please pay attention

WeChat public account

Shulou

QingStor object Storage Architecture Design and Best practices

2025-01-19 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Servers >

Share

Shulou(Shulou.com)06/01 Report--

QingStor object storage architecture design and best practices, I believe that many inexperienced people do not know what to do, so this paper summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

The concept and characteristics of object Storage

Before introducing the architecture and design principles within QingStor ®️object storage, let's first take a look at the concept of object storage, that is, what are the characteristics of object storage from an external perspective and how we should use it.

Object storage is essentially a storage product, and other storage, such as file storage, block storage, the function is similar, the main function is data reading and writing. The biggest difference is that object storage manages data as an object, which is its main feature, in which all data is treated as an object.

Object storage has some very distinct features:

Its structure is flat, unlike file storage, there is no directory level, and there is no need to search and open the directory when reading and writing data.

Object storage has the ability to store massive data, which refers not only to hundreds of GB, but also to hundreds of terabytes or even PB.

Object storage is suitable for the storage of unstructured data, which means that no assumptions are made about the type and format of the data. Simple text, pictures, video and audio can be stored in object storage and treated as objects.

Object storage provides services through Restful interface, that is, HTTP protocol, which makes the access of object storage very convenient, and data can be uploaded and downloaded anytime and anywhere.

Core benefits of QingStor ®️object Storage

! [0_1591683403636_1.png] (https://community.qingcloud.com/assets/uploads/files/1591683404995-1-resized.png

The above points are the general features of object storage products. Next, let's introduce the unique core advantages of QingStor ®️object storage, a storage product developed by Qingyun QingCloud, which mainly includes three aspects:

First, for the scene of a large number of small files, we have done targeted optimization and optimization on storage and IO.

Second, the system of QingStor ®️object storage has the ability to expand infinitely. When the amount of data and access increases, the computing and storage capacity can be improved by adding nodes.

Third, QingStor ®️object storage is a platform for data storage and flow, which is reflected in two aspects:

First of all, the API of all functions is open, and the business logic can be completed by arbitrarily calling API.

Second, it provides some very distinctive features, such as lifecycle management, cross-region replication, and custom callbacks, which can be adapted to more business scenarios.

QingStor ®️object Storage Global data Model

The figure above is the global data model of the QingStor ®️object store and can be understood as a logical view.

This includes several main concepts: Global means global, Global consists of multiple Zone, and Zone means region, which can be understood as a data center. For example, a set of object storage is deployed in the Beijing area, and one set is also deployed in the Shanghai data center. These two sets of object storage belong to the same Global.

QingStor ®️objects are stored with corresponding management services at both the Zone level and the Global level.

Zone consists of many Bucket (buckets). When using object storage, you must apply for a bucket before you can upload object data to the bucket to store and manage object data in buckets.

There can be many buckets under the same Zone. There are a variety of object data in the bucket. There is no limit on the type and size of object data, and there is no limit on the number of objects in a single bucket. You can upload object data indefinitely.

Analysis of QingStor ®️object Storage Architecture

This is the background system architecture for QingStor ®️object storage, and this architecture diagram has been simplified and abstracted.

First of all, the access subsystem, object storage provides online services, access through Restful, essentially equivalent to the background of online services, there needs to be an access subsystem to receive requests, resolve protocols and other work.

Under the access subsystem is the index subsystem. The index subsystem is used to store and manage the metadata of the object. Metadata refers to the Meta information of the object, including Object type, size, writing time and other information, which is managed by the index subsystem.

The storage subsystem is responsible for storing and managing the data entity itself to ensure the reliable persistent storage of the data.

The main work of event subsystem is asynchronous event processing and distributed task scheduling, which is the underlying mechanism of life cycle management and other functions.

Several solid-line arrows in the figure show the flow of data, or request processing flow. When the request comes down from the access subsystem, the access subsystem will interact with the index subsystem and storage subsystem to obtain metadata and data entities. This is the core read and write process.

Dotted lines indicate that events may be generated and sent to the event subsystem when some functions are turned on. There are two dotted arrows: one is that some events will be triggered in the read-write process and will be sent to the subsystem; the other is that the user can actively submit some events and enter the event subsystem through the access subsystem.

These subsystems constitute the main modules of QingStor ®️object storage background.

Judging from this architecture diagram, its internal implementation is not particularly complex.

QingStor ®️object storage is a storage product, its core function is to read and write data, the logic must not be too complicated.

However, it is not so easy to do this system well, because object storage mainly deals with massive data scenarios, in which there are many architectural challenges.

First of all, data should be reliably stored and persisted to prevent any data loss.

Second, in the case of a large number of visitors, it is necessary to ensure the continuous service ability of the system.

Third, the system needs to have good scalability to cope with the increasing amount of data and requests.

In addition, QingStor ®️object storage is a distributed system, which is coordinated by multiple nodes to provide services. In this case, the failure of a single node is common. We need to ensure the availability of services and reliable storage of data in the case of a single node failure. These are the key issues to be considered when designing the architecture.

Next, we will interpret how these subsystems are implemented in turn.

Implementation of QingStor ®️object Storage Subsystem

In the access subsystem, Gateway service is the most important. Gateway service is essentially a Server running in the background, which runs on the gateway node.

The main function of Gateway service is to receive requests from upstream, do protocol parsing, data processing and data reading and writing. The most important data reading and writing functions in object storage are basically completed here.

The Gateway service itself is stateless, that is, the request is handled by which Gateway is the same, so the Gateway can be easily extended, that is, the increase and decrease of service instances.

Take a look at how the access layer ensures high availability from the overall link.

Before users start to access COS, they will visit the DNS server and get a virtual IP address through the domain name of the COS service. The virtual IP will point to a gateway node. If this node fails, the virtual IP will automatically migrate to another surviving node.

That is, QingStor ®️object storage can guarantee that the acquired IP will always point to a living node, and the business will always be available.

After the request arrives at the node, the object store will further do load balancing and distribute the request to the Gateway services of multiple nodes.

When the request is transferred to a Gateway and it is found that there is something wrong with the Gateway, the system will automatically forward the request again, which is the request level Failover.

Through these two mechanisms, the QingStor ®️object store ensures that all access requests can be answered.

In addition, by implementing Gateway as a stateless service, it is very convenient to scale horizontally by increasing the number of Gateway service instances to withstand high concurrent visits and ensure service availability.

Next, let's talk about the architecture of the index subsystem. The main function of the index subsystem is to store and manage the metadata of the object data, including the type, size, writing time and storage location of the object.

This information is very important, if the metadata is lost, the data itself cannot be read, so it is necessary to ensure that the metadata is stored absolutely safely and reliably.

On the other hand, the scene of massive data needs to support the index of massive data, and the index subsystem must be able to deal with the increasing amount of data. On this basis, the processing performance of the system should be improved as much as possible.

How does QingStor ®️object storage do that?

First of all, through the way of data slicing to deal with massive data. The data is stored as a slice, which means to slice according to the alphabetical order of the object name. For example, the first node stores the Amurf object and the second node stores the Umurz object. Each node is responsible for an interval of data, and then uses a coordination service to record the corresponding relationship between the node and its responsible area.

In this way, it is very convenient to scale horizontally. If there is too much object data in Amurf, the system will split it into two and add a node. Amurf will be split into Amurf and Amurd will remain in the original node, and Dmurf will be placed on the new node. In this way, more data can be processed and the ability to read and write can be improved.

The addition of new nodes is carried out through the coordination service, and the process of data splitting and rebalancing is completely automatic.

The Gateway service of QingStor ®️object storage maintains a real-time connection with the orchestration service to get the latest data distribution. Through this slicing storage mechanism, it ensures that the storage capacity can be improved by adding nodes in the case of massive data.

In terms of data security and reliability, QingStor ®️object storage uses a copy mechanism, and each metadata is stored in a three-copy way. If one copy of the machine fails, the data can be read from other copies to ensure that it will always be available when accessed.

If the node where the data resides changes, Gateway immediately knows through the coordination service which node to access to get the data. In addition, the three copies are persisted to ensure that the data is safe and reliable.

QingStor ®️object storage uses KV storage engine when storing metadata on a single node. The index structure of KV storage engine is the biggest advantage of LSM,LSM index structure is that writing is very fast, which can improve the overall write performance of the system.

In addition, LSM also has a feature, its underlying data stored on disk is orderly, that is, a sst file, can provide efficient sequential query.

When QingStor ®️objects are stored in a query interface that lists objects in buckets, this storage feature can be well applied to efficiently sequence data in alphabetical order. Because it is ordered storage itself, the read speed is very fast, and the interface processing is very efficient, which is the advantage of using KV storage.

Generally speaking, the index subsystem of QingStor ®️object storage makes it scalable by means of slicing and coordinating services, and ensures the security of data through copies.

In addition, the KV storage engine is used on a single node to improve writing efficiency, while supporting efficient query of the list interface.

The storage subsystem stores the object data itself, and object storage deals with massive scenarios. The amount of data is very large and will continue to increase, and the number of visits will be very large, and there is a possibility of increase. Therefore, the design of the architecture should ensure that the system performance can be continuously improved. In addition, it is necessary to ensure the security of the data and the stability of the cluster.

QingStor ®️object storage divides storage into storage groups under a unified named storage space, each consisting of its own distributed file system.

The advantages of adopting a storage group design include three main aspects:

One is to distinguish between hot and cold data, that is, storage levels. QingStor ®️object storage provides two storage levels: low-frequency storage and standard storage.

Compared with low-frequency storage, standard storage has more access and less data.

The amount of data stored at low frequency is large, but the number of visits will be lower.

Different storage groups can be used for two different storage requirements, and there is no relationship between groups, so different storage groups can use heterogeneous hardware devices. For low-frequency storage, high-capacity disks and a lower CPU configuration can be used to further optimize costs.

Second, the use of storage groups can make cluster expansion more flexible and convenient, for example, the system needs to be expanded, how to do it?

By adding a storage group, the new data can be written directly to the new storage group without the need for complex operations such as movement and migration of historical data.

QingStor ®️object storage also supports another expansion method, which migrates part of the existing data to the new storage group, making the whole cluster more balanced.

Third, the use of storage group has the effect of fault isolation. If there is a node failure in storage group 1 or the whole storage group is broken, it has no effect on other storage groups, and other storage groups can process data normally and have the effect of fault isolation.

In a single storage group, the core technology of QingStor ®️file storage is applied, and three copies are used to store the data. Each time the data is written, and the success of the write will not be returned until all the three copies are written, so as to ensure the strong consistency and security of the data.

The local file system directly deals with the underlying block devices, leaving out the Linux local file system, which improves the performance of IO processing.

The data transmission of QingStor ®️object storage adopts RDMA efficient transfer protocol. RDMA is a mechanism for transferring data between different nodes. It does not need to go through CPU, it directly copies data from one node to another through hardware control, and the execution of CPU is parallel, so it is an efficient way to transfer data.

Event subsystem is not in the core process of data reading and writing in the architecture of QingStor ®️object storage, but it provides many very important functions. Lifecycle management, cross-region replication and custom callback are all designed based on the event subsystem.

The basic logic of the event subsystem is relatively simple, generating and handling events, where events can be understood as a message system.

There are two links to generate messages. When a link is reading or writing data, when you finish reading, writing, deleting or updating a piece of data, the index subsystem will generate events and send them to the event subsystem. Another link is that the user submits an event directly through the access subsystem, which can enter the event subsystem.

The event subsystem has many preset consumer processes, and consumers handle events, which are saved through distributed message queues, and consumers read events and process them according to the preset logic. For example, there is a feature for consumers to deal with the lifecycle, and a feature for consumers to handle custom callbacks.

Optimization practice of QingStor ®️object Storage for massive small Files

Next, let's talk about the optimization of massive small file scenes by QingStor ®️object storage.

Why do you need to optimize a large number of small files?

A large number of small files are difficult to deal with in many storage products, and the difficulties are mainly reflected in several aspects:

First, when there are a lot of small files, it will lead to a lot of random read and write, and the performance of random read and write is much worse than that of sequential read and write.

Second, the utilization of small files in the underlying resources is relatively low. How do you understand it? When small files are finally stored on disk, most of the time, the underlying file system is used to store, each small file will correspond to a file, the file system will use a separate structure, that is, inode, to record the meta-information of each small file, including execution permissions, user groups and other information, but these information is often meaningless to users.

This leads to a situation in a massive scenario: there is not so much data in the file itself, but there is a lot of unnecessary metadata, which takes up a lot of storage space, which will result in low utilization of the underlying resources.

To solve this problem, QingStor ®️object storage has been optimized in two aspects:

In terms of improving storage utilization, QingStor ®️object Storage merges many small files into one large file.

As shown in the figure above, there is a merge file that contains a number of small files, including x, a, b, c, all of which are small Object files that are written to the same file. In this way, additional metadata storage is reduced and resource utilization is improved.

When using this method, if you need to delete the previous small files, the QingStor ®️object store will only make a mark, and there will be processes in the background to compress the merged files in real time and periodically to release the deleted resources.

In terms of improving write performance, QingStor ®️object only appends it to the end of the merged file when writing to a small file, and does not write to the specified Offset after opening the file, that is, no random write, ensuring that all writes are sequential writes, greatly improving write performance.

In addition, if there are concurrent write requests, such as a, b, and c are all small files, one write request to Gateway,Gateway will package the three requests into one, which needs to be written three times. Here, it is merged directly into one.

Reducing three IO to one IO can do such a merge because the underlying storage is merged, and the merging of concurrent IO further improves the write performance of QingStor ®️object storage.

When reading, the QingStor ®️object store finds the data of the file for reading by merging the Path of the file with the Offset in the small file.

For example, now you have to read x and a, b, c at the same time. After reading x, opening the file handle can be reused, and you don't need to open the file every time. This way improves the performance of reading small files stored in QingStor ®️object.

Introduction to the key functions of QingStor ®️object Storage

QingStor ®️object storage supports two deployment modes: one is standard deployment, and the cluster is composed of gateway nodes and storage nodes.

The access, index and event subsystem will be deployed on the gateway node. The storage node only deploys storage services, separates computing and storage, and supports the deployment of at least six nodes. This method is more suitable for scenarios with a large amount of data and large requests. Both the gateway node and the storage node can be expanded independently.

The second is converged deployment. The cluster is composed of all-purpose nodes, and all services are deployed on all-around nodes. This method is suitable for scenarios where the scale is small, the amount of data is relatively small, and the number of visits is not large. If the amount of data increases, it is very convenient to expand the storage and indexing services in a single node.

The first is the life cycle management function, the life cycle can preset some logic to the stored data, and automatically process the data after a period of time.

There are some typical application scenarios in the life cycle, such as the processing of expired logs. Most of the time, the logs are stored to meet policy requirements and can be deleted in a few months.

Lifecycle management can set automatic rules and automatically delete expired log data when it expires. In addition, the lifecycle management function can also separate hot and cold data. For example, businesses only need to analyze data from the past two months, but longer data is not needed.

However, in order to meet the needs of compliance, more remote data also need to be saved and cannot be lost. At this time, the lifecycle management function can make a distinction between hot and cold data and automatically set the data to be transferred to lower-cost low-frequency storage after a period of time.

The life cycle function of QingStor ®️object storage is carried out in buckets, and its core is the event subsystem for message processing and distribution.

After the user writes an object, there will be an event, and the event subsystem will analyze how to deal with the event, such as deleting it after a week or a month, or moving to low frequency, so as to process the data accordingly.

If the system already has some historical data before turning on the lifecycle function management, the event subsystem will actively pull the historical data, analyze and process it.

Another important function of QingStor ®️object storage is cross-region replication, which is suitable for data backup and disaster recovery scenarios.

Application scenarios for cross-region replication mainly include:

Backup and disaster recovery: in order to ensure the security of the data, you need to make a backup in another computer room in addition to the local one. The data in the local computer room is lost and can be recovered with remote data.

Nearest access: if the users of the business are distributed in different areas, each user wants to obtain data more quickly when accessing their picture and video resources.

QingStor ®️object storage can store data by region, one in Beijing and one in Shanghai. The data can be written to Beijing and replicated to Shanghai across regions to meet the needs of nearby access.

Improve computing efficiency: if there are multiple machine learning, data mining applications or business programs that want to access the same data, these business programs may be deployed in different places. In order to ensure computational efficiency, QingStor ®️object storage can replicate a copy of data nearby by replicating across regions.

The core function of cross-region replication is still the event subsystem. After the user writes an object, an event will be generated, and the event subsystem will handle the event and copy the data to the bucket of the remote specified destination.

If the system already has some historical data before configuring this feature, after configuration, the event subsystem will actively pull the data and copy it to a remote location.

Next, we focus on the data processing capabilities of QingStor ®️object storage.

If you want to process the data stored in the object storage, including video transcoding, thumbnails, pictures with watermarks, etc., you can package the actions to be done into a task for submission, enter the event subsystem through the Gateway service, and the event subsystem will read out the data according to the specified actions, process them and write them to the target location.

The premise of data processing is that the data is stored in the bucket of object storage, which is the process of data processing, and its core is still the event subsystem of QingStor ®️object storage.

Let's see whether QingStor ®️object storage is convenient and easy to use from the user's point of view.

All the functions of QingStor ®️object storage API are open, can be accessed and accessed, while providing 8 mainstream SDK, covering mainstream programming languages, these SDK codes are open source, are on GitHub, and you can use them directly.

In addition, the QingStor ®️object store provides two advanced command-line tools:

The first is qsctl, which provides powerful Unix commands for managing local data resources and remote data resources in QingStor ®️object storage. It is very convenient to operate on the data, including RM, CP and other operations, providing one-click synchronization to synchronize your local data with the data in the object storage.

The second command line tool is qscamel, which is used for efficient data migration between different object storage systems. For example, if there is data in Aliyun's OSS, you can use this tool to migrate it to QingStor. This tool currently supports mainstream object storage, including QingStor, AliyunOSS, AWSS3, and GoogleCloud Storage, and can migrate data between these systems.

S3 is the standard access interface of Amazon object Storage Service. QingStor ®️object Storage is very compatible with S3. Many applications, SDK and third-party services that have been developed based on S3 can easily access QingStor ®️object Storage without modifying the code, including Hadoop, Spark, ElasticSearch and Docker.

QingStor ®️object Storage Transportation Industry Best practices

Finally, let's start with the intelligent transportation platform scenario to see how to use QingStor ®️object storage in the business. The main applications of the intelligent transportation platform are video surveillance on the highway, as well as photography, toll collection, license plate recognition and other services at toll stations.

This is a typical high-concurrency access scenario, with a lot of high-speed cameras and toll stations. Every camera and toll station will generate a large number of pictures and video resources, which will be sent to QingStor ®️object storage.

Data is very important, every video on the highway may be used to see what happened afterwards, and the data must be stored reliably.

On the other hand, the storage system is required to interface seamlessly with third-party business applications, such as taking a picture of a vehicle at a toll station, identifying its license plate number and querying where it comes from, which requires QingStor ®️object storage to interface with other business systems.

The access service of QingStor ®️object storage can support high concurrency scenarios in an extended way, using multiple replicas to provide reliable and strong consistent storage.

After the data is written, the QingStor ®️object store generates an event, which goes to the event subsystem to process the data according to the preset logic, and the user can customize the third-party interface. For example, do a license plate recognition, or do deduction logic, can be associated with the business system through a custom interface.

In addition, the data stored by QingStor ®️objects is precipitated, which can support big data's analysis and decision-making in the later stage.

QingStor ®️object Storage is an enterprise-level general data storage platform for massive unstructured data, with unlimited expansion and cross-region service capabilities, which can perfectly support the storage needs of various Internet applications, big data analysis, audio and video, backup systems and other enterprise application scenarios, and help you build a future-oriented and business-enabled data storage platform.

After reading the above, have you mastered the methods of QingStor object storage architecture design and best practices? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.

Views: 285

*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.

Share To

Servers

Wechat

© 2024 shulou.com SLNews company. All rights reserved.

12
Report