Shulou(Shulou.com)06/02 Report--

This article will explain in detail how to parse SOFARegistry. The content of the article is of high quality, so the editor will share it for you as a reference. I hope you will have some understanding of the relevant knowledge after reading this article.

SOFAStack (Scalable Open Financial Architecture Stack) is a financial-level distributed architecture independently developed by Ant Financial Services Group, which includes all the components needed to build a financial-level cloud native architecture. It is a best practice honed in financial scenarios.

SOFARegistry is Ant Financial Services Group's open source and highly available service registry with massive service registration and subscription capabilities. Driven by the business development of Alipay / Ant Financial Services Group, it has evolved to the fifth generation in the past decade.

Overview

As we mentioned in the previous section, the biggest difference between SOFARegistry and other products in the field of service discovery is that it supports huge amounts of data. This chapter will cover some of the features of SOFARegistry in supporting huge amounts of data.

This article will explain from the following aspects:

DataServer overall architecture: give a brief description of the overall architecture supporting massive data in SOFARegistry, and explain the key technical points involved in the data fragmentation and synchronization scheme.

DataServer startup: explain the services started by DataServer, so as to pave the way for a more intuitive understanding of data fragmentation, trigger timing and trigger mode of data synchronization.

Data slicing: explain the reason and implementation method of data slicing using consistent Hash algorithm in SOFARegistry.

Data synchronization scheme: explain the data synchronization scheme adopted by SOFARegistry

Overall architecture of DataServer

In most service registry systems, each server stores a full amount of service registration data, and the servers replicate the data through consistency protocols (paxos, Raft, etc.), or adopt algorithms that only guarantee the final consistency to achieve asynchronous data replication. This kind of design has no problem for systems of general business scale, but when applied to huge business systems with massive services, it will encounter performance bottlenecks.

To solve this problem, SOFARegistry adopts the method of data slicing. The full service registration data is no longer stored in a single machine, but is distributed in each node. Each server stores a certain amount of service registration data and makes multiple copies at the same time, which theoretically achieves unlimited service expansion and high availability, and finally achieves the goal of supporting massive data.

Among the various data slicing algorithms, SOFARegistry uses the industry mainstream consistent Hash algorithm for data slicing. When nodes are dynamically expanded and scaled down, the data can still be evenly distributed and the data balance can be maintained.

In data synchronization, the pre-slicing mechanism similar to that in Dynamo, Casandra, Tair, Codis, Redis cluster and other projects is not adopted, but the operation log is recorded at the granularity of dataInfoId in DataServer memory. To some extent, this implementation also realizes "pre-fragmentation", thus ensuring the effectiveness of data synchronization.

Figure 1 overall architecture diagram of SOFARegistry

DataServer Startup entry

Each bean of the DataServer module is configured uniformly in JavaConfig, the JavaConfig class is DataServerBeanConfiguration, and the startup entry class is DataServerInitializer, which is not managed and configured by JavaConfig, but inherits the SmartLifecycle interface, and its start method is called by the Spring framework at startup.

The DataServerBootstrap#start method (figure 2) is called in this method to start a series of initialization services.

You can see from the code that when the DataServer service starts, it starts three bolt services, DataServer, DataSyncServer, and HttpServer. When these Server are started, DataServer registers a series of Handler to process various messages.

Figure 2 start method in DataServerBootstrap

The functions of these Server are as follows:

DataServer: data service, push to get data, notification of online and offline service, etc.

DataSyncServer: data synchronization service

HttpServer: provides a series of REST interfaces for dashboard management, data query, etc.

The specific role of each Handler is shown in figure 3:

Fig. 3 the effect of each Handler

At the same time, RaftClient is started to ensure the distributed consistency between DataServer nodes, and various startup tasks are started, as shown in figure 4:

Figure 4 DataServer startup tasks

The startup listening ports for each service are shown in figure 5:

Figure 5 listening port

Other initialization Bean

In addition to the above startup services, some bean are initialized when the module is started. All the bean when the system is initialized are registered through JavaConfig in the DataServerBeanConfiguration, which is mainly reflected by the following configuration classes (configuration classes will be changed. For more information, please see source code):

DataServerBootstrapConfigConfiguration: this configuration class is mainly used to provide some basic Bean when DataServer services are started, such as DataServerConfig basic configuration Bean, DataNodeStatus node status Bean, DatumCache cache Bean, etc.

LogTaskConfigConfiguration: this configuration class is mainly used to provide some Bean related to log processing

SessionRemotingConfiguration: the main purpose of this configuration class is to provide some Bean that communicates with SessionServer, as well as some request processing Bean during the connection. For example, Bean used to start services such as BoltExchange and JerseyExchange, and Bean for node online and offline, data release, etc., are key configuration classes.

DataServerNotifyBeanConfiguration: the Bean configured in this configuration class is mainly used for event notification, such as DataChangeHandler used to handle data changes, etc.

DataServerSyncBeanConfiguration: the Bean configured in this configuration class is mainly used for data synchronization operations

DataServerEventBeanConfiguration: the Bean configured in this configuration class is mainly used to handle events related to data nodes, such as event center EventCenter, data change event center DataChangeEventCenter, etc.

DataServerRemotingBeanConfiguration: the Bean configured in this configuration class is mainly used for connection management of DataServer

ResourceConfiguration: the Bean configured in this configuration class is mainly used to provide some Rest interface resources

AfterWorkingProcessConfiguration: this configuration class configures some post-processing Handler Bean to handle some post-processing actions after the end of some business logic.

ExecutorConfiguration: this configuration class mainly configures some thread pool Bean to perform different tasks

Data slicing explanation

Data slicing mechanism is the core of SOFARegistry to support massive data. DataServer is responsible for storing specific service data. Data is stored in consistent Hash slices according to dataInfoId, supporting multi-copy backup and ensuring high availability of data.

(students who are interested in the consistent Hash algorithm and want to know more about it can read the original paper of Karger and his collaborators: Consistent hashing and random trees: distributed caching protocols for relieving hot spots on the World Wide Web. )

Before we explain the data sharding of SOFARegistry, let's take a look at the simplest traditional data slicing Hash algorithm.

Traditional data slicing Hash algorithm

In the traditional data slicing algorithm, we first label the ID of each node from 1 to K, and then use the Hash algorithm for each data to be stored on the node. The calculated value is taken as a module for K, and the result is the node ID to fall on.

The algorithm is simple and commonly used, and it is used for data slicing in many scenes.

Fig. 6 traditional Hash slicing algorithm

Under this algorithm, when a node goes offline (such as Node 2 in figure 6), all nodes after that node need to be re-labeled. All data should be re-calculated and modeled by Hash value, and then re-stored in the corresponding node. (figure 7)

In the massive data scenario, this method will bring a lot of performance overhead.

Fig. 7 traditional Hash slicing algorithm, when a node goes offline, it will affect the global data distribution.

Traditional consistent Hash for data slicing

In order to make the service nodes online and offline will not affect the global data distribution, in the actual production environment, many systems use consistent Hash algorithm for data slicing. The representative projects that the industry uses consistent Hash include Memcached, Twemproxy and so on.

Data range

The consistent Hash algorithm uses $2 ^ {32} $$buckets to store all Hash values, 0 ~ $2 ^ {32}-1 $$as the range of values, and forms a ring.

Data slicing principle

In figure 8, NodeA#1, NodeB#1 and NodeC#1 are the positions where the ID of three nodes A, B and C falls on the ring after the calculation of the consistent Hash algorithm.

The triangle is the position where different data fall on the ring after the consistent Hash algorithm. Each data goes clockwise to find the nearest node as the node for data storage.

Fig. 8 consistent Hash algorithm

It is not difficult to imagine from figure 8 that when a node goes up and down, it only affects the data distribution between the upper and lower line node and the node closest counterclockwise to that node. At this point, you only need to rearrange the data that falls into this range. (figure 9)

Figure 9 NodeB#1 offline in consistent Hash algorithm

Characteristics of consistent Hash algorithm

In this algorithm, the ID of each node needs to be calculated by the consistent Hash algorithm and mapped to the circle, which brings two characteristics of the consistent Hash algorithm.

When the total number of nodes is small, multiple virtual nodes can be virtual (such as figure 10, which may be crossed in practice and easily described here). When there are enough virtual nodes, it can ensure that data can be evenly distributed on the real nodes, which is the advantage of consistent Hash algorithm.

After adopting consistent Hash, the distribution range of data in node ring is not fixed. When the node is dynamically expanded and scaled down, part of the data will be redistributed, which will bring some problems in data synchronization.

Figure 10 arrangement of virtual nodes

Consistent Hash Code implementation of SOFARegistry

In SOFARegistry, the consistent Hash class diagram is implemented by the ConsistentHash class, as shown in figure 11:

Figure 11 consistent Hash class diagram for SOFARegistry

In this class, SIGN is the delimiter of ID, numberOfReplicas is the number of virtual nodes of each node, realNodes is the list of nodes, hashFunction is the Hash algorithm, and circle is the Hash ring in the pre-slicing mechanism.

ConsistentHash uses the MD5 digest algorithm for hash by default, and the constructor supports customization of the hash function, so users can customize their own Hash algorithm. At the same time, the implementation of circle in this class is TreeMap, which skillfully uses the tailMap () method of TreeMap to achieve the node lookup capability of consistent Hash. The nearest node hash value calculation code for the data is shown in figure 12:

Fig. 12 calculation method of hash value of nearest node in data

Pre-slicing mechanism

The traditional consistent Hash algorithm has the characteristic that the data distribution range is not fixed, which makes the service registration data need to be re-stored and arranged after the server node is down, offline and expanded, which brings difficulties for data synchronization. Most of the data synchronization operations are carried out by using the contents of the operation log. In the traditional consistent Hash algorithm, the data operation log is divided by node fragments, and the node change leads to the change of the data distribution range.

In the computer field, most of the problems can be solved by adding an intermediate layer, so the problem of data synchronization caused by irregular data distribution can also be solved by the same way.

The problem here is that after the node goes offline, if the data is synchronized with the ID consistency Hash value of the current surviving node, the data operation log of the failed node cannot be obtained. Since the data is stored in a place that will change, can the feasibility of data synchronization be ensured if the data is stored in a place that will not change? The answer is yes, this middle layer is the pre-sharding layer, and the problem of data synchronization can be solved by matching the data with the immutable layer.

At present, the main representative projects in the industry, such as Dynamo, Casandra, Tair, Codis, Redis cluster and so on, have adopted the pre-slicing mechanism to achieve this unchanged layer.

The data storage range is divided into N slot slots in advance, the data corresponds directly to the slot, the operation log of the data corresponds to the corresponding solt, and the number of slot will not change due to the up and down lines of the nodes, thus ensuring the feasibility of data synchronization. In addition, the concept of "routing table" needs to be introduced, such as figure 13, which is responsible for storing the mapping between each node and N slot, and ensuring that all slot is evenly distributed to each node as far as possible. In this way, when the node goes online and offline, only the contents of the routing table need to be modified. Keeping the slot unchanged ensures flexible expansion and scaling, and greatly reduces the difficulty of data synchronization.

Fig. 13 pre-slicing mechanism

Fragment selection of SOFARegistry

In order to realize distributed storage of service registration data, SOFARegistry adopts data slicing based on consistent Hash. Due to historical reasons, in order to achieve data synchronization between nodes, we use dataInfoId as the granularity of data synchronization between DataServer.

Node fragmentation

When the DataServer node is initialized successfully, the task is started to connect to the MetaServer automatically. This task registers a DataServerChangeEvent event with the event center EventCenter, which will be triggered after registration, after which the Hash value of the new node will be calculated and the tube will be sliced at the same time.

After the DataServerChangeEvent event is triggered, the DataServerChangeEventHandler handles it accordingly, which is divided into the following steps:

Initialize the consistent Hash value of the current data node and add the current node to the consistent Hash ring. (figure 14)

Figure 14 initializing a consistent Hash ring

Gets the changed DataServer nodes that are obtained from the MetaServer when the DataServer service is started and passed in through the DataServerChangeItem in the DataServerChangeEvent event. (figure 15)

Figure 15 gets the changed DataServer node

After obtaining the current DataServer node, if the node list is not empty, then traverse each node, establish a connection between the current node and the rest of the data nodes, and delete the locally maintained node data that is not in the node list. At the same time, if the current node is a DataCenter node, the LocalDataServerChangeEvent event is triggered.

At this point, the node initialization and shredding into the Hash ring have been completed.

The data related to the data node is stored in Map, and the related data structure is shown in figure 16.

Figure 16 DataServer node consistent Hash storage structure

Data fragmentation

When the service is online, the dataInfoId Hash value of the new service is calculated, so that the service is fragmented. Finally, the nearest node is found and stored on the corresponding node.

As mentioned earlier, the DataServer service adds a publishDataProcessor at startup to handle the corresponding service publisher data publishing request, which is PublishDataHandler. When a new service publisher comes online, DataServer's PublishDataHandler will be triggered.

The Handler first determines the status of the current node and returns a failed request if it is not working. If it is working, the onChange method of the data change event center DataChangeEventCenter is triggered.

An array of DataChangeEventQueue queues is maintained in DataChangeEventQueue, and each element in the array is an event queue. When the onChange method mentioned above is triggered, the Hash value of the dataInfoId of the change service is calculated, thus the queue number of the service registration data is further determined, and the changed data is encapsulated into a data change object and passed into the queue.

The DataChangeEventQueue#start method is called by a new thread when the DataChangeEventCenter is initialized, and it continues to fetch new events from the queue and distribute them. The new data will be added to the node to achieve sharding.

Explanation of data synchronization scheme

SOFARegistry is a three-tier architecture of Client, SessionServer and DataServer. At the same time, Session and Data clusters are managed by MetaServer. In the process of service registration, the data has not only inter-layer data synchronization, but also inter-layer node synchronization.

Intra-layer synchronization-data playback

The Client side has stored the service data that needs to be subscribed and published in local memory. After connecting to Session, the service data will be played back to Session, and finally published to Data. At the same time, Session stores all Pub data published by the client and maintains consistency with Data through data comparisons on a regular basis. When the data changes, the Data that holds the data initiates the change notification, and the synchronous SessionServer is required for version comparison. When it is determined that the data needs to be updated, the latest data operation log will be pulled.

The operation log is stored in a stack manner, and the log is obtained by synchronizing the operation logs after all versions through the location of the current version number in the stack.

Inter-layer synchronization-multiple copies

In order to ensure the availability of data in the Data layer, SOFARegistry makes a multi-copy mechanism in the Data layer. When a Data node is downsized or downtime occurs, the backup node can immediately become the master node through the backup data, provide services to the outside world, and back up the corresponding backup data to the new node according to the new list.

When the capacity of a Data node is expanded, the new node enters the initialization state, during which new data writing is prohibited, and the read request is forwarded to the subsequent available Data node to obtain data. After the backup data of other nodes are synchronized according to the information of the new node, the status of the newly expanded Data node becomes Working and begins to provide services.

Summary

In the massive service registration scenario, in order to ensure that DataServer can infinitely expand the business scenario of massive data, different from other service registries, SOFARegistry uses consistent Hash algorithm for data slicing, which ensures the scalability of data. At the same time, the consistency of the data is ensured by recording the operation log with the granularity of dataInfoId in DataServer memory and synchronizing the data with the granularity of dataInfoId between DataServer.

On how to parse SOFARegistry to share here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.