Shulou(Shulou.com)06/02 Report--

In the second half of 2018, the UCloud Seoul data center can no longer be used for external reasons, and all the computer rooms need to be relocated in a very short time. In order not to affect the user's existing network business, we abandoned the offline migration scheme and chose the very challenging overall thermal migration of the computer room. After five months of multi-department cooperation, we have finally achieved the goal of completely relocating all business to the new computer room also located in Seoul without users' perception.

This article will detail one of the most difficult tasks in this large project: the scheme design and practice of the migration of common components and core management modules.

plan

The whole project is divided into four stages (preparation stage, construction of new computer room, relocation of new and old computer room, and cancellation of old computer room). As one colleague said, the heat transfer of the computer room is equivalent to moving the users of one high-speed train to another. The high-speed rail is our computer room, and the users on the high-speed rail are our business. One way to make the relocation feasible requires two high-speed trains to be relatively stationary. One way is to turn two high-speed trains into one, so that the speed of the two is the same. The heat transfer of UCloud computer room adopts a similar scheme, which logically turns the two computer rooms into one computer room. To this end, the business logic of the upper layer should be seamlessly migrated in the new and old rooms, and the management system below should be unified into one set.

Among them, our SRE and application operation and maintenance team are mainly responsible for the following tasks: 1) expansion and expansion of core zookeeper services in computer room; 2) deployment and expansion of udatabase services in the middle layer of core database; 3) deployment and migration of most management services; 4) deployment and migration of core database. The above involves all aspects of pre-planning, scheme design, project implementation, stability assurance, change verification and so on.

Challenges

When we first received the demand for the overall heat transfer of the computer room, we really had a headache. The Seoul computer room is one of the earlier deployed computer rooms, and the related technical architecture is relatively old. Core database, core configuration service (zookeeper), core database middle tier (udatabase) and other services are all important basic components, and the old architecture may have complex large-scale anomalies due to changes in the basic level, thus affecting the daily use of stock users.

Fortunately, in the past year, the SRE team has comprehensively combed the resource data of various services, and we have developed a cluster resource management system (Mafia-RMS). The system collates the stock and incremental service resources by means of dynamic service discovery, static registration and so on. What are the services, clusters and servers in each computer room? The port, status, configuration and other information of each instance are recorded in our resource management system, as shown in the following figure:

Figure 1: UCloud SRE Resource Management system-Cluster Management function

Through the SRE resource management system, we can clearly know the cluster information of the internal services in the Seoul computer room stock and the status of each instance. We also build a Prometheus-based SRE monitoring system based on SRE resource system. Through the Monitor button on the right side of the above figure, you can jump to the monitoring page to obtain the real-time running status of the entire business.

With these resource data, all that is left is to consider how to expand and migrate these services.

Expansion and reduction of ZooKeeper service

The first is the expansion and expansion of the internal service registry zookeeper.

UCloud uses zookeeper as the internal service registration and service discovery center on a large scale. Most services exchange visits by using zookeeper to obtain service registration addresses. Wiwo framework (C++) and uframework (Golang), which are widely used within UCloud, register their Endpoint information into zookeeper based on active state machines. Services with the same Endpoint prefix belong to the same cluster, so some services are expanded. The new node can use the same Endpoint prefix. The clients of wiwo and uframework frameworks have the ability to parse the zookeeper configuration, and the real IP and port information can be obtained by parsing the Endpoint. Then, through the way of client load balancing, the request is sent to the real business service instance, so as to complete the mutual invocation between services. As shown in the following figure:

Figure 2:UCloud Seoul computer room deployment invocation and service registration / discovery path map

The zookeeper cluster of the old computer room in Seoul is an ordinary cluster with 3 nodes (at that time, the scale is relatively small, 3 nodes are sufficient). However, the scale of the new computer room in Seoul is much larger, so the cluster scale of the new computer room zookeeper also needs to be expanded. After our evaluation, the zookeeper cluster of the new computer room can be expanded to 5 nodes, which can basically meet the needs.

In fact, an ideal migration architecture should be, as shown in figure 3, the whole new data center uses the same technical architecture as the old one (unified architecture and version). The new architecture is completely deployed independently, and there is no data interaction with the old server room, while users' business services (such as UHost/UDB/EIP/VPC, etc.) smoothly realize the migration of the control and management side, as well as the physical location in some way.

Figure 3: schematic diagram of service migration of old computer room in ideal condition

However, the ideal state can not be achieved in reality, and the limitations of internal architecture and code logic make it impossible for business instances to achieve smooth migration at the logical and control level, let alone at the physical level. The newly deployed management service needs to communicate with the management service of the old server room, so we have adjusted the deployment architecture of the new server room service and adapted the actual situation to use two deployment modes, as shown in figures 4 and 5:

Figure 4: cross-room service deployment with the same cluster expansion mode

Figure 5: cross-room service deployment of a new cluster grayscale migration model

Whether it is the same cluster expansion mode in figure 4 or the new cluster grayscale migration mode in figure 5, the zookeeper cluster of the new and old data centers must be integrated at the zookeeper level, and the data of the two clusters need to be consistent and synchronized in real time. Therefore, in the technical level of zookeeper, the two clusters must be turned into one cluster, and the original 3-node zookeeper cluster must be expanded to 8 nodes (1 leader,7 follower) by expanding the capacity of remote computer rooms. only in this mode can the data be consistent and real-time.

For the services that need to be registered for the new deployment of the new computer room, the configuration of the zookeeper address in their configuration file is not a list of 8 new ip, but a list of only 5 IP of the new computer room. In this way, the back-end services of the new and old data center use the same set of zookeeper, but the configuration is different. The purpose of this is that when the old data center goes offline, all the services of the new data center do not need to restart and update the configuration because of the reduction of the zookeeper cluster. As long as the three nodes where the old data center is located in the cluster are offline, the configuration updates of the remaining 5 nodes can be re-selected.

Therefore, in the expansion plan of the computer room of zookeeper, we adopt the mode of expanding the capacity of the cluster first and then splitting it. The expansion of zookeeper is the first step in the expansion of the entire data center, and all subsequent services will rely on the zookeeper configuration of the 5 nodes newly built by the operation. The reduction of zookeeper cluster is the final operation. After all the services are expanded and all business instances are migrated, the zookeeper cluster will be scaled down and re-selected as the master, so that the entire server room can be eliminated.

Migration of database middle tier udatabase

Next is the migration of the database middle tier udatabase.

The two modes of figure 4 and figure 5 deal with zookeeper in the same way, but the difference lies in the expansion and migration of internal management and control plane services at the back end. The udatabase migration uses the figure 4 mode, which is equivalent to the remote server room expansion in the original cluster. The new instances of the expansion use the same Endpoint prefix as the original cluster, that is, they belong to the same cluster. When the service is started, the status of the new expanded instance will be the same as that of the original cluster, and the framework (wiwo or uframework) layer will find the change of the cluster node from the zookeeper (new) through the client mode. At the same time, some kind of load balancing algorithm is used to route the request traffic to the new node. In this way, instances that belong to the same cluster but are located in two address locations have partial traffic. The way to scale down is to directly offline the service of the old data center and the cluster. In this way, the client will forward all the traffic of the cluster to the node where the new server room is expanded, thus completing the smooth service expansion. Udatabase completes the migration of the cluster in this way.

New cluster grayscale transfer mode

In fact, figure 4 mode is feasible for most services, but why is there a new cluster grayscale migration mode shown in figure 5? Because figure 4 can be uncontrollable in some scenarios. If the newly-built instance (such as Service An Instance 2 in figure 4) has problems with software stability and reliability, such as abnormal configuration, abnormal software version, or abnormal network, it may lead to problems with requests routed to new nodes, which will directly affect online business. The scale of the impact is determined by the proportion of expanded nodes to the total nodes of the cluster, such as our 1:1 expansion mode. If there is a problem with the service, 50% of the requests may be directly abnormal. Udatabase uses the scheme in figure 4 because of its high code stability, simple function and configuration, and mainly depends on its high-performance forwarding capability.

For some businesses with complex functional logic (such as ULB/CNAT), the more secure figure 5 pattern is used. Because the business level supports cross-cluster migration, you can build a new cluster without business traffic. The Endpoint path prefix in zookeeper is different from the original cluster, using a new path, and then at the business level, through the migration platform or tools. Migrate the backend service or instance on demand, the whole process is controllable, and roll back immediately if there is a problem, which is the most secure migration solution. Our general grayscale migration platform SRE-Migrate is shown in figure 6.

Figure 6:UCloud internal general business migration system SRE-Migrate

Computer room deployment platform SRE-Asteroid

There are a large number of UCloud product lines and services under the product name, and both the scenarios in figure 4 and figure 5 require a lot of service deployment. The server room deployment optimization project promoted by the SRE team in 2018 is intended to solve the problems of long delivery time and huge labor costs in the construction of new computer rooms in UCloud (domestic and overseas data centers, proprietary clouds, private clouds, etc.). By the end of 2018, the project was successfully put into production, covering the deployment management of nearly one hundred services of core business such as mainframe and network, and solving a series of problems such as configuration management, deployment specification and software version. It is precisely because of this achievement that the relocation of computer room in Seoul has been able to complete the deployment or expansion of nearly 100 new clusters in a very short time. Figure 7 shows our new computer room deployment platform SRE-Asteroid.

Figure 7:UCloud internal computer room deployment platform SRE-Asteroid

Deployment and migration of core database

Finally, it is how to deploy and migrate at the core database level. The database service used by the internal UCloud service is MySQL, and the internal MySQL cluster is built by physical / virtual machines in the management network and deployed as one master library, one high availability slave library, two read-only slave libraries and one backup library. The MHA+VIP method is used to solve the high availability problem of the master database, and the BGP/ECMP+VIP method is used to solve the load balance and high availability problems of the slave database. The general architecture is shown in figure 8:

Figure 8:UCloud internal MySQL service architecture diagram

The architecture of the internal MySQL database cluster used by the new and old data center in Seoul is similar to the figure above. In order to switch the cluster between the new and old data center, we have designed the following solution, as shown in figure 9:

Figure 9: schematic diagram of Seoul cluster internal database cluster migration

On the whole, in order to ensure that the core database cluster can complete the migration work stably, we abandon the switching scheme of double main database and double write, so as to prevent the problems of data inconsistency and abnormal synchronization between new and old clusters caused by network or other factors. We adopt the simplest solution, stop the console service during the business trough, and directly modify the database middle-tier configuration switching scheme.

In the deployment phase, we deployed a MySQL cluster with the same high availability architecture in the new server room in Seoul. The logical backup of the database of the old server room was imported, and the cluster of the old server room was made into a cascading mode (green dotted line in figure 9). The master library of the new server room serves as the slave library of the old server room and synchronizes the data through MySQL asynchronous synchronization (binlog). We use the pt-table-checksum tool to check the data consistency of the two clusters regularly to ensure that the data of the new and old computer rooms are completely consistent. At the same time, use the internally developed topology analysis tool to confirm all the business conditions that call the master-slave database of the old cluster database (mainly which udatabase clusters).

After the deployment is completed, data consistency and real-time performance are ensured through cascading. Udatabase still accesses the VIP of the MySQL main library of the old data center (blue dotted line in figure 9). At this time, no business is written to the main library of the new data center through direct connection (to ensure data consistency, the main library of the new data room is temporarily set to read-only mode).

After determining the migration time and migration plan, after announcing the users at a certain business trough, the whole console operation of the Seoul computer room stops for a period of time (during which the API request of the Seoul computer room may fail). Under the premise of determining that the flow is very low, the business will be switched from the old computer room MySQL cluster to the new computer room MySQL cluster by modifying the configuration of the database master-slave library VIP in the database middle tier (udatabase cluster). At this point, all requests for the business will flow into the new cluster (red dotted line in figure 9). In order to prevent the old cluster from still having business to write or read, we set the main database of the old cluster as read-only, and then continue to analyze the possible requests on the old cluster through tcpdump and process them manually, finally ensuring that all businesses use the new MySQL cluster.

Due to the need for cluster switching of several services, such as host, network, storage and monitoring, in order to ensure that they do not affect each other, cluster-by-cluster processing is used, and the time of overall handover and detection takes nearly 1 hour.

In the whole process of data center switching, only the database cluster is a stateful business, so the importance and danger are relatively high. After the switching of the service is completed, the most important link is also completed, and the migration of the rest of the business level (UHost/UDB/EIP, etc.) can be completed by each business team.

Wrap up

After all business instances are migrated, you can theoretically complete the migration of the server room. However, you still need to monitor the traffic of the instances that are still running in the old data center. After confirming that there is no traffic, go offline and stop the service. Finally, the request monitoring and connection monitoring are carried out on the zookeeper cluster of the old computer room (3 zookeeper nodes of the old computer room) to confirm that there is no request from the local computer room and the new computer room (excluding the autonomous synchronization of the zookeeper cluster). After the confirmation, the final zookeeper cluster change is carried out, and the whole cluster (8 nodes) is divided into the old computer room (3 nodes) and the new computer room (5 nodes). The cluster of the old computer room stops service directly. The new zookeeper cluster in the new data center completes the new primary operation, confirming that the synchronization is normal and the service is normal.

Write at the end

After all the operations mentioned above, the migration of the entire computer room in Seoul is completed. The whole project takes 5 months, most of which is spent on the migration process of business instances, mainly for different users to determine different migration strategies and migration time; it takes less time to migrate and deploy internal management services. UCloud has worked out a detailed plan for each step of this migration, including service dependency analysis, operation steps, verification method, switching risk, rollback plan, and so on. In order to complete such a huge hot migration of the new data center, the team has invested sufficient manpower and time. We believe that the new computer room in Seoul has better construction planning, hardware configuration and software architecture, and can provide better services to users.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.