Shulou(Shulou.com)06/03 Report--

With the continuous expansion of the scale of enterprise IT, enterprises begin to consider not only the disaster recovery of a single server node, but also the disaster recovery of the entire data center, once there is a disaster in my entire data center, how should I recover? so in the past two years, many enterprises have realized this and began to build multiple data centers in the same city or in different locations to achieve comprehensive high availability at the data center level. Disaster recovery, then some problems that will be encountered in the process of making this kind of disaster recovery, how should we make a disaster recovery plan? if I use Microsoft WSFC as a disaster recovery method, how should I consider what we need to pay attention to? that's what we need to discuss today.

First of all, when it comes to disaster recovery, many enterprises have thought about doing it, but in some cases, they have made a disaster recovery plan, but in the end, when the disaster occurred, it did not succeed for three reasons.

Disaster recovery mechanism did not take effect

There is no clear disaster recovery plan and operators do not test, resulting in delayed disaster recovery time

There is no automatic mechanism, manual operation is required when a disaster occurs, and personnel are unable to operate because of the disaster.

To sum up, there are no more than two points. The disaster recovery plan has not been thoroughly tested. The disaster recovery mechanism is not automated.

Automation technology is a sharp tool for IT operation and maintenance, which can help us to save a lot of efficiency. Sometimes we should use automation, but we should not all use automation, but in the field of disaster recovery, when we really adopt a disaster recovery method, the more automatic the better. That is to say, in my disaster recovery plan, the less manual operation, the better. We should implement a set of established disaster recovery mechanism, everything is automatically executed when a disaster occurs, and the application is restored. If the disaster recovery relies too much on human operation, then this is not necessarily a good solution, because once a real disaster occurs, people may not be able to recover the application at the scene as soon as possible.

Therefore, when we want to build a remote data center, we must seriously sit down, make a disaster recovery plan, and define a disaster recovery plan. We must first consider the following.

After the disaster, we should do those things, achieve those goals, and how to achieve these goals as soon as possible.

What is the priority order of these things? I should do that first and then that one.

These things should be done by those people, whether everything is done by the most suitable person, and whether everyone is prepared for each step.

The specific steps for disaster recovery should be to form a standardized manual.

After we have thought about the general content, we can begin to develop a specific disaster recovery plan, and a complete disaster recovery plan should at least include the following

Scope of disaster recovery: define the scope of disaster recovery, the larger the scope, the more you have to write here, the data center, the servers below, the applications above, and so on.

Disaster recovery system dependency: here, it is mainly at the business system level, and the dependency of each system is viewed according to the application system level, so it is convenient for us to formulate the recovery strategy process.

Disaster recovery strategy: according to the definition of disaster recovery scope and dependency, write recovery strategy, each system, every component, how to recover in the event of a disaster, what method should be used, what order should be followed, how to operate, how to verify after the operation is completed, and how to avoid risk.

Disaster recovery method: the specific disaster recovery strategy should be implemented in a manual, highly available cluster, replica, backup, etc., as far as possible in a business system-supported, familiar, automated way.

Emergency contact information: the contact information of disaster recovery stakeholders, leaders, disaster recovery operators and application verifiers to ensure that they can be contacted as soon as possible

Regular disaster drills: a large part of the failure of disaster recovery is due to the lack of regular drills, resulting in the failure of the established disaster recovery plan and cannot be implemented when a disaster occurs, so regular disaster drills are very important. it is recommended to perform a disaster drill at least once a year.

Disaster recurrence: if it is best to do this, after each disaster recovery exercise, or after the actual disaster recovery, ask the operator, or disaster recovery team, to record the disaster recovery process and look back in hindsight, whether it is carried out according to the plan, and those areas that can be optimized, as valuable knowledge.

Above, Lao Wang combined his own study and summarized the basic theory of disaster recovery for everyone. The reason for writing this is because Lao Wang saw that many enterprises obviously want to do many data centers and want to do double-live data centers. Disaster preparedness data center, but a pat on the head, there is no prior planning, it is meaningless, hope to see friends can gain something It can bring some help in the development of disaster preparedness data center.

Well, today we are mainly talking about the ways of disaster recovery. Microsoft has many ways of disaster recovery, such as hyper-v replication, backup, WSFC clustering, ASR, and so on.

Among them, we focus on WSFC cluster today. WSFC is used for multi-site data center clustering. One advantage is that the WSFC system itself can achieve a fully automated failover mechanism. As long as the cluster is correctly configured, WSFC will automatically switch over when a failure occurs, without human intervention, unless you WSFC the application running above the cluster, you need to make additional configuration after failover.

What WSFC really started to support for multi-site data centers is WSFC 2008. In the 2008 era, WSFC began to support multi-subnet cluster architecture, that is to say, you can have 10 network segments in Beijing and 20 network segments in Shanghai. It also allows you to create a cluster. When a Beijing node crashes, applications can drift to Shanghai with 20 network segments to continue to work, but not in 2003. In the 2003 era, all cluster nodes must be the same subnet.

In order to realize the multi-subnet technology, the most important thing is that WSFC began to support the customization of cluster group network name dependency relationships in 2008. For a cluster group, we can make the network name correspond to the IP addresses of many subnets. These different subnet IP addresses can be OR relationships. As long as one of them can register online, the application can provide services normally. After the failover, the name is registered online at another subnet address, and the application switches to another subnet address to provide services.

In the era of WSFC 2008, although WSFC itself has realized the support for multi-subnet, some applications above the cluster can not support multi-subnet very well, for example, SQL 2005, SQL 2005, SQL 2008, Hypermuri V 2008, real-time migration, although we have deployed clusters with multiple subnets, but these applications do not support multi-subnet, it is still meaningless. After SQL2008R2,Hyper-V 2012, everything has been improved.

When we consider WSFC multisite, we can mainly look at it from the following aspects

The network

Arbitration

Storage

The network

For the WSFC multi-site network, we should first consider what kind of network architecture is adopted in the whole multi-site environment.

Multiple subnets

Whether the nodes at different sites should use different subnets, if different subnets are used, whether the upper layer applications support it, whether it will bring additional manual operation, whether multiple subnets are external networks with multiple subnets, or whether heartbeat networks also need multiple subnets. If the heartbeat network has multiple subnets, whether static routes need to be added.

two。 Extend VLAN and get through the network.

For the nodes at different sites, the network has been connected, so it is not necessary for each node to use different subnets, and all nodes are in the same subnet. For the cluster, the application is the most convenient and the support is the best. However, network personnel may need to make some additional configuration.

Thinking under the Network Environment of Multi-site Cluster

Threshold for cross-site heartbeat detection

As the cluster is deployed as a multi-site, there must be more or less delays in the network, so how to adjust the heartbeat detection threshold to the most appropriate, where the heartbeat detection threshold is the most critical, once the heartbeat detection threshold is reached due to network delay, or network quality, it will trigger failover, so it is important to ensure that the network quality is reliable and adjust according to the actual network delay. The detection threshold that can most accurately reflect the fault. If the multi-site network architecture uses extended VLAN, you can use the cross-site threshold definition function in WSFC 2016.

two。 Whether cross-site cluster communication is encrypted

By default, cluster communication with nodes in the same subnet will be signed, and usually there is no need to change this content. If your cluster architecture is cross-site and will go through internet, you can change the cluster communication security level to encryption, so that inter-cluster communication will be more secure through encryption. If your cross-site architecture is built through a separate secure channel, you can also unsign and encrypt. It should be noted that canceling cluster communication signature and encryption will improve performance, and if cluster communication encryption is used, it will bring a little performance degradation, because each time the node sends and receives traffic, there will be one more encryption and decryption process. If you need to change, it is recommended to test in advance to confirm that the performance degradation caused by encryption is acceptable, and then change it to encryption.

3. How to connect VM in multi-subnet environment

If we deploy a virtual machine in a multi-subnet environment, the network connection of the virtual machine is a problem. If the static IP configured by the virtual machine at the Beijing site is sent out through the Beijing virtual switch, the original IP of the virtual machine will not be able to communicate when the subnet is different in Shanghai.

Therefore, for VM in a multi-site environment, we usually have the following methods

Use DHCP IP addresses for virtual machines

Use a static IP for the virtual machine, but write a script inside the virtual machine to switch to the target static IP once a change in the network environment is detected

Extended VLAN network architecture is used for multi-site environment, and virtual machines are connected to the same subnet.

Use network virtualization for virtual machines to migrate virtual machines to different sites with IP

A better solution in Hyper-V replication and ASR can automatically set the virtual machine as the target IP after disaster recovery, so for virtualized disaster recovery, if you consider that multi-subnet WSFC is not convenient, you can also choose Hyper-V replication or ASR.

4. The problem of client connection delay in multi-site environment

The so-called client connection delay, that is, the cluster completed the failover, but the client is still unable to access the application, usually, there are two reasons, 1. The application requires additional configuration to provide access to 2.DNS client latency after the cluster failover is completed

Here we are mainly talking about the problem of DNS client delay. What is DNS client delay? the following figure is an example. If we use a multi-site and multi-subnet network architecture, we will face such a problem. VCO is a 10-segment IP in SiteA, register to DNS,DNS, copy this record to the SiteB,SiteB client to access VCO, and assume that the address is 10-segment. When a failover occurs, the address of the cluster transferred from SiteA to SiteB,VCO has changed. The modified record is replicated to DNS Server 2, and although the cluster has completed the failover and the DNS record has been replicated, the SiteB client will not be able to access the transferred service within 1200 seconds, because each record on the DNS server will have a HostRecordTTL time, during which the client will use the cached address instead of requesting a new address, so this is what we need to consider.

There are several ways to solve the DNS client latency problem

The network architecture that uses extended VLAN is all on the same subnet. There is no need to modify the address and DNS cache is not involved.

Using a network abstract device, the cluster network name is always registered with an abstract network device, and then the network device registers an abstract address with DNS, whether it is Site An or Site Bme DNS Server, which always faces the address of the abstract network device and does not involve DNS cache

Using the priority local transfer scheme, the preferred owner of the application is configured without a local node, and when the local owner fails, it is transferred to a cross-site

Optimize the DNS cache time and mechanism under multiple subnets: for multi-sites in the 2008 era, two new attributes have been added, namely, the time when HostRecordTTL and RegisterAllProvidersIP,HostRecordTTL attributes can modify the DNS cache. The default is 1200 seconds for the client to request a new address with DNS. The time for us to modify the network name of a cluster is 300 seconds, so that the client requests a new address with the DNS server more frequently. Microsoft recommends no more than 300 seconds at least, otherwise it will cause DNS server performance problems. RegisterAllProvidersIP attribute allows a network name to register the addresses of multiple subnets at the same time. By default, the network name corresponds to multiple OR relationship IP. Only one address will be registered at the same time. If the address of this network is not available, switch to another site and register another one. RegisterAllProvidersIP directly supports the registration of DNS records of all sites, but this function requires application support. SQL 2012 will start to support this function. The application actually tries to connect to an IP first, and if the attempt fails, it automatically connects to another address.

Arbitration

For multi-site clusters, arbitration is also a problem worth thinking about.

The witness should be there.

For a multi-site cluster, it is best not to place the witness on the multi-site itself, because there will be a certain bias effect. When a network partition occurs, as long as the party receiving the witness will start to provide services.

Therefore, it is recommended that for multi-station witness arbitration, it is best to put the file witness on the third site, disk witness, or use the cloud witness function of WSFC 2016, so that there is no favouritism effect, and that site can normally connect with the third site or cloud, that is, survive.

two。 Witness how the network should be designed

A failed witness network design is designed together with the heartbeat network and the external network, for example, if the external network lines of multiple sites are completely paralyzed, and the witness connection network and the external network use the same network link, then the witness connection network will also be paralyzed, and the disaster recovery site may not be able to start normally, so the witness connection must use a separate network to prevent network failure. And cause the witness to lose its effect.

3. Do you want to build a cold backup site?

Some enterprises may have the demand for cold backup sites, that is, under normal circumstances, a site that does not provide services will be activated only when there is a major disaster, such as a site in Beijing, a site in Tianjin, and a site in Shanghai. I hope Beijing is broken under normal circumstances, as long as it is transferred to Tianjin, and only as a last resort, you can build a cold backup site. There are two options for operation. 1. Disqualify the Shanghai site from voting, so the Shanghai site will not be eligible unless you force the Shanghai site to start again and give it a vote. two。 The setup application may only be owned by Beijing and Tianjin, which can achieve a similar effect, but if there are too many cluster applications, it will be troublesome to operate and need to be modified one by one.

4. Do you want to give priority to local site transfer?

When a disaster occurs, if a certain threshold is not met, there is no need to start a disaster recovery plan at the data center level. Disaster recovery can be achieved at the host level within the data center. In this case, you can configure the preferred owner of the application to be local, and there is no way to transfer it to cross-site, or if you use WSFC 2016, you can make use of the application site awareness feature to achieve multi-site operation of the application.

Or within the data center, for important applications, set up several cold backup machines, shut them down at ordinary times, turn them on in an emergency, force them to start, give votes, and join the cluster, but only if you witness the survival of the disk. The cold standby machine can get the latest cluster configuration database.

5. Code of practice for how to deal with brain fissure or a few site conditions

In the 2008R2 era, if we deploy a multi-site architecture, it is easy to encounter network problems and lead to brain cracks in the cluster. since 2012, Microsoft has added dynamic arbitration function. in the case of dynamic arbitration, we rarely see brain cleavage. In general, if there is a brain fissure, we will choose the most suitable site according to business needs and force it to start. Other sites need to be blocked when they start later. In order to force the startup site to synchronize the latest cluster database, therefore, the multi-site architecture needs to consider how to evaluate which party is the authoritative site and how to start the authoritative site in the case of brain fissure.

WSFC launched the dynamic arbitration function in 2012, that is to say, when the cluster is an even node, without witness, the cluster will always automatically remove a node's vote and maintain the cluster's non-odd voting. When the network partition occurs, the sites that have been removed to vote will decline, and the sites that have not been removed to vote will continue to provide services. We can specify through the LowerQuorumPriorityNodeID of the 2012 era or the PreferredSite function of the 2016 era. Let the cluster always remove the vote of a node, and finally achieve the effect of controlling the startup of the site. The use of this function can also be considered in the multi-site WSFC architecture. If there are multiple sites, it is hoped that a site will never win in the case of 50-50 nodes.

There is another situation, that is, a small number of nodes, when a disaster recovery occurs, there may be several sites, some sites have most nodes, some sites have a few nodes, normally the sites with most nodes should win, but we know that the sites with a small number of nodes are the sites that we most want to provide services, so we can prevent the majority nodes from starting and force the few nodes to start. This function needs to be planned in advance, and the application should be started at those sites first after disaster recovery. If something unexpected happens, the ideal site is a small number of nodes. What should I do?

Storage

Sharing storage there is a problem for multi-site clusters because we need to ensure that the cluster can be fully started at another site in the event of a disaster

If the shared storage of the cluster is placed in either data center at both ends, the other site cannot continue to provide services in the event of a disaster in that data center, because shared storage cannot be contacted.

Therefore, to set up a multi-site cluster, we also need to consider the problem of shared storage placement.

In general, when disaster recovery occurs at multiple sites, people will implement replication mechanisms for storage.

Device-based storage replication: directly select the array that supports storage replication, which is replicated when the storage is delivered to the cluster node, and the device is replicated based on the storage block level. If such device-level replication is implemented at multiple sites, it is best to have a dedicated line, so it will cost a lot of money.

Storage replication based on host software level: native storage replication similar to Symantec, SteelEye DataKeeper Cluster Edition, or Windows Server 2016 can be used. This kind of software will build the storage on multiple node operating systems into a logic, and the replicated disks will be delivered to the cluster disk identification. Now more and more people are using this scheme to achieve cross-site storage replication.

Storage replication based on application level: use directly similar to exchange dag,SQL ag, etc., applications can have storage replication technology

In addition to choosing the appropriate storage replication mechanism to ensure that storage is continuously available, we also need to choose the method of storage replication.

Use synchronous or asynchronous replication

Using synchronous replication, data will not be lost, and each write requirement will ensure that both sites are written at the same time before it is completed, which is easy to bring application delay and high network performance requirements.

With asynchronous replication, it is possible to lose data. Each write request ends only when it is written to the site, and then replicated to another site later, so that the application will not feel the delay, and the replication will be carried out bit by bit in the background later. It does not require high network performance, but there may be a disaster before replication, resulting in data loss.

In the real environment, Lao Wang sees that most enterprises are still using synchronous replication to ensure the integrity of the data.

Many people will consider DFS replication. In fact, the applicable scenario of Microsoft DFS replication is the information working group, which is used to store videos, files, pictures, and other materials. For clusters, or VMM libraries, DFS is not suitable, because DFS will only copy the data after it is closed. If we have virtual machines and databases in our cluster, DFS will not copy these files that will not be closed.

From the point of view of the Internet, arbitration and witness, Lao Wang explained to you the points that need to be considered in WSFC multi-sites, hoping to bring benefits to interested friends.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.