Shulou(Shulou.com)06/03 Report--

In the previous two articles, Lao Wang and everyone introduced the function of storage replication. In the practice of stand-alone to stand-alone and extended clusters, basically you can see that this is a block-based storage replication, which is natively built into Windows Server 2016, which can help us to achieve disaster recovery in stand-alone scenarios. It can also be combined with clustering to achieve the combination of storage and application, when a site-level disaster occurs. Automatic transfer of storage and application

For engineers, now there is a new choice in architecture. We can use Microsoft's native OS to achieve high availability + disaster recovery without the help of devices or third-party products.

Although it is good to extend the cluster, there is still a drawback, that is, each node cannot directly use the local disk of each node to complete storage replication, and each site node must also access the storage of its own site. the two sites need to complete replication in the way of asymmetric shared storage. Lao Wang guessed that the original intention of Microsoft is to use the function of extended clustering for shared storage at different sites. For example, each node of the Beijing site accesses the storage of the Beijing site, and each node of the Tianjin site accesses the storage of the Tianjin site. Then, storage replication is done between the two Site to achieve the effect of extending the cluster. Even if the storage and computing nodes of the local site all crash, the other site can also use the

However, Lao Wang guessed that it was still due to the clustering mechanism, which led to the fact that the extended cluster could not use the local disks of each node at present, because at the beginning of Microsoft's cluster design, it was required to achieve permanent retention of cluster storage and shared storage of cluster data. in order to switch direct mount, if the local disk directly enters the cluster disk, there will be some problems on the failover.

Although Microsoft 2016 has S2D technology and can achieve super-converged storage, there are still some differences between its function and the disaster recovery we are talking about. Lao Wang believes that S2D is a technology that combines storage and clustering. We enable S2D on the cluster, and then pool the disks of each node together through a storage bus, but pay attention. This is not to say that the local disks of each node go directly to the cluster disk, but the available disks of all nodes can be seen in the area of the chassis in the cluster storage. after we open S2D, we can logically build a storage cabinet, the storage in this storage cabinet is the disk from each node, how to use it next? We also need to divide the Lun on this logical storage cabinet to connect to each node, so we create storage pools, virtual disks, fault tolerance, tiering, caching, and finally deliver virtual disks to the cluster. The virtual disk can be regarded as a cluster disk, but in fact only one cluster disk will be displayed, even if the virtual disk is created by multiple nodes through the logical storage cabinet.

So, in Lao Wang's view, S2D implements a logical storage cabinet with the help of clustering, and then allocates storage to cluster applications, so you can see that this concept is still somewhat different from our storage replication. Relatively speaking, it seems that the industry extends the cluster through a super-converged architecture like S2D.

Direct underlying S2D logical storage cabinet level to do a good job of site awareness, and then assigned to the cluster, storage at the bottom has been done fault tolerance, why Microsoft did not implement this architecture, Lao Wang guessed that it should be a problem of optimization and mechanism

1. Currently, S2D can only be rack-aware and chassis-aware, but not site-aware, that is, there is no way to control the spread of data to different sites.

2.S2D is too demanding for network performance to extend cluster scenarios

Also maybe, after a few versions of Microsoft will do site-aware optimization for S2D, then we can achieve super-fusion extension cluster, or disaster recovery extension cluster

As far as the current situation is concerned, we can only achieve the effect of extending the cluster through disaster recovery, so many friends may ask why the extended cluster cannot be used in combination with S2D.

First of all, Lao Wang believes that these are two solutions to different scenarios in the Microsoft ecological environment.

Extended clustering is designed to handle disaster recovery of cross-site clusters and storage, replicate storage at both sites, and ensure that site-level disasters are acceptable

S2D is to deal with expensive storage devices, cluster storage through local disks, and management functions of storage devices through operating system software.

If we use S2D, two sets of storage will not be displayed in the same cluster, only one set of storage will be displayed, because S2D is a storage fault tolerance implemented at the bottom.

The extended cluster requires two sets of storage to achieve storage replication at different sites.

So according to the current position, Lao Wang believes that the extended cluster will not fit in with S2D unless there is a big change in the later version.

Discard the extended cluster currently implemented through the storage replication mechanism, optimize S2D site awareness, and achieve extended clustering

Optimize S2D site awareness to implement two extended cluster models

S2D architecture enables a click key, whether to achieve extended cluster, if so, not through the underlying fault tolerance, but to provide two sets of storage to achieve storage replication

As far as Lao Wang is concerned, I think the odds of 1 and 3 are small, and the probability of 2 is greater. Anyway, the network is not a bottleneck now. If the enterprise has money, of course, you can choose to use S2D to extend the cluster.

In fact, Lao Wang thinks that the extended clustering effect of the current storage replication is also quite good. There is no advanced storage device for the enterprise, but he also wants to achieve cross-site disaster recovery. Through storage replication + clustering, very low RTO and RPO can be achieved, and disaster switching can be fully automated, which is a major advantage.

So this is some discussion of extended cluster storage replication, S2D, and for Windows Server 2016 storage replication, we have another scenario, cross-cluster replication, extending storage replication beyond the boundaries of the cluster.

In this scenario, we can combine the S2D scheme with the storage replication scheme.

As we mentioned earlier, the storage replication solution does not require storage, and the underlying layer can be local SCSI/SATA,ISCSI,Share SAS, SAN,SDS. For extended clusters, it may be an architecture that requires asymmetric shared storage, such as ISCSI,SAN,JBOD, but for stand-alone or cluster-to-cluster, there is not so much. We can use local disk and SDS architecture. In the stand-alone architecture, it is equivalent to, I provide you with a storage for the system OS, no matter how I got it, as long as both sides meet the requirements, storage replication can be established, and the cluster is basically the same to the cluster. One cluster can be regarded as a large host, and two clusters are two big hosts. Never mind how the storage of my cluster comes from. In short, my cluster has storage that meets the requirements. Cluster-to-cluster replication can be performed as long as the storage configurations of both clusters meet the requirements.

There are a lot of scenes to play with.

For example: Beijing site cluster storage architecture is ISCSI, Tianjin site cluster storage architecture is S2D, and then do storage replication for two clusters. Once the ISCSI primary storage fails to provide services, immediately switch to Tianjin station to provide services, or Beijing site uses physical machine deployment, Tianjin site uses virtual machine deployment, Beijing site uses private cloud deployment cluster, and then deploys cluster in public cloud. Local clusters and storage are down, and clusters and storage in the public cloud can be started

Benefits of clustering to cluster replication

1. There is no need to configure additional site awareness. If the Beijing cluster is at the Beijing site, it will only be a failover within the cluster.

two。 The underlying storage architecture of the two sites can be different to avoid failure of a single type of storage.

3. Support for synchronous and asynchronous replication

4. Support more complex architectures, such as North China cluster and South China cluster, North China cluster consists of Beijing and Tianjin nodes, South China cluster consists of Guangdong and Shenzhen nodes, site fault awareness is configured within each site, it does not matter that Beijing node is broken or Tianjin node is broken, the application will drift to similar sites with the same cluster, if all clusters in North China are down You can also restart the cluster role in the South China cluster.

5. Help achieve cluster-level disaster recovery. For example, if the Beijing cluster configuration fails, it will not affect the Tianjin cluster, and the Tianjin cluster can use the replicated storage to start the cluster role.

6. Two sites can use different network architectures to achieve fault tolerance of subnets

The disadvantages of cluster to cluster replication

1. Need to manually use commands for failover, no graphical interface operation

two。 When failing over, you need to prepare the cluster role in the backup cluster in advance, and then mount the storage again.

3. For file servers, access with a new name after a cross-cluster failover

Deployment requirements for cross-cluster replication technology

The operating system of each replication node needs to be Windows Server 2016 data center version

At least four servers (two in each of the two clusters), supporting up to two 64-node clusters

Two sets of shared storage, SAS JBOD, SAN, Share VHDX, S2D or iSCSI, with each storage group set to be available only for each cluster

Replication nodes need to install File Server role, storage copy function, and failover clustering function

Active Directory domain environment, which provides Kerberos verification for each node in the replication process

Each replication cluster requires at least two disks, one data disk and one log disk

The format of data disk and log disk must be GPT. MBR format disk is not supported.

Both cluster data disks and partitions must be the same size, maximum 10TB

Both cluster log disk size and partition size must be the same, at least 8GB

The CSV or file server roles need to be assigned to the disks that need to be replicated within each cluster

Storage replication uses port 445 (SMB-replication transfer protocol), port 5895 (management protocol of WSManHTTP-WMI / CIM / PowerShell), port 5445 (iWARP SMB-only required when using iWARP RDMA networks)

Cross-cluster replication planning recommendations

It is recommended to use SSD or NVME SSD for the log disk. Storage replication first writes data to the log disk. Good log disk performance can help improve write efficiency.

It is recommended that you plan for larger log space, which allows faster recovery from larger outages, but consumes space costs.

To prepare reliable and high-speed network bandwidth for synchronous replication scenarios, it is recommended that 1Gbps start, preferably 10Gbps. Synchronous replication scenarios will delay the write request time of the application if the bandwidth is insufficient.

Asynchronous replication is recommended for cross-cloud, multinational, or long-distance cross-cluster replication

Other Microsoft technologies that can be integrated in cross-cluster replication

Deployment: Nano Server, SCVMM

Management: PS,Azure ASR,WMI,Honolulu,SCOM,OMS,Azure Stack

Integration: Hyper-V,Storage Spaces Direct, Scale-Out File Server,SMB Multichannel,SMB Direct, duplicate data deletion, ReFS,NTFS

Microsoft implementation of cross-cluster replication scenario

This example simulates two remote clusters replicating each other. There are two nodes in Beijing site, two nodes in Tianjin site, one DC in Beijing site, which also hosts ISCSI server role. Beijing site cluster uses ISCSI storage architecture, Tianjin site two nodes, using S2D architecture to simulate node downtime, site downtime, and reverse replication after site recovery.

AD& Beijing ISCSI

Lan:10.0.0.2 255.0.0.0

ISCSI:30.0.0.2 255.0.0.0

16Server1

MGMT: 10.0.0.3 255.0.0.0 DNS 10.0.0.2

ISCSI:30.0.0.3 255.0.0.0

Heart:18.0.0.3 255.0.0.0

16Server2

MGMT: 10.0.0.4 255.0.0.0 DNS 10.0.0.100

ISCSI:30.0.0.4 255.0.0.0

Heart:18.0.0.4 255.0.0.0

Tianjin site

16Server3

MGMT: 10.0.0.5 255.0.0.0 DNS 10.0.0.2

S2D:30.0.0.5 255.0.0.0

Heart:19.0.0.5 255.0.0.0

16Server4

MGMT: 10.0.0.6 255.0.0.0 DNS 10.0.0.2

S2D:30.0.0.6 255.0.0.0

Heart:19.0.0.6 255.0.0.0

Because we use the cluster-to-cluster architecture, the network is also more flexible, for example, the heartbeat network does not have to be in the same network segment, because the heartbeat network is at the cluster level, even if you are in one network segment after crossing the cluster. the other cluster does not know that the management network can also use different subnets, for example, if you do not do multiple replication groups and do not do more work. Then the external network of Tianjin can be set under a secure network segment and normally does not provide external services, and then restrict the storage replication traffic from the ISCSI card to the S2D card through the network policy, so that you only need to get through the storage traffic of the two clusters. If Tianjin has DC, it is best for the cluster to use Tianjin site DC. When a disaster occurs, connect the network in Beijing with the management network in Tianjin, and let Beijing users access the role of Tianjin.

Operation flow

1. Beijing site access storage

two。 Beijing site creates a cluster

3. Beijing site creates cluster roles

4. Tianjin site access storage

5. Tianjin site creates a cluster

6. Tianjin site opens S2D

7. Tianjin site creates cluster roles

8. Assign mutual cluster permissions

9. Create storage replication

10. Site disaster recovery

Beijing site access storage

16server1

16server2

Beijing site creates clusters and configures cluster arbitration as file sharing witness or cloud witness

Beijing site creates file server cluster role

As you can see, the cluster role created by Lao Wang here is SOFS, which is the difference between cross-cluster replication and extending the cluster. Microsoft explained that the load is only CSV and the traditional highly available file server role when extending the cluster, but when replicating across clusters, the scenario becomes capable of supporting SOFS. We all know that SOFS mainly emphasizes the dual activation of file servers and transparent failover. Because failover requires storage and then role transfer in an extended cluster, it is difficult to achieve transparent failover. In cross-cluster replication, SOFS can be deployed within each cluster to achieve transparent failover within each cluster, but there is no way to be completely transparent for site failures.

Each node in Tianjin accesses local storage of arbitrary size. Finally, we will recreate the cluster disk on the cluster storage space.

16server3

16server4

After getting the storage, each node puts the disk online and initializes it to a GPT disk. Do not partition the disk directly. We need to deliver it to the cluster through the upper layer of the S2D package.

Tianjin site creates S2D cluster

New-Cluster-Name TJCluster-Node 16Server3J 16Server4-StaticAddress 10.0.0.20-NoStorage

Note: ensure that the node meets the S2D requirements before execution, and cluster verification is recommended in the production environment

Enable the cluster S2D function

Enable-ClusterS2D-CacheState disabled-Autoconfig 0-SkipEligibilityChecks

If you simulate this experiment on vmware workstation, you will find that the process of opening S2D will always be stuck here, and there is no way to proceed. Check the log and find that S2D has been unable to identify the disk.

But what is the reason why the disk can not be identified? the disk on the vmware is SAS bus, which also meets the requirements of S2D. After some research, Lao Wang found the reason. The virtual machine virtualized through vmware does not have Serial Number, while the number that opens S2D will find the disk, so the disk cannot be declared to be S2D available.

You can add parameters by shutting down the vmware virtual machine vmx configuration file.

Disk.EnableUUID= "true"

After the addition is completed, you can see that the Serial Number can be displayed normally.

Configure each node disk MediaType label

Open S2D again and pass smoothly. Now we can simulate the S2D experiment on vmware workstation!

Create a clustered storage pool

Create a new virtual disk (storage space), and the virtual disk delivered here is the cluster disk, so we need to ensure that the size of the cluster disk is the same as that of the Beijing cluster in order to achieve storage replication.

Data disk

Log disk

Tianjin site configuration cluster role

Grant permission to Beijing site cluster on any node of Tianjin site

Grant-SRAccess-ComputerName 16server3-Cluster BJcluster

Grant permission to the Tianjin site cluster on any node of the Beijing site

Grant-SRAccess-ComputerName 16server1-Cluster TJcluster

Create a cluster-to-cluster replication relationship

New-SRPartnership-SourceComputerName bjcluster-SourceRGName rg01-SourceVolumeName C:\ ClusterStorage\ Volume1-SourceLogVolumeName R:-DestinationComputerName tjcluster-DestinationRGName rg02-DestinationVolumeName C:\ ClusterStorage\ Volume1-DestinationLogVolumeName R:

After the completion of the replication relationship, the data disk of the current Tianjin cluster will become online (no access). Here, we need to be aware that for cross-cluster, we now put the boundary of replication across the cluster. therefore, during the operation of each replication group, there will be a cluster data disk in standby state, for standby cluster, the disk in the replication group cannot be used. If multiple cross-cluster replication groups are deployed, cluster applications can be on standby with each other.

The current file server cluster role provides services at the Beijing site, the owner node is 16server1, and there are already some data files

Direct power outage 16server1dSOFS automatic transparent failover to 16server2 within the same site

Storage replication continues at the same site 16server2, so you can see that storage replication, like Hyper-V replication, has a replication coordination engine within the cluster, which replicates through the cluster name and the external cluster, and then coordinates internal replication by that node. Once one cluster node goes down, it automatically coordinates another node to participate in replication. The difference is that Hyper-V replication has an explicit cluster role in the cluster. The cluster role of storage replication is hidden

Next, when all the disasters occurred at the Beijing site, and 16 server1 and 16 server2 all lost power, the cluster that came to the Tianjin site could see that the current cluster disk was offline and would not automatically complete the cross-cluster failover.

Manually force a cross-cluster failover

Set-SRPartnership-NewSourceComputerName tjcluster-SourceRGName rg02-DestinationComputerName bjcluster-DestinationRGName rg01-force

After the execution is completed, the current data can be read at the Tianjin site.

When you open the data disk CSV, you can see that the data of the Beijing site has been completely copied.

After the actual test of cross-server failover, the shared folder permissions will not be automatically migrated. By default, they will be in an unshared state and can be shared manually. It is guessed that it is because of disk transfer across servers. If there are more permissions, you can try to cooperate with the permission transfer. However, in fact, for this cross-server failover load, it is recommended to use SQL files in the actual production environment. The VHDX,VHD file is the main file, and the database or virtual machine role is restarted directly in the new cluster after a cross-cluster failover.

Now that we have completed the cross-cluster disaster recovery, if we adopt the network separation architecture, we can expose the access to the Tianjin site to Beijing users. As we said before, the transfer will be accessed with a new name.

Even if another node of the Tianjin site is broken, the SOFS can transparently fail over to the only remaining node to survive, provided the arbitration is properly configured.

When there is only one last node left, storage replication is paused, waiting for the other nodes to resume

After waiting for a period of time, the servers of each site have been repaired and launched one after another, and the storage replication automatically returns to normal continuous replication. The current main replication site is the Tianjin site.

If you want to continue to restore the primary site from the Beijing site at this time, you can perform reverse replication, copy the content of the Tianjin site back to the Beijing site, and restore the master / standby relationship.

Through three introductions to the storage replication function of Windows Server 2016, I believe everyone has learned about this technology. I am glad to hear that some friends have read my blog and have begun to use it in the actual environment. Storage replication technology is a major progress of 2016 storage. Native block-level replication, if there are few nodes and do not want to maintain the cluster, you can use stand-alone scenarios to achieve disaster recovery. If you want the cluster to get the lowest RTO/RPO, you can use the extended cluster function to achieve automated disaster recovery. If you want to plan a more complex architecture, you want to use different network and storage architectures for different sites to achieve cross-cluster and cross-site-level disaster recovery. It can also be achieved now. Finally, I hope everyone can have their own thoughts after reading the article.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.