Shulou(Shulou.com)06/03 Report--

SMB multi-channel is a new function of Microsoft Windows Server in 2012. it is a part of Microsoft SMB3.0 protocol, which is mainly used to detect multiple communicable network cards when transferring SMB files, and automatically use multiple network cards for transmission to improve throughput and link fault tolerance.

One of the biggest differences between Windows Server 2012 and previous server operating systems is that 2012 uses the operating system itself to implement many technologies that only hardware can do, while working with major hardware manufacturers to enable devices of major hardware manufacturers to perform the best performance on Microsoft OS. For example, 2012 implements native SMB multi-channel, NIC teaming, and storage device awareness, ODX technology, and so on.

SMB multi-channel, many people may not care about, in fact, this is a very useful technology, we do not need to configure anything, Windows Server 2012 / Win8 after the OS native, as long as you have two machines, they need to SMB transmission, machine 1 has two network cards 192.168.1.1 and 192.168.2.1, machine 2 has two network cards 192.168.1.2 192.168.2.2 Then SMB multi-channel technology will automatically aggregate the bandwidth of the two network cards for protocol transmission. Once one of the network cards is broken, it will automatically use a single channel for transmission.

Through SMB multi-channel, we can achieve an effect similar to MPIO. Its advantage is to automatically aggregate multi-network card bandwidth, maximize the use of network card performance, automatic fault tolerance, automatic expansion, and find available channels to automatically join the transmission. The disadvantage is that SMB multi-channel can only be used in SMB protocol transmission, while other protocols are invalid, but SMB multi-channel technology can cooperate with NIC teaming technology, SMB protocol transmission uses SMB multi-channel, and other traffic uses NIC teaming.

SMB multi-channel technical requirements:

At least two computers running Windows Server 2012 Server 2012R2 or Windows 8 are required.

No additional features are required-SMB Multichannel is enabled by default

Multiple communicable network adapters

Optional requirement

One or more network adapters that support RSS

One of multiple network adapters configured using NIC Teaming

One or more network adapters that support remote direct memory access (RDMA)

SMB multi-channel scenarios

Multiple network cards that support RSS

This is a typical SMB multi-channel configuration. Before 2012, even if the two nodes had multiple network cards that could communicate, only one interface would be used to establish a connection when transmitting traffic, and all interface bandwidths would not be automatically aggregated. Starting from 2012, once multiple communicable network cards are detected, all network cards will be automatically aggregated. If the network card supports RSS, SMB creates multiple TCP / IP connections for a single session. Each interface has at least one or more connections, and this configuration enables SMB to use the available combined network adapter bandwidth and allows SMB clients to continue without interruption in the event of a network adapter failure.

Network cards in multiple NIC Teaming

Windows Server 2012 R2 and Windows Server 2012 support the ability to merge multiple network adapters into one network adapter by using a feature called network card binding. Although a Teaming always provides fault tolerance, when SMB does not have an SMB multichannel deployment, SMB can only create one TCP / IP connection per team. This configuration limits the number of CPU cores used and the maximum usage of queue bandwidth.

When deploying SMB using SMB multichannel, SMB creates multiple TCP / IP connections for a single session to achieve a better balance between CPU cores and better use of available bandwidth. NIC Teaming continues to provide failover, which is faster than using SMB Multichannel alone, and we recommend using network card combination binding because it provides failover for other workloads that do not depend on SMB, which cannot benefit from SMB multichannel failover.

Single or multiple network adapters that support RDMA

SMB multichannel detects the RDMA function of the network adapter, which makes the SMB Direct function (SMB Direct over RDMA) possible. When SMB is deployed using SMB multichannel, SMB detects the RDMA capability of the network adapter and creates multiple RDMA connections for this single session, each interface has two RDMA connections. Through this configuration, SMB can take advantage of the high throughput, low latency and low CPU utilization provided by the network adapters supported by RDMA. It also provides fault tolerance when you use multiple RDMA interfaces.

The performance improvement of RDMA technology in large data transmission is particularly significant. In 2012R2, this technology does not support NIC teaming, so if you want to obtain SMB Direct technology, in the 2012 era, you can only choose SMB multi-channel + RDMA,2016 era to support converged network architecture. RDMA technology can be obtained.

After a brief introduction to SMB multi-channel technology, let's take a look at its relationship with WSFC. In WSFC 2012, the cluster began to support SMB multi-channel technology, SMB multi-channel and clustering. We look at it from two levels.

Storage traffic

CSV redirect traffic

A typical scenario is Hyper-V over SMB, or SQL over SMB, with the application cluster in front and the storage cluster behind. The application accesses the storage cluster through SMB protocol, and uses RDMA technology and SMB multi-channel technology to improve transmission efficiency.

In this scenario, SMB multi-channel will only use the cluster network of role 1, that is, select the network with client and cluster communication enabled for storage traffic load balancing.

SMB multichannels in WSFC scenarios usually follow the following rules

1. Select cluster networks with different subnets and metrics as close as possible for SMB multi-channel summary.

two。 Only those network cards with the same network card speed, RSS, RDMA and other parameters will be included in the SMB multi-channel transmission.

3. If multiple network cards with different rates can be used, SMB multi-channel will establish a channel for the network cards with the preferred rate block.

What is a measure? in fact, this is an operating parameter hidden inside the cluster network, and we can't see it on the interface.

Starting with WSFC 2008, you can obtain it through the Powershell command.

Get-ClusterNetwork | ft Name, Metric, AutoMetric

In the era of WSFC 2008, by default, the internal network metric of the cluster is 1000, and the first internal network discovered by the cluster is 1000, and then the internal network increases by 100 in turn. The cluster selects the internal network on the basis of no network card gateway.

The cluster network metric for all external networks defaults to 10000, and then increases by 100. the cluster selects the external network on the basis of setting the network card gateway.

All cluster network metrics that do not have cluster communication default to 10100, followed by an increment of 100, based on the detection of network cards with ISCSI communication.

A typical architecture is as follows

Cluster networks get the lowest metrics, so in the era of WSFC 2008, CSV traffic, cluster database traffic, and heartbeat detection traffic will first pass through this network.

The live migration network card obtains the second lowest metric, so the cluster will use this network card for real-time migration. This decision can be overridden by selecting the real-time migration network card directly through the GUI interface.

By default, cluster network metrics are automatically generated by the cluster according to the algorithm, and can also be modified manually for high-end users

I have seen some foreign designs that usually set up backup network cards for management network cards and cluster network cards, such as this architecture.

Cluster1 900

Cluster2 910

MGMT1 10200

MGMT2 10300

One of the advantages of this design is that it can prevent the use of other network cards that should not be used once the cluster function or management function network card is broken, and the metric network card is planned in advance, and then the cluster automatically selects the next degree network card to use.

In the era of Windows Server 2012, the evaluation of cluster network metrics has changed, which can be based not only on roles but also on hardware functions.

Next, we will actually see the effect by actually building a SOFS cluster that uses SMB multi-channel.

Experimental environment

The current SOFS cluster has been configured

The SOFS cluster role has been configured

The cluster network settings are as follows

SMB multichannel is enabled by default, and you can see that when we copy files between the two nodes, all the network cards are called.

Disable the SMB multichannel feature

Copy the file again, it is found that there is no SMB multi-channel, only a network card will be called to copy.

Enable the SMB multichannel feature again

The copy process reinvokes all network cards

Looking at the cluster network metric, it is found that it is not very standard, and it is not what we want. The CLUS network card metric is not the lowest, and the CLUS MGMET SMB02 metric is too close, so it is easy to produce load balancing.

Redefine the measure CLUS3000 with a SMB01 of 50000, SMB02 of 50001, and MGMET of 70000. Planning and controlling SMB multi-channel transmission only over SMB01 SMB02 network

Above we looked at the effect of SMB multi-channel inside the SOFS cluster. Let's install the hyper-v cluster and take a closer look.

The current HV01 HV02 node has completed the Hyper-V cluster configuration

Create a new virtual machine and set the storage path to the SOFS cluster path

Looking at the SMB multi-channel in the node where the virtual machine is located, you can see that the current HV node is connected to the SOFS storage cluster node through 3040 channels, making use of the bandwidth of the two network cards at the same time, and achieving fault tolerance.

By looking at the network configuration, we can find that in the HVAC cluster, the network metric is standard, only the SMB01,SMB02 network metric is close, and other conditions are met, so the HV cluster gives priority to using these two links to connect.

In some cases, it may not be so smooth, and sometimes the metric calculation may not be like this, as we hope, there will be some deviation, resulting in other networks, such as the management network, the real-time migration network is also close to the storage network, so it is possible to produce load balancing, once you deploy the SOFS cluster and the front-end cluster. It is found that the front-end cluster does not use the desired link to connect to the SOFS cluster. In addition to modifying the metric, you can also choose to manually create a SMB multi-channel restriction policy to force the restriction of SMB multi-channel only on the network card we want.

First, in the front-end cluster, which refers to the HV or SQL cluster connected to the SOFS cluster, run Get-NetAdapter on it, view the network card index record, and write it down

Create a SMB multichannel restriction policy

New-SmbMultichannelConstraint-ServerName wisdom-InterfaceIndex 13pm 16

New-SmbMultichannelConstraint-ServerName wisdom.oa.com-InterfaceIndex 13pm 16

It is recommended to enter the SOFS VCO name and FQDN name in the place of ServerName. In this way, any name that the HV cluster connects to the SOFS cluster will only be called to our designated network card to complete the SMB multi-channel.

This setting needs to be set on each node of the HV cluster or SQL cluster

It should be noted that the SMB multi-channel restriction policy is meaningful only if the network card meets the SMB multi-channel requirements. For example, three network cards with the same speed and RSS,RDMA parameters, and similar metrics, but if you do not want to use these three network cards at the same time, you only want to use two of them, then you can use the SMB multi-channel restriction policy. If it is two network cards, but the speed is different, it is meaningless to create a SMB multi-channel restriction policy. Because SMB multi-channel must have the same speed of multiple network cards to build multiple channels, in the case of different rates, only high-speed network cards will be used to establish channels.

To set SMB multi-channel restriction policy, there are mainly the following scenarios

Prevent storage traffic from flooding the management network, hoping to manage more traffic on the network to do other things

Prevent CSV redirect IO traffic from flooding the management network

About CSV redirect IO traffic, Lao Wang will mention it later.

I believe you have a preliminary understanding of the application of SMB multi-channel in WSFC clustering. To put it simply, it is an innovation made by Microsoft in the field of SMB, which allows you to use its SMB architecture more. We can deploy a SOFS and deliver it to the front-end HV,SQL. The application uses RDMA SMB multi-channel technology to connect to the back end, achieving high performance and fault tolerance, which provides a new choice for IT infrastructure. SMB multi-channel can also be applied in clusters. In the WSFC 2012 era, as long as it meets the multi-subnet cluster network and the network card rate parameters are the same, SMB multi-channel technology can be used.

There are two main application scenarios of SMB multi-channel technology in WSFC 2012 cluster.

Traditional SMB traffic, the HV SQL SOFS cluster itself uses the SMB protocol to transport storage traffic. This type of SMB multi-channel selects the network in the cluster network that can be accessed by the client.

CSV redirect traffic

Under normal circumstances, all nodes in all a cluster directly access IO for CSV-storage, and a write request is written directly to storage through CSV, which becomes north-south traffic, and there is a kind of east-west traffic. Assuming that a single node in the cluster loses access to storage, that is, there is a problem with the node-to-link, then the CSV volume will enter redirect mode. Although the application connected to the CSV in the upper layer will not be interrupted, it will feel a significant decline in performance, so each time the IO is written, the disqualified node redirects the IO to the node that can access the storage, and then the node completes the IO request.

In general, CSV volumes enter redirect mode, and there are several possibilities

Single node loses access to storage

Incompatible filter drivers are installed on the node

CSV backup, 2008 R2 is unable to create a single synchronous VSS snapshot of CSV, resulting in a large number of redirects of IO during backup operations, and some storage systems are under pressure, causing the 2008 R2 Hyper-V cluster to crash

In the 2012 era, the CSV backup mechanism has been changed, no longer using VSS backup operations, but using the distributed snapshot mechanism to simply back up CSV volumes

So there are probably several possibilities for CSV redirection in the 2012 era. Storage link 2. Node filter driver interference 3. Metadata update

In general, the amount of metadata update data is very small, we do not need to care too much about it, filter drivers usually do not appear if the node is clean

In the era of WSFC 2012, CSV redirection was most likely to occur in the scenario of single-node storage link failure or restore operation.

In fact, CSV redirect traffic is a very dangerous traffic. In the 2008 era, many people did not know this, so they put a cluster communication network with very low bandwidth. It was found that when there was a problem with a single node storage link, the whole cluster could hardly perform IO operations in CSV redirection mode.

In the era of WSFC 2008, for CSV redirected traffic, only the cluster network with the lowest metric was used by default to complete the east-west traffic when redirected.

Since the beginning of WSFC 2012, CSV redirect traffic has a new change. Instead of selecting based on metrics, SMB protocol and SMB multichannel are used to complete the traffic transmission during CSV redirection. Using the new way to transmit CSV redirect traffic will be four times higher than the original way, and better deal with the performance problems of CSV redirection. For SMB multi-channel selection when CSV redirection, the following rules will be followed

1.SMB multichannel takes precedence over the network priority of NetFT to determine which subnets are used for CSV redirect traffic

2.CSV redirection will only use the internal cluster network for SMB Multichannel, this behavior can be changed to also use the external network to modify the UseClientAccessNetworksForSharedVolumes cluster parameters

3.SMB multichannel requires the same link speed and characteristics (RSS and / or RDMA) to simultaneously carry CSV redirected traffic on different subnets

4. If the adapters are different, SMB Multichannel will use faster adapters to carry CSV redirected traffic

5. The failover cluster fails back to NetFT to decide which subnet to use only if SMB Multichannel is not available or disabled. The lowest metric logic is then applied and CSV redirected traffic is sent through the lowest metric subnet

If you do not want to redirect traffic using SMB multichannel, but only a single network, then you can directly turn off the node SMB multichannel feature

Set-SMBClientConfiguration-EnableMultichannel $False "

In an ideal situation, a cluster with SMB architecture

Two different SMB multichannel types should be used, and both SMB multichannel network types are executed according to the rules.

Storage traffic uses external network for SMB multi-channel

CSV redirect traffic uses cluster internal network for SMB multichannel

If you encounter CSV redirect traffic flooding other networks in the WSFC 2012 era, such as failed to restore a backup, CSV redirect traffic calls the CSV network card and the management network card, flooding the management network card, causing the management network card to be inaccessible. At this time, you can use New-SmbMultichannelConstraint to force restrictions on CSV multi-channel SMB multi-channel traffic.

Recommendation 1. Do not use backup tools that generate CSV redirects. 2. Plan specific network cards to complete CSV redirection of SMB multichannel traffic

For the scenario here, Lao Wang believes that not many people use this kind of SMB architecture for enterprise production in China, and it is also a little troublesome to complete two sets of SMB multi-channels in a cluster, which requires different SMB multi-channel strategies, so how to achieve a set of SMB multi-channels to complete storage traffic and CSV redirect traffic? Lao Wang thought of a way to modify a parameter of the cluster, UseClientAccessNetworksForSharedVolumes. Let CSV redirect traffic also use external type of cluster network. In this way, as long as the storage traffic and CSV redirect network card type Nic, rate, RSS, RDMA parameters are the same, you can use SMB multi-channel feature. At that time, it will be a group of SMB multi-channel to complete storage traffic and CSV redirect traffic. You can also use New-SmbMultichannelConstraint to force restrictions on CSV multi-channel SMB multi-channel network cards.

So a SOFS cluster is to manage one network card and two heartbeats for external storage cards. The external storage network completes not only SMB storage traffic multichannel, but also CSV redirect traffic multichannel.

We have seen the above.

Introduction of SMB multi-channel technology

Application and requirements of SMB Multi-Channel in WSFC Cluster

Application of SMB Multi-Channel in SOFS Cluster

Application of SMB Multi-Channel between SOFS Cluster and HV Cluster

Two different types of traffic for SMB multichannel: storage connection traffic and CSV redirect traffic

In the era of WSFC 2016, the combination of SMB multi-channel technology and cluster becomes more harmonious. The cluster no longer requires multiple subnets to be SMB multi-channel in the cluster. Now multiple network cards in the same subnet can also build multi-channel in the cluster. In fact, SMB multi-channel technology itself supports multiple network cards in the same subnet, or multiple network cards in different subnets. Only that the WSFC of the 2012 era is limited, when building a network topology. Multiple NICs in the same subnet will not be built into the cluster network topology. The cluster modified this rule in the WSFC 2016 era, and multiple NICs in the same subnet can also be built into the cluster network topology to achieve cluster SMB multi-channel.

The so-called construction of multiple network cards on the same subnet to the cluster network topology does not mean that multiple cluster networks will appear in the same subnet, but that multiple network cards under a single subnet can be included in a cluster network.

In this example, there are two 16node1 16node2 nodes, with four network cards in each node, which are MGMET SMB01 SMB02 Heart respectively.

Among them

Node 1 SMB network card 30.0.0.18 30.0.0.19

Node 2 SMB network card 30.0.0.0.20 30.0.0.21

By copying files between two nodes, you can see that SMB multi-channel invokes all network cards

Create a SMB multi-channel restriction policy and execute it on each node separately

Copy it again and find that only the selected network card is called

Run the command to check and find that the current file copy for the HV02 node is only carried out through four network cards of 30 network segments.

In addition to limiting SMB default traffic, we can also limit CSV redirect traffic

Configure clustering to allow CSV redirect traffic to use external networks (optional, it is recommended that multiple internal communication cards can be deployed separately)

Create a new storage node CSV redirect SMB multi-channel restriction policy respectively

For the CSV redirect SMB multi-channel restriction policy, server name cannot enter a regular machine name. According to the result, it should be a virtual address from NETFT. This name can be obtained from the following figure command.

This strategy requires you to set the names of other nodes on each CSV node that connects to the storage.

After the creation is completed, simulate CSV redirection

As can be seen from the command, node 1 simulates the CSV redirection mode, so node 1's east-west traffic access to CSV will be completed through node 2.

Careful friends may find that the commands here are different from those in 2012. 2016 Get-SmbMultichannelConnection commands have an extra SmbInstance parameter.

From this parameter, we can see that the network cards used by SMB multichannel in different types

Default: traditional SMB transport

CSV:CSV redirection

SBL:2016 SDS specific traffic

SR:2016 storage replicates specific traffic

Because of the limited environment, I did not fully show the back-end SOFS cluster and the front-end HV cluster in the WSFC 2016 scenario.

In this case, we take a fusion cluster as an example.

In fact, if it is the front-end HV cluster, the back-end SOF cluster is the best architecture.

In this architecture,

The SMB multi-channel restriction policy should be designed like this

The HV cluster design is completed by the two network cards to complete the storage connection SMB storage connection multi-channel

The SOFS cluster design is performed by the two network cards to complete the CSV redirection multichannel.

For SMB storage multi-channels, you only need to design on the HV side. The SOFS cluster detected that the other party only opened two channels to me, and automatically selected two channels to establish a connection.

SOFS clusters connect directly to storage and carry CSV volumes, so it is necessary to design CSV redirect traffic multi-channel, which can allow CSV redirection to use external network cards by adding multiple internal cluster network cards, and then through policy restrictions, or by modifying parameters.

It should be noted that the SMB multi-channel restriction policy requires entering a ServerName name. In the completion scenario, this policy should be created on each node of HV or SQL. Enter the VCO and VCO FQDN names of the SOFS.

In this example, we are used to simulate the name of the other node entered by the test, which is not true in the actual scenario!

The restriction on CSV redirection SMB multichannel is special, because its Server Name does not use the node name, but uses a strange address, which can be obtained by simulating the CSV redirection in advance, and then create a SMB multichannel restriction policy for CSV redirection.

The above is an introduction to the new features of WSFC 2016 SMB multi-channel. Starting from WSFC 2016, clusters are supported to build SMB multi-channel using different network cards in a single subnet, and a new SMB Instance command is added to view the use of SMB multi-channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.