Shulou(Shulou.com)06/01 Report--

What is the principle and usage of multipath multi-path under LINUX? in order to solve this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

1. What is multipath

Ordinary computer mainframe is a hard disk attached to a bus, here is an one-to-one relationship. When it comes to the SAN environment composed of optical fiber, or the IPSAN environment composed of iSCSI, because the host and storage are connected through the optical fiber switch or multiple network cards and IP, then it constitutes a many-to-many relationship. That is, there can be multiple paths from the host to the storage. The IO from the host to the storage can be selected by multiple paths. Each host can go through several different paths to the corresponding storage. If it is used at the same time, how is the Icano traffic allocated? One of the paths is broken, how to deal with it? And from the operating system's point of view, each path will be regarded as an actual physical disk, but in fact it is only a different path to the same physical disk, which brings confusion to the user when using it. Multipath software came into being to solve the above problems.

The main function of multipathing is to work with storage devices to achieve the following functions:

1. Failure switching and recovery

Load balancing of 2.IO traffic

3. Virtualization of disk

Because multipath software needs to be used in conjunction with storage, different vendors provide different versions based on different operating systems. And some manufacturers, software and hardware are not sold together, if you want to use multipath software, you may need to buy license from manufacturers. For example, EMC's multipath software based on linux needs to buy license separately. Fortunately, the kernel of RedHat and Suse 2.6 comes with a free multipath package that can be used for free, and it is also a general-purpose package that can support devices from most storage vendors, even if some are not well-known vendors, they can support and run well by making minor changes to the configuration file.

2. Introduction to multipath under Linux requires the following toolkits:

In CentOS 5, when you install the minimum system, multipath has already been installed, and check to see if multipath is installed as follows:

1. Device-mapper-multipath: multipath-tools. Mainly provides tools such as multipathd and multipath and configuration files such as multipath.conf. These tools create and configure multipath devices (calling device-mapper 's user-space library) through the interface of device mapper's ioctr. The multipath device created will be in / dev / mapper).

2. Device-mapper: it mainly includes two parts: the kernel part and the user part. The kernel part is mainly composed of the device mapper core (dm.ko) and some target driver (md-multipath.ko). The core completes the mapping of the device, while target specifically deals with the iUnix from mappered device according to the mapping relationship and its own characteristics. At the same time, in the core part, an interface is provided, and users can communicate with the kernel part through ioctr to guide the behavior of kernel drivers, such as how to create mappered device, the properties of these divece, and so on. The user space section of linux device mapper mainly includes the package device-mapper. These include dmsetup tools and libraries to help create and configure mappered device. These libraries are mainly abstract and encapsulate the interface to communicate with ioctr to facilitate the creation and configuration of mappered device. These libraries need to be called in multipath-tool 's program.

3. Dm-multipath.ko and dm.ko:dm.ko are device mapper drivers. It is the basis for implementing multipath. Dm-multipath is actually a target driver for dm.

4. Scsi_id: included in the udev package, you can configure the program in multipath.conf to get the serial number of the scsi device. By the serial number, it can be judged that multiple paths correspond to the same device. This is the key to multipath implementation. Scsi_id uses the sg driver to send EVPD page80 or page83 inquery commands to the device to query the identity of the scsi device. However, some devices do not support EVPD's inquery command, so they cannot be used to generate multipath devices. However, scsi_id can be rewritten to virtualize an identifier for devices that cannot provide a scsi device identity, and output to standard output. When the multipath program creates a multipath device, it calls scsi_id to get the scsi id of the device from its standard output. When rewriting, you need to modify the return value of the scsi_id program to 0. Because in the multipath program, the straightforward is checked to determine whether the scsi id has been successfully obtained.

3. Basic configuration process of multipath in CentOS 5:

1. Install and load multipath software packages

# yum-y install device-mapper device-mapper-multipath

# chkconfig-level 2345 multipathd on # set to boot self-boot multipathd

# lsmod | grep dm_multipath # to check whether the installation is normal

If the module does not load successfully, initialize the DM or restart the system with the following commands

-Use the following commands to initialize and start DM for the first time:

# modprobe dm-multipath

# modprobe dm-round-robin

# service multipathd start

# multipath-v2

2. Configure multipath:

The configuration file of Multipath is / etc/multipath.conf. If you want multipath to work properly, you only need the following configuration: (if you need more detailed configuration, please see the introduction later in this article)

Blacklist {

Devnode "^ sda"

}

Defaults {

User_friendly_names yes

Path_grouping_policy multibus

Failback immediate

No_path_retry fail

}

# vi / etc/multipath.conf

3. Multipath basic operation command

# / etc/init.d/multipathd start # enable mulitipath service

# multipath-F # Delete an existing path

# multipath-v2 # format path

# multipath-ll # View multipath

If configured correctly, there will be more devices such as mpath0, mpath2, and so on in the / dev/mapper/ directory.

With the fdisk-l command, you can see the disks created by multipath software, such as / dev/dm- [0-3] in the following figure.

4. Basic operation of multipath disk

To operate on the disks generated by multipath software, you can directly manipulate the disks in the / dev/mapper/ directory.

It is a good idea to run the pvcreate command before partitioning disks generated by multipathing software:

# pvcreate / dev/mapper/mpath0

# fdisk / dev/mapper/mpath0

When using fdisk to partition and save the disk generated by multipath software, there will be an error, which should be ignored.

After fdisk partitions the disks generated by multipath software, the resulting partitions are not immediately added to the / dev/ directory. At this point, we need to restart the IPSAN or FCSAN drivers.

Iscsi-initiator to connect to IPSAN's restart ISCSI service can find the generated disk partitions.

# service iscsi restart

# ls-l / dev/mapper/

For example, mpath0p1 and mpath2p1 in the figure above are the partitions of the multipath disk.

# mkfs.ext3 / dev/mapper/mpath0p1 # format mpath2p1 partitions into ext3 file system

# mount / dev/mapper/mpath0p1 / ipsan/ # Mount mpath2p1 partition

IV. High configuration of multipath

All of the above use the default configuration of multipath to complete the configuration of multipath, such as mapping device name and multipath load balancer. Is there any way to configure multipath according to our own definition? yes, we can.

1. Configuration of multipath.conf file

The next job is to edit the configuration file for / etc/multipath.conf

Multipath.conf mainly includes the configuration of blacklist, multipaths and devices.

Blacklist configuration

Blacklist {

Devnode "^ sda"

}

The Multipaths part configures the configuration of multipaths and devices.

Multipaths {

Multipath {

Wwid * # this value multipath-v3 can be seen

Alias after alias iscsi-dm0 # mapping, you can take whatever you want

Path_grouping_policy multibus # path Group Policy

Path_checker tur # method of determining path status

Path_selector "round-robin 0" # method of selecting that path for the next IO operation

}

}

Devices partial configuration

Devices {

Device {

Vendor "iSCSI-Enterprise" # Vendor name

Product "Virtual disk" # product model

Path_grouping_policy multibus # default path group policy

Getuid_callout "/ sbin/scsi_id-g-u-s / block/%n" # gets the default program for unique device numbers

Default program used by prio_callout "/ sbin/acs_prio_alua% d" # to obtain finite order values

Path_checker readsector0 # method of determining path status

Path_selector "round-robin 0" # method of selecting that path for the next IO operation

Failback immediate # failure recovery mode

No_path_retry queue # the number of times the system attempted to use the failure path before disable queue

Rr_min_io 100 # the number of IO requests in the current user group before switching to another path

}

}

The following is a complete configuration file

Blacklist {

Devnode "^ sda"

}

Defaults {

User_friendly_names no

}

Multipaths {

Multipath {

Wwid 14945540000000000a67854c6270b4359c66c272e2f356321

Alias iscsi-dm0

Path_grouping_policy multibus

Path_checker tur

Path_selector "round-robin 0"

}

Multipath {

Wwid 14945540000000000dcca2eda91d70b81edbcfce2357f99ee

Alias iscsi-dm1

Path_grouping_policy multibus

Path_checker tur

Path_selector "round-robin 0"

}

Multipath {

Wwid 1494554000000000020f763489c165561101813333957ed96

Alias iscsi-dm2

Path_grouping_policy multibus

Path_checker tur

Path_selector "round-robin 0"

}

Multipath {

Wwid 14945540000000000919ca813020a195422ba3663e1f03cc3

Alias iscsi-dm3

Path_grouping_policy multibus

Path_checker tur

Path_selector "round-robin 0"

}

}

Devices {

Device {

Vendor "iSCSI-Enterprise"

Product "Virtual disk"

Path_grouping_policy multibus

Getuid_callout "/ sbin/scsi_id-g-u-s / block/%n"

Path_checker readsector0

Path_selector "round-robin 0"

}

}

How to get wwid:

(1) by default, each multipath device name is set using the configuration in / var/lib/multipath/bindings. If each wwid alias is set in / etc/multipath.conf, the alias will override this setting.

(2) # multipath-v3 command search

2. Load balancing test

Use the dd command to write to the device, and at the same time use iostat to check the Ihop 0 status. The command and output are as follows:

# dd if=/dev/zero of=/dev/mapper/iscsi-dm1p1

Open another terminal and check the IO situation with the following command

# iostat 10 10

From the above output, we can see that when reading and writing to / dev/mapper/iscsi-dm1p1, we actually complete the writing process to the actual LUN by writing to all the devices of the current active contained in / dev/md-1, that is, / dev/sde1,/dev/shl.

3. Path switching test

First, we unplugged a network cable from the server, and after less than 10 seconds, we saw that MPIO successfully switched from the above "failed" path / dev/sel to another path / dev/sdh2.

This is the answer to the question about the principle and use of multipath multi-path under LINUX. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel for more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.