Shulou(Shulou.com)06/01 Report--

This article is about how to understand the multi-path of Redhat5.4 Linux multipath dm-multipath. The editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

1. What is multipath

Ordinary computer mainframe is a hard disk attached to a bus, here is an one-to-one relationship. When it comes to the SAN environment composed of optical fiber, or the IPSAN environment composed of iSCSI, because the host and storage are connected through the optical fiber switch or multiple network cards and IP, then it constitutes a many-to-many relationship. That is, there can be multiple paths from the host to the storage. The IO from the host to the storage can be selected by multiple paths. Each host can go through several different paths to the corresponding storage. If it is used at the same time, how is the Icano traffic allocated? One of the paths is broken, how to deal with it? And from the operating system's point of view, each path will be regarded as an actual physical disk, but in fact it is only a different path to the same physical disk, which brings confusion to the user when using it. Multipath software came into being to solve the above problems.

The main function of multipathing is to work with storage devices to achieve the following functions:

1. Failure switching and recovery

Load balancing of 2.IO traffic

3. Virtualization of disk

Because multipath software needs to be used in conjunction with storage, different vendors provide different versions based on different operating systems. And some manufacturers, software and hardware are not sold together, if you want to use multipath software, you may need to buy license from manufacturers. For example, EMC's multipath software based on linux needs to buy license separately. Fortunately, the kernel of RedHat and Suse 2.6 comes with a free multipath package that can be used for free, and it is also a general-purpose package that can support devices from most storage vendors, even if some are not well-known vendors, they can support and run well by making minor changes to the configuration file.

2. Introduction to multipath under Linux requires the following toolkits:

In Redhat 5, check to see if multipath is installed as follows:

[root@vxfs01 ~] # rpm-qa | grep device-mapper

Device-mapper-1.02.39-1.el5

Device-mapper-multipath-0.4.7-34.el5

Device-mapper-1.02.39-1.el5

Device-mapper-event-1.02.39-1.el5

1. Device-mapper-multipath: multipath-tools. Mainly provides tools such as multipathd and multipath and configuration files such as multipath.conf. These tools create and configure multipath devices (calling device-mapper 's user-space library) through the interface of device mapper's ioctr. The multipath device created will be in / dev/mapper).

2. Device-mapper: it mainly includes two parts: the kernel part and the user part. The kernel part is mainly composed of the device mapper core (dm.ko) and some target driver (md-multipath.ko). The core completes the mapping of the device, while target specifically deals with the iUnix from mappered device according to the mapping relationship and its own characteristics. At the same time, in the core part, an interface is provided, and users can communicate with the kernel part through ioctr to guide the behavior of kernel drivers, such as how to create mappered device, the properties of these divece, and so on. The user space section of linux device mapper mainly includes the package device-mapper. These include dmsetup tools and libraries to help create and configure mappered device. These libraries are mainly abstract and encapsulate the interface to communicate with ioctr to facilitate the creation and configuration of mappered device. These libraries need to be called in multipath-tool 's program.

3. Dm-multipath.ko and dm.ko:dm.ko are device mapper drivers. It is the basis for implementing multipath. Dm-multipath is actually a target driver for dm.

4. Scsi_id: included in the udev package, you can configure the program in multipath.conf to get the serial number of the scsi device. By the serial number, it can be judged that multiple paths correspond to the same device. This is the key to multipath implementation. Scsi_id uses the sg driver to send EVPD page80 or page83 inquery commands to the device to query the identity of the scsi device. However, some devices do not support EVPD's inquery command, so they cannot be used to generate multipath devices. However, scsi_id can be rewritten to virtualize an identifier for devices that cannot provide a scsi device identity, and output to standard output. When the multipath program creates a multipath device, it calls scsi_id to get the scsi id of the device from its standard output. When rewriting, you need to modify the return value of the scsi_id program to 0. Because in the multipath program, the straightforward is checked to determine whether the scsi id has been successfully obtained.

3. Basic configuration process of multipath in Redhat 5:

1. Install and load multipath software packages

# rpm-ivh device-mapper-1.02.39-1.el5.rpm # install the mapping package

# rpm-ivh device-mapper-multipath-0.4.7-34.el5.rpm # install multipath package

# chkconfig-level 2345 multipathd on # set to boot self-boot multipathd

# lsmod | grep dm_multipath # to check whether the installation is normal

If the module does not load successfully, initialize the DM or restart the system with the following commands

-Use the following commands to initialize and start DM for the first time:

# modprobe dm-multipath

# modprobe dm-round-robin

# service multipathd start

# multipath-v2

2. Configure multipath:

The configuration file of Multipath is / etc/multipath.conf. If you want multipath to work properly, you only need the following configuration: (if you need more detailed configuration, please see the introduction later in this article)

# on the default devices.

Blacklist {

Devnode "(ram | raw | loop | fd | md | dm- | sr | sr | scd | st) [0-9] *"

Devnode "^ hd [a murz]"

}

Devices {

Device {

Vendor "EMC"

Path_grouping_policy multibus

Features "1 queue_if_no_path"

Path_checker readsector ()

Failback immediate

}

}

3. Multipath basic operation command

# / etc/init.d/multipathd start # enable mulitipath service

# multipath-F # Delete existing unused paths

# multipath-v2 # format path, detect path, merge path

# multipath-ll # View multipath status

If configured correctly, there will be more devices such as mpath0, mpath2, and so on in the / dev/mapper/ directory. Usually the display is not complete and the system needs to be rebooted.

With the fdisk-l command, you can see the disks created by multipath software, such as / dev/dm- [0-3] in the following figure.

# vxfs01 fdisk-l

Disk / dev/dm-1: 5368 MB, 5368709120 bytes

255 heads, 63 sectors/track, 652 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

[root@vxfs01 ~] #

4. Basic operation of multipath disk

To operate on the disks generated by multipath software, you can directly manipulate the disks in the / dev/mapper/ directory.

It is a good idea to run the pvcreate command before partitioning disks generated by multipathing software:

# pvcreate / dev/mapper/mpath0

# fdisk / dev/mapper/mpath0

When using fdisk to partition and save the disk generated by multipath software, there will be an error, which should be ignored.

After fdisk partitions the disk generated by multipath software, the generated partition is not immediately added to the / dev/ directory. At this point, we need to restart the drive of IPSAN or FCSAN. If you use iscsi-initiator to connect to the restart ISCSI service of IPSAN, you can find the generated partition.

# service iscsi restart

# ls-l / dev/mapper/

For example, mpath0p1 and mpath2p1 in the figure above are the partitions of the multipath disk.

# mkfs.ext3 / dev/mapper/mpath0p1 # format mpath0p1 partitions into ext3 file system

# mount / dev/mapper/mpath0p1 / ipsan/ # Mount mpath0p1 partition

# mkfs.ext3 / dev/mapper/mpath2p1 # format mpath2p1 partitions into ext3 file system

# mount / dev/mapper/mpath2p1 / ipsan/ # Mount mpath2p1 partition

Both have to be formatted, and both paths are hung at a mount point, which may be the case! This still needs to be figured out.

IV. Advanced configuration of multipath

All of the above use the default configuration of multipath to complete the configuration of multipath, such as mapping device name and multipath load balancer. Is there any way to configure multipath according to our own definition? yes, we can.

1. Configuration of multipath.conf file

The next job is to edit the configuration file for / etc/multipath.conf

Multipath.conf mainly includes the configuration of blacklist, multipaths and devices.

Blacklist configuration

Blacklist {

Devnode "^ sda"

}

The Multipaths part configures the configuration of multipaths and devices.

Multipaths {

Multipath {

Wwid * # this value multipath-v3 can be seen

Alias after alias iscsi-dm0 # mapping, you can take whatever you want

Path_grouping_policy multibus # path Group Policy

Path_checker tur # method of determining path status

Path_selector "round-robin 0" # method of selecting that path for the next IO operation

}

}

Devices partial configuration

Devices {

Device {

Vendor "iSCSI-Enterprise" # Vendor name

Product "Virtual disk" # product model

Path_grouping_policy multibus # default path group policy

Getuid_callout "/ sbin/scsi_id-g-u-s / block/%n" # gets the default program for unique device numbers

Default program used by prio_callout "/ sbin/acs_prio_alua% d" # to obtain finite order values

Path_checker readsector0 # method of determining path status

Path_selector "round-robin 0" # method of selecting that path for the next IO operation

Failback immediate # failure recovery mode

No_path_retry queue # the number of times the system attempted to use the failure path before disable queue

Rr_min_io 100 # the number of IO requests in the current user group before switching to another path

}

}

The following is a complete configuration file

Blacklist {

Devnode "^ sda"

}

Defaults {

User_friendly_names no

}

Multipaths {

Multipath {

Wwid 14945540000000000a67854c6270b4359c66c272e2f356321

Alias iscsi-dm0

Path_grouping_policy multibus

Path_checker tur

Path_selector "round-robin 0"

}

Multipath {

Wwid 14945540000000000dcca2eda91d70b81edbcfce2357f99ee

Alias iscsi-dm1

Path_grouping_policy multibus

Path_checker tur

Path_selector "round-robin 0"

}

Multipath {

Wwid 1494554000000000020f763489c165561101813333957ed96

Alias iscsi-dm2

Path_grouping_policy multibus

Path_checker tur

Path_selector "round-robin 0"

}

Multipath {

Wwid 14945540000000000919ca813020a195422ba3663e1f03cc3

Alias iscsi-dm3

Path_grouping_policy multibus

Path_checker tur

Path_selector "round-robin 0"

}

}

Devices {

Device {

Vendor "iSCSI-Enterprise"

Product "Virtual disk"

Path_grouping_policy multibus

Getuid_callout "/ sbin/scsi_id-g-u-s / block/%n"

Path_checker readsector0

Path_selector "round-robin 0"

}

}

How to get wwid:

(1) by default, each multipath device name is set using the configuration in / var/lib/multipath/bindings. If each wwid alias is set in / etc/multipath.conf, the alias will override this setting.

[root@vxfs01 ~] # cat / var/lib/multipath/bindings

# Multipath bindings, Version: 1.0

# NOTE: this file is automatically maintained by the multipath program.

# You should not need to edit this file in normal circumstances.

#

# Format:

# alias wwid

#

Mpath0 36006016051d50e0035744871c912de11

Mpath2 36006016051d50e0034744871c912de11

Mpath3 36006016051d50e0032744871c912de11

Mpath4 36006016051d50e0039744871c912de11

Mpath5 36006016051d50e003a744871c912de11

(2) # multipath-v3 command search

[root@vxfs01 ~] # multipath-v3 | grep 3600

Sdb: uid = 36006016051d50e003a744871c912de11 (callout)

Sdc: uid = 36006016051d50e003a744871c912de11 (callout)

Sdd: uid = 36006016051d50e003a744871c912de11 (callout)

Sde: uid = 36006016051d50e003a744871c912de11 (callout)

36006016051d50e003a744871c912de11 1:0:0:0 sdb 8:16 0 [undef] [ready] DGC,RAI

36006016051d50e003a744871c912de11 1:0:1:0 sdc 8:32 1 [undef] [ready] DGC,RAI

36006016051d50e003a744871c912de11 2:0:0:0 sdd 8:48 1 [undef] [ready] DGC,RAI

36006016051d50e003a744871c912de11 2:0:1:0 sde 8:64 0 [undef] [ready] DGC,RAI

Found matching wwid [36006016051d50e003a744871c912de11] in bindings file.

The above is how to understand the multi-path of Redhat5.4 Linux multipath dm-multipath. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.