Shulou(Shulou.com)06/02 Report--

In both Linux and AIX platforms, some storage products use the multipath software that comes with the operating system, including the most common HP and IBM storage products. The multipath software that comes with Linux is called multipath. This article takes the use of IBM N series storage on Linux platform as an example to discuss the use of Linux platform multipath.

1. Make sure the following packages are installed: device-mapper device-mapper-multipath

two。 Edit the configuration file / etc/multipath.conf [root@rac01 ~] # vi / etc/multipath.conf

Defaults {user_friendly_names yes max_fds max queue_without_daemon no flush_on_last_del yes}

# # Blacklist non-SAN devices # sample: # devnode "^ (ram | raw | loop | fd | md | dm- | sr | scd | st) [0-9] *" # devnode "^ hd [amurz]" # devnode "^ ccissicic [0-9] d [0-9] * [p [0-9] *]"

Blacklist {wwid 3600605b002b6b890163d47661e8d4fbe}

# FCP configuration, for a NetApp FAS3xxx / IBM Nxx00 # Vendor is "NETAPP", spaces are important # Product is "LUN", though "*" will do

The annotation configuration under devices {device {vendor "NETAPP" product "LUN" getuid_callout "/ sbin/scsi_id-g-u-s / block/%n" prio_callout "/ sbin/mpath_prio_ontap / dev/%n" features "1 queue_if_no_path" hardware_handler "0" path_grouping_policy group_by_prio failback immediate rr_weight uniform rr_min_io 128 path_checker directio} # is not required. After configuration, you can fix the name of the system device. This is used when the disk device corresponding to the server restart device file changes. # multipaths # {# multipath {# wwid 360a9800065344e6f465a6d5773745558 # alias mocr1 #} # multipath {# wwid 360a9800065344e6e536f6d586d306630 # alias mocr2 #} #}

3. Start the multipathd service and set it to start automatically. Execute the following command: # service multipathd restart # chkconfig-- level 345 multipathd on # chkconfig-- list | grep multipathd

4. Check the device name after multipath aggregation, as well as the link corresponding to the device. [root@rac2 ~] # multipath-ll mpath3 (360a9800065344e6f465a6d5773747545) dm-3 NETAPP,LUN size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw `- +-policy='round-robin 0' prio=1 status=active |-1 sdp 0ll mpath3 0active ready running mpath2 (360a9800065344e6f465a6d5773746878) dm-1 NETAPP LUN size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw `- +-policy='round-robin 0' prio=1 status=active |-1prio=1 status=active 0sdd 0prio=1 status=active 3 sdd 8:48 active ready running`-2VAND 0VOUR 0VOULAR 3 sdn 8JUL208 active ready running mpath0 (360a9800065344e6f465a6d5773745558) dm-0 NETAPP LUN size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw `- +-policy='round-robin 0' prio=1 status=active |-1 sdb 0prio=1 status=active 0prio=1 status=active 1 sdb 8:16 active ready running`-2WR 0RO active ready running `- 2 sdl 8Rd 176active ready running mpath9 (360a9800065344e6e536f6d586d33666e) dm-9 NETAPP LUN size=3.9T features='1 queue_if_no_path' hwhandler='0' wp=rw `- +-policy='round-robin 0' prio=1 status=active |-1 active ready running 010 sdk 8 active ready running 010 sdu 65:64 active ready running mpath8 (360a9800065344e6e536f6d586d32766b) dm-7 NETAPP LUN size=150G features='1 queue_if_no_path' hwhandler='0' wp=rw `- +-policy='round-robin 0' prio=1 status=active |-1 sds 0prio=1 status=active 0sdi 812 active ready running`-2 sds 65:32 active ready running mpath7 (360a9800065344e6e536f6d586d313851) dm-6 NETAPP LUN size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw `- +-policy='round-robin 0' prio=1 status=active |-1 sdg 0sdg 8:96 active ready running`-2 prio=1 status=active 0prio=1 status=active 0prio=1 status=active 0 sdq 65:0 active ready running mpath7 (360a9800065344e6e536f6d586d307464) LUN size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw `- +-policy='round-robin 0' prio=1 status=active |-1prio=1 status=active 0prio=1 status=active 0sde 0sde 8:64 active ready running`-2WH 0RO active ready running `- 2VOR 0R 4 sdo 8lane 224 active ready running mpath6 (360a9800065344e6e536f6d586d306630) dm-2 NETAPP LUN size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw `- +-policy='round-robin 0' prio=1 status=active |-1 sdm 0 sdc 0 sdc 8:32 active ready running`-2 sdm 8 sdc 192 active ready running mpath5 (360a9800065344e6f465a6d577376304f) LUN size=3.9T features='1 queue_if_no_path' hwhandler='0' wp=rw `- +-policy='round-robin 0' prio=1 status=active |-1 sdt 0prio=1 status=active 0sdj 8 sdj 144 active ready running`-2 sdt 65:48 active ready running mpath4 (360a9800065344e6f465a6d5773766c58) dm-5 NETAPP LUN size=150G features='1 queue_if_no_path' hwhandler='0' wp=rw `- +-policy='round-robin 0' prio=1 status=active |-1 sdh 0prio=1 status=active 0prio=1 status=active 7 sdh 8 prio=1 status=active 112 sdh`-2

As can be seen from the above results, each disk device corresponds to two links, which vary depending on the storage that is not stored and different configurations. Each device has two multipath device names, mpath [x] and dm- [x], mpath in the / dev/mpath directory and dm- [x] in the / dev directory. It is generally recommended to use the device name in the / dev/mpath directory, which is more convenient for us.

5. Let's focus on the results of fdisk-l execution. [root@rac2 ~] # fdisk-l

Disk / dev/sda: 145.9 GB, 145999527936 bytes 255heads, 63 sectors/track, 17750 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/sda1 * 1 64 514048 + 83 Linux / dev/sda2 65 51199155 83 Linux / dev/sda3 6439 7743 10482412 + 82 Linux swap / Solaris / dev/sda4 7744 17750 80381227 + 5 Extended / dev/sda5 7744 17750 80381196 83 Linux

Disk / dev/sdb: 2147 MB, 2147483648 bytes 255heads, 63 sectors/track, 261cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/sdb1 1 261 2096451 83 Linux

Disk / dev/sdd: 2147 MB, 2147483648 bytes 255heads, 63 sectors/track, 261cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/sdd1 1 261 2096451 83 Linux

Disk / dev/sdf: 2147 MB, 2147483648 bytes 255heads, 63 sectors/track, 261cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/sdf1 1 261 2096451 83 Linux

Disk / dev/sdc: 2147 MB, 2147483648 bytes 255heads, 63 sectors/track, 261cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/sdc1 1 261 2096451 83 Linux

Disk / dev/sde: 2147 MB, 2147483648 bytes 255heads, 63 sectors/track, 261cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/sde1 1 261 2096451 83 Linux

Disk / dev/sdh: 161.0 GB, 161061273600 bytes 255heads, 63 sectors/track, 19581 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/sdh2 1 19581 157284351 83 Linux

Disk / dev/sdg: 2147 MB, 2147483648 bytes 255heads, 63 sectors/track, 261cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/sdg1 1 261 2096451 83 Linux

Disk / dev/sdi: 161.0 GB, 161061273600 bytes 255heads, 63 sectors/track, 19581 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/sdi1 1 19581 157284351 83 Linux

WARNING: GPT (GUID Partition Table) detected on'/ devdeband sdjacks! The util fdisk doesn't support GPT. Use GNU Parted.

WARNING: The size of this disk is 4.2 TB (4241280204800 bytes). DOS partition table format can not be used on drives for volumes larger than 2.2 TB (2199023255040 bytes). Use parted (1) and GUID partition table format (GPT).

Disk / dev/sdj: 4241.2 GB, 4241280204800 bytes 255heads, 63 sectors/track, 515639 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/sdj1 1 267350 2147483647 + ee EFI GPT

WARNING: GPT (GUID Partition Table) detected on'/ devamp SDK! The util fdisk doesn't support GPT. Use GNU Parted.

WARNING: The size of this disk is 4.2 TB (4241280204800 bytes). DOS partition table format can not be used on drives for volumes larger than 2.2 TB (2199023255040 bytes). Use parted (1) and GUID partition table format (GPT).

Disk / dev/sdk: 4241.2 GB, 4241280204800 bytes 255heads, 63 sectors/track, 515639 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/sdk1 1 267350 2147483647 + ee EFI GPT

Disk / dev/sdl: 2147 MB, 2147483648 bytes 255heads, 63 sectors/track, 261cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/sdl1 1 261 2096451 83 Linux

Disk / dev/sdm: 2147 MB, 2147483648 bytes 255heads, 63 sectors/track, 261cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/sdm1 1 261 2096451 83 Linux

Disk / dev/sdn: 2147 MB, 2147483648 bytes 255heads, 63 sectors/track, 261cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/sdn1 1 261 2096451 83 Linux

Disk / dev/sdp: 2147 MB, 2147483648 bytes 255heads, 63 sectors/track, 261cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/sdp1 1 261 2096451 83 Linux

Disk / dev/sdo: 2147 MB, 2147483648 bytes 255heads, 63 sectors/track, 261cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/sdo1 1 261 2096451 83 Linux

Disk / dev/sdr: 161.0 GB, 161061273600 bytes 255heads, 63 sectors/track, 19581 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/sdr1 1 19581 157284351 83 Linux

Disk / dev/sdq: 2147 MB, 2147483648 bytes 255heads, 63 sectors/track, 261cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/sdq1 1 261 2096451 83 Linux

Disk / dev/sds: 161.0 GB, 161061273600 bytes 255heads, 63 sectors/track, 19581 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/sds1 1 19581 157284351 83 Linux

WARNING: GPT (GUID Partition Table) detected on'/ devamp sdttle! The util fdisk doesn't support GPT. Use GNU Parted.

WARNING: The size of this disk is 4.2 TB (4241280204800 bytes). DOS partition table format can not be used on drives for volumes larger than 2.2 TB (2199023255040 bytes). Use parted (1) and GUID partition table format (GPT).

Disk / dev/sdt: 4241.2 GB, 4241280204800 bytes 255heads, 63 sectors/track, 515639 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/sdt1 1 267350 2147483647 + ee EFI GPT

WARNING: GPT (GUID Partition Table) detected on'/ devram sdug! The util fdisk doesn't support GPT. Use GNU Parted.

WARNING: The size of this disk is 4.2 TB (4241280204800 bytes). DOS partition table format can not be used on drives for volumes larger than 2.2 TB (2199023255040 bytes). Use parted (1) and GUID partition table format (GPT).

Disk / dev/sdu: 4241.2 GB, 4241280204800 bytes 255heads, 63 sectors/track, 515639 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/sdu1 1 267350 2147483647 + ee EFI GPT

The device before this is the device name of / dev/sd [n], and this part of the device is a device without aggregation. According to the two links shown above, then the same LUN should correspond to two sd [n] devices. This part is not the device used and can be ignored.

Disk / dev/dm-0: 2147 MB, 2147483648 bytes 255heads, 63 sectors/track, 261cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/dm-0p1 1 261 2096451 83 Linux

Disk / dev/dm-1: 2147 MB, 2147483648 bytes 255heads, 63 sectors/track, 261cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/dm-1p1 1 261 2096451 83 Linux

Disk / dev/dm-2: 2147 MB, 2147483648 bytes 255heads, 63 sectors/track, 261cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/dm-2p1 1 261 2096451 83 Linux

Disk / dev/dm-3: 2147 MB, 2147483648 bytes 255heads, 63 sectors/track, 261cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/dm-3p1 1 261 2096451 83 Linux

Disk / dev/dm-4: 2147 MB, 2147483648 bytes 255heads, 63 sectors/track, 261cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/dm-4p1 1 261 2096451 83 Linux

Disk / dev/dm-5: 161.0 GB, 161061273600 bytes 255heads, 63 sectors/track, 19581 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/dm-5p1 1 19581 157284351 83 Linux

Disk / dev/dm-6: 2147 MB, 2147483648 bytes 255heads, 63 sectors/track, 261cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/dm-6p1 1 261 2096451 83 Linux

Disk / dev/dm-7: 161.0 GB, 161061273600 bytes 255heads, 63 sectors/track, 19581 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/dm-7p1 1 19581 157284351 83 Linux

WARNING: GPT (GUID Partition Table) detected on'/ devamp dmmer8! The util fdisk doesn't support GPT. Use GNU Parted.

WARNING: The size of this disk is 4.2 TB (4241280204800 bytes). DOS partition table format can not be used on drives for volumes larger than 2.2 TB (2199023255040 bytes). Use parted (1) and GUID partition table format (GPT).

Disk / dev/dm-8: 4241.2 GB, 4241280204800 bytes 255heads, 63 sectors/track, 515639 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/dm-8p1 1 267350 2147483647 + ee EFI GPT

WARNING: GPT (GUID Partition Table) detected on'/ devamp dmmer9! The util fdisk doesn't support GPT. Use GNU Parted.

WARNING: The size of this disk is 4.2 TB (4241280204800 bytes). DOS partition table format can not be used on drives for volumes larger than 2.2 TB (2199023255040 bytes). Use parted (1) and GUID partition table format (GPT).

Disk / dev/dm-9: 4241.2 GB, 4241280204800 bytes 255heads, 63 sectors/track, 515639 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System / dev/dm-9p1 1 267350 2147483647 + ee EFI GPT

These devices are respectively / dev/dm- [n] and their partition / dev/dm- [n] p1. This part is generated after installing multipath multipath software and partitioning for each disk device, but actually there is no dm- [n] p1 device file in the / dev/ directory, you just need to know.

Disk / dev/dm-10: 2146 MB, 2146765824 bytes 255heads, 63 sectors/track, 260 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Disk / dev/dm-10 doesn't contain a valid partition table

Disk / dev/dm-11: 2146 MB, 2146765824 bytes 255heads, 63 sectors/track, 260 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Disk / dev/dm-11 doesn't contain a valid partition table

Disk / dev/dm-12: 2146 MB, 2146765824 bytes 255heads, 63 sectors/track, 260 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Disk / dev/dm-12 doesn't contain a valid partition table

Disk / dev/dm-13: 2146 MB, 2146765824 bytes 255heads, 63 sectors/track, 260 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Disk / dev/dm-13 doesn't contain a valid partition table

Disk / dev/dm-14: 161.0 GB, 161059175424 bytes 255heads, 63 sectors/track, 19580 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Disk / dev/dm-14 doesn't contain a valid partition table

Disk / dev/dm-15: 2146 MB, 2146765824 bytes 255heads, 63 sectors/track, 260 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Disk / dev/dm-15 doesn't contain a valid partition table

Disk / dev/dm-16: 161.0 GB, 161059175424 bytes 255heads, 63 sectors/track, 19580 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Disk / dev/dm-16 doesn't contain a valid partition table

Disk / dev/dm-17: 4241.2 GB, 4241280170496 bytes 255heads, 63 sectors/track, 515639 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Disk / dev/dm-17 doesn't contain a valid partition table

Disk / dev/dm-18: 4241.2 GB, 4241280170496 bytes 255heads, 63 sectors/track, 515639 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Disk / dev/dm-18 doesn't contain a valid partition table

Disk / dev/dm-19: 2146 MB, 2146765824 bytes 255heads, 63 sectors/track, 260 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Disk / dev/dm-19 doesn't contain a valid partition table

The name of this part of the device is / dev/dm- [n]. This part of the device is not a disk device aggregated by multipath multipath software, but a partition device file. You cannot see this part of device information by executing fdisk-l just after partitioning the previous / dev/dm-0,dev/dm-1. Only after reboot can you see these device information and generate the corresponding device files. For example, / dev/dm-10 corresponds to / dev/dm-0p1,/dev/dm-11 corresponds to / dev/dm-1p1, and so on, these device files can be used, but this method is not recommended because these device files correspond to disk devices aggregated by multipath.

The device files in the / dev/mpath directory are discussed below: after the multipath software is installed and configured, the aggregated disk device files will be generated in the / dev/mpath directory, which is actually a link file to / dev/dm- [n]: [root@rac2 mpath] # ls-al total 0 drwxr-xr-x 2 root root 440 Sep 21 02:05. Drwxr-xr-x 13 root root 7060 Sep 21 02:12.. Lrwxrwxrwx 1 root root 7 Sep 21 02:05 mpath0->.. / dm-0 lrwxrwxrwx 1 root root 8 Sep 21 02:05 mpath0p1->.. / dm-19 lrwxrwxrwx 1 root root 7 Sep 21 02:05 mpath2->.. / dm-1 lrwxrwxrwx 1 root root 8 Sep 21 02:05 mpath2p1->.. / dm-10 lrwxrwxrwx 1 root root 7 Sep 21 02:05 mpath3->.. / dm-2 lrwxrwxrwx 1 root root 8 Sep 21 02:05 mpath3p1->.. / dm-17 lrwxrwxrwx 1 root root 7 Sep 21 02:05 mpath4->.. / dm-5 lrwxrwxrwx 1 root root 8 Sep 21 02:05 mpath4p1->.. / dm-12 lrwxrwxrwx 1 root root 7 Sep 21 02:05 mpath5->.. / dm-7 lrwxrwxrwx 1 root root 8 Sep 21 02:05 mpath5p1->.. / dm-14 lrwxrwxrwx 1 root root 7 Sep 21 02:05 mpath6->.. / dm-3 lrwxrwxrwx 1 root root 8 Sep 21 02:05 mpath6p1->.. / dm-18 lrwxrwxrwx 1 root root 7 Sep 21 02:05 mpath7- >.. / dm-4 lrwxrwxrwx 1 root root 8 Sep 21 02:05 mpath7p1->.. / dm-11 lrwxrwxrwx 1 root root 7 Sep 21 02:05 mpath7->.. / dm-6 lrwxrwxrwx 1 root root 8 Sep 21 02:05 mpath7p1->.. / dm-13 lrwxrwxrwx 1 root root 7 Sep 21 02:05 mpath8->.. / dm-9 lrwxrwxrwx 1 root root 8 Sep 21 02:05 mpath8p1->.. / dm-16 lrwxrwxrwx 1 root root 7 Sep 21 02:05 mpath9->.. / dm- 8 lrwxrwxrwx 1 root root 8 Sep 21 02:05 mpath9p1->.. / dm-15

From the output above, we can easily understand the corresponding relationship between disk partition device files and disk device files, so it is more convenient to use files in the / dev/mpath directory than to use the files represented by dm- [n] in both the disk device and the disk partition device files in the / dev/ directory. After that, when we use oracleasm to create an ASM disk, we use fdisk-l / dev/mpath/ path [n] p1 to view disk information, and directly use / dev/mpath/ path [n] p1 as an ASM disk, which is very convenient.

Also need to pay attention to the following points: 1). The device file (/ dev/mpath/mpath [n]) aggregated by multipath multipath software will not change with the restart of the server, but the disk LUN corresponding to / dev/sd [n] may change after restart.

2)。 If a single disk is larger than 2TB, instead of using the fdisk tool (traditional MBR), you should use parted (gpt). The steps to create a gpt partition using the parted tool are: # parted PARTED > select / dev/mpath/mpath0 PARTED > mklabel gpt PARTED > mkpart primary 0100%

3)。 If we use ASM as the storage method for database data files, the size of a single ASM disk cannot exceed 2TB, so smaller partitions should be made when the system is partitioned, for example, the size of / dev/mpath/mpath0 disk is 3.8TB Then you should do the following when partitioning: # parted PARTED > select / dev/mpath/mpath0 PARTED > mklabel gpt PARTED > mkpart primary 050% PARTED > mkpart primary 50% 100%. Restart the server, and you will see mpath0p1 and mpath0p2 device files in the / dev/mpath/ directory. You can use these two device files as ASM disks.

-- end--

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.