Shulou(Shulou.com)06/03 Report--

This article mainly explains "how to solve the problem of Multipath". The explanation in this article is simple and clear, easy to learn and understand. Please follow the ideas of Xiaobian and go deep into it slowly to study and learn "how to solve the problem of Multipath" together.

Today I use Packer to make SUSE images, previously using IDE disk format, this time SCSI, due to the change in disk type, to be on the safe side, I regenerated the initramfs load ahci driver to support loading SATA at boot stage.

BINARY_DEPS="tail head awk ifconfig cut expr route ping nc wget tftp grep" DRACUT_DRIVERS="virtio virtio_net virtio_blk" dracut -f -N \ --install "$BINARY_DEPS" \ --kernel-cmdline "rd.shell rd.driver.pre=ahci" \ --kver "$(uname -r)" \ --add-drivers "$DRACUT_DRIVERS" \ --add lvm \ --mdadmconf \ --lvmconf \ -o "dash plymouth" \ initrd-$(uname -r)

As for why you need to load the lvm module, this is the environment requires that the root file system must do LVM, in fact, I do not recommend the use of cloud environment.

fstab uses all block device UIDs instead of logical volume paths (/dev/sdX), worrying that logical volume paths will change due to disk drives, who knows if they will change from/dev/sda1 to/dev/vda1.

At that time, I thought it must be foolproof to do so. After starting the virtual machine verification, the result was unexpected. The system entered emergency mode. Fortunately, I configured rd.shell in initramfs, otherwise I would not know how to enter debugging.

Enter the root password to enter bash, and find that the reason is that/boot is not mounted, try to manually mount to see what error:

mount -a #reinstall fstab mount volume

Equipment already mounted or mount point busy.

mount point mount point/boot must not be mounted, it must be a device problem, from the error prompt is that the disk device has been mounted, but in any case df, lsblk can not find the mount record.

Tried various lsof, fuser commands and found that no process occupied the device.

There's obviously only one case, and I guess the device is mapped by dm(device mapper).

Use dmsetup to view all dm lists, except LV mapping, there are indeed many dm devices starting with 0QEMU_QEMU_HARDDISK:

0QEMU_QEMU_HARDDISK_bcd82475-9f57-43e6-a-part

lVM is mapped by dm, no problem, but/boot is a physical partition and does not do PV to any VG, so who dm this device?

Use dmsetup info to see the following clue:

dmsetup info 0QEMU_QEMU_HARDDISK_bcd82475-9f57-43e6-a-part Name: 0QEMU_QEMU_HARDDISK_bcd82475-9f57-43e6-a-part State: ACTIVE Read Ahead: 1024 Tables present: LIVE Major, minor: 254,1 Number of targets: 1 UUID: part1-mpath-0QEMU_QEMU_HARDDISK_bcd82475-9f57-43e6-a-part

Other information is not useful, only see UUID contains mpath, guess may be due to multipath.

Use multipath -l command to see sure enough:

size=40G features='0' hwhandler='0' wp=rw policy='service-time 0' prio=0 status=active 2:0:0:0 sda 8:0 active undef running

The entire SDA is mapped by multipath.

Find out the reason and think that the solution is clear. Check the multipathd service and find that it is disabled. It should be turned on because the multipathd service is not started.

Use lsmod to find that multipath kernel modules are indeed loaded, are they pulled up by other dependent modules?

To determine if this is caused by the multipath service, run the following command to ensure that multipath is stopped and to force multipath kernel modules to be added to the blacklist.

systemctl disable multipathd echo "blacklist multipath" >>/etc/modprobe.d/50-blacklist.conf

The/etc/multipath.conf configuration file was modified to explicitly add sda to the blacklist.

blacklist { devnode "^(td|hd|vd)[a-z]" devnode "sda" }

Restart the virtual machine, the result is still the same problem, multipath or strong up.

Could it really be that other services started multipath? To eliminate this possibility,/usr/lib/systemd/system/multipath.service was removed, so no one can start this service.

Restart the virtual machine, the result is still surprising, this multipath or up, enough mysterious.

Even worse, I removed/sbin/multipath and/sbin/multipath, which made it impossible for the service to run.

Restart the virtual machine again, let people suspect that life is multipathd service like to beat the undead cockroach rather die than surrender, still running.

Google found a similar problem with System exit to emergency shell at boot with multipath enabled (SLES12, MPIO)[1], which was solved by disabling multipath services or adding devices to the multipath blacklist, but it didn't work at all.

In the case of a sudden epiphany, is it the problem of initramfs? This multipath has already been done in initramfs mounting rootfs?

Following this line of thought, the only way is to re-mirror initramfs and add the multipath module to the omit blacklist:

dracut -f -N \ ... -o "... multipath" \ ...

After replacing the original initrd-xxx file, the virtual machine was started, and the virtual machine finally started successfully.

In the final analysis, it turns out that the multipath module is loaded in the initramfs boot phase, so it is useless to modify and disable multipath in user mode. Always Keep the initrd in Synchronization with the System Configuration.

This problem has been tossing for more than two hours, although the problem is very simple, spent a lot of time on the OS trace, never thought that initramfs is the root of the problem.

Thank you for reading, the above is "how to solve the problem of Multipath" content, after the study of this article, I believe we have a deeper understanding of how to solve the problem of Multipath, the specific use of the situation still needs to be verified by practice. Here is, Xiaobian will push more articles related to knowledge points for everyone, welcome to pay attention!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.