Multipath to SAS Array

Thanks. If I see correctly, each LUN has a partition table and a single partition, and this single partition is set up as an LVM physical volume. If I set up multipath in this scenario in my test system, I can reproduce the problem that, after a reboot, the multipath device does not appear and multipath -v3 shows errors like the following:
Code: 
15.366065 | libdevmapper: ioctl/libdm-iface.c(1980): device-mapper: reload ioctl on 35566777712345678 (253:6) failed: Device or resource busy

I think the problem might be that since the SAN is connected via FC (and not iSCSI), each LUN is already visible as two block devices in early userspace after boot (when only the initramfs is available). But the multipath device is not set up during early boot, so LVM picks up on one of the block devices (or rather: the PV on top of the partition of one of the block devices) instead. And later, this somehow prevents multipath from setting up the multipath device (hence the error "Device or resource busy").

Can you check whether installing the package multipath-tools-boot helps? By installing multipath-tools-boot, the file /etc/multipath/wwids becomes available in the initramfs. If that file is present and multipath_component_detection is set to 1 (the default), LVM will ignore any (non-multipath) block devices whose WWIDs are listed in /etc/multipath/wwids. This should allow multipath to set up the multipath device later. When that multipath device is available, LVM will pick up on it (or rather, the partition on that multipath device).
 
  • Like
Reactions: Olego
Hi,

thanks for the response, after installation this package (multipath-tools-boot) nothing happened.
The outputs of all commands are the same.
fweber said:
Thanks. If I see correctly, each LUN has a partition table and a single partition, and this single partition is set up as an LVM physical volume. If I set up multipath in this scenario in my test system, I can reproduce the problem that, after a reboot, the multipath device does not appear and multipath -v3 shows errors like the following:
Code: 
15.366065 | libdevmapper: ioctl/libdm-iface.c(1980): device-mapper: reload ioctl on 35566777712345678 (253:6) failed: Device or resource busy

I think the problem might be that since the SAN is connected via FC (and not iSCSI), each LUN is already visible as two block devices in early userspace after boot (when only the initramfs is available). But the multipath device is not set up during early boot, so LVM picks up on one of the block devices (or rather: the PV on top of the partition of one of the block devices) instead. And later, this somehow prevents multipath from setting up the multipath device (hence the error "Device or resource busy").

Can you check whether installing the package multipath-tools-boot helps? By installing multipath-tools-boot, the file /etc/multipath/wwids becomes available in the initramfs. If that file is present and multipath_component_detection is set to 1 (the default), LVM will ignore any (non-multipath) block devices whose WWIDs are listed in /etc/multipath/wwids. This should allow multipath to set up the multipath device later. When that multipath device is available, LVM will pick up on it (or rather, the partition on that multipath device).
Click to expand...
Regards,
p.
 
piotrzu said:
thanks for the response, after installation this package (multipath-tools-boot) nothing happened.
Click to expand...
Can you try to reboot after having installed the package? Forgot to mention that.
 
I see -- it's strange that it doesn't make a difference. Some more questions:
  • Do I understand correctly that PVE was installed on top of Debian?
  • Could you open /var/log/apt/term.log, look for the log entry corresponding to the installation of multipath-tools-boot and provide it here (begins with Log started, ends with Log ended)?
  • Could you post the output of the following commands, to check whether the multipath config was indeed copied to the correct initramfs:
    Code: 
    uname -r
for f in /boot/initrd.img-*; do echo $f:; lsinitramfs $f | grep etc/multipath; done
  • Could you attach the output of the following commands:
    Code: 
    dmsetup deps -o devname
multipath -v3
If an output is too long, you should also be able to attach them as files to your post.
 
fweber said:
I see -- it's strange that it doesn't make a difference. Some more questions:
  • Do I understand correctly that PVE was installed on top of Debian?
  • Could you open /var/log/apt/term.log, look for the log entry corresponding to the installation of multipath-tools-boot and provide it here (begins with Log started, ends with Log ended)?
  • Could you post the output of the following commands, to check whether the multipath config was indeed copied to the correct initramfs:
    Code: 
    uname -r
for f in /boot/initrd.img-*; do echo $f:; lsinitramfs $f | grep etc/multipath; done
  • Could you attach the output of the following commands:
    Code: 
    dmsetup deps -o devname
multipath -v3
If an output is too long, you should also be able to attach them as files to your post.
Click to expand...
  • Yes, i am sure that is installed okay.
  • Sure, here you go:
    Code: 
    Log started: 2024-03-07  11:04:39Selecting previously unselected package multipath-tools-boot.
(Reading database ... 89656 files and directories currently installed.)
Preparing to unpack .../multipath-tools-boot_0.9.4-5~bpo11+1_all.deb ...
Unpacking multipath-tools-boot (0.9.4-5~bpo11+1) ...
Setting up multipath-tools-boot (0.9.4-5~bpo11+1) ...
Processing triggers for initramfs-tools (0.142) ...
update-initramfs: Generating /boot/initrd.img-6.5.13-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
Log ended: 2024-03-07  11:04:49
  • Okay, here
    'uname -r':

    Code: 
    6.5.13-1-pve

    'for f in /boot/initrd.img-*; do echo $f:; lsinitramfs $f | grep etc/multipath; done'

    Code: 
    /boot/initrd.img-6.5.13-1-pve:etc/multipath
etc/multipath.conf
etc/multipath/bindings
etc/multipath/wwids

  • I've cut out a portion of unnecessary lines from this part.
    dmsetup deps -o devname

    Code: 
    FC_STORAGE_00-vm--130--disk--0: 1 dependencies  : (sdj1)
FC_STORAGE_00-vm--130--disk--1: 1 dependencies  : (sdj1)
FC_STORAGE_00-vm--130--disk--2: 1 dependencies  : (sdj1)
FC_STORAGE_01: 2 dependencies : (sdk) (sdw)
FC_STORAGE_01-part1: 1 dependencies : (FC_STORAGE_01)
FC_STORAGE_01-vm--130--disk--0: 1 dependencies  : (FC_STORAGE_01-part1)
FC_STORAGE_01-vm--130--disk--1: 1 dependencies  : (FC_STORAGE_01-part1)
FC_STORAGE_01-vm--130--disk--2: 1 dependencies  : (FC_STORAGE_01-part1)
FC_STORAGE_02-vm--130--disk--0: 1 dependencies  : (sdl1)
FC_STORAGE_02-vm--130--disk--1: 1 dependencies  : (sdl1)
FC_STORAGE_02-vm--130--disk--2: 1 dependencies  : (sdl1)
FC_STORAGE_03: 2 dependencies : (sdm) (sdy)
FC_STORAGE_03-part1: 1 dependencies : (FC_STORAGE_03)
FC_STORAGE_03-vm--128--disk--0: 1 dependencies  : (FC_STORAGE_03-part1)
FC_STORAGE_03-vm--128--disk--1: 1 dependencies  : (FC_STORAGE_03-part1)
FC_STORAGE_03-vm--128--disk--2: 1 dependencies  : (FC_STORAGE_03-part1)
FC_STORAGE_04-vm--128--disk--0: 1 dependencies  : (sdn1)
FC_STORAGE_04-vm--128--disk--1: 1 dependencies  : (sdn1)
FC_STORAGE_04-vm--128--disk--2: 1 dependencies  : (sdn1)
FC_STORAGE_05-vm--128--disk--0: 1 dependencies  : (sdaa1)
FC_STORAGE_05-vm--128--disk--1: 1 dependencies  : (sdaa1)
FC_STORAGE_05-vm--128--disk--2: 1 dependencies  : (sdaa1)
FC_STORAGE_06-vm--128--disk--0: 1 dependencies  : (sdp1)
FC_STORAGE_06-vm--128--disk--1: 1 dependencies  : (sdp1)
FC_STORAGE_06-vm--128--disk--2: 1 dependencies  : (sdp1)
FC_STORAGE_06-vm--142--disk--0: 1 dependencies  : (sdp1)
FC_STORAGE_07: 2 dependencies : (sdq) (sdac)
FC_STORAGE_07-part1: 1 dependencies : (FC_STORAGE_07)
FC_STORAGE_08: 2 dependencies : (sdad) (sdr)
FC_STORAGE_08-part1: 1 dependencies : (FC_STORAGE_08)
FC_STORAGE_09: 2 dependencies : (sds) (sdae)
FC_STORAGE_09-part1: 1 dependencies : (FC_STORAGE_09)
FC_NETAPP-vm--100--disk--0: 1 dependencies  : (FC_NETAPP_04-part1)
FC_NETAPP-vm--1001--disk--0: 1 dependencies : (FC_NETAPP_04-part1)
FC_NETAPP-vm--1003--cloudinit: 1 dependencies : (FC_NETAPP_01-part1)
.
.
.
FC_NETAPP-vm--105--disk--0: 1 dependencies  : (FC_NETAPP_04-part1)
FC_NETAPP-vm--106--disk--0: 1 dependencies  : (FC_NETAPP_04-part1)
FC_NETAPP-vm--107--disk--0: 1 dependencies  : (FC_NETAPP_04-part1)
FC_NETAPP-vm--108--disk--0: 1 dependencies  : (FC_NETAPP_01-part1)
FC_NETAPP-vm--109--disk--0: 1 dependencies  : (FC_NETAPP_01-part1)
FC_NETAPP-vm--110--disk--0: 1 dependencies  : (FC_NETAPP_03-part1)
FC_NETAPP-vm--1102--disk--0: 1 dependencies : (FC_NETAPP_04-part1)
FC_NETAPP-vm--1103--disk--0: 1 dependencies : (FC_NETAPP_03-part1)
FC_NETAPP-vm--1103--disk--1: 3 dependencies : (FC_NETAPP_04-part1) (FC_NETAPP_07-part1) (FC_NETAPP_06-part1)
FC_NETAPP-vm--111--disk--0: 1 dependencies  : (FC_NETAPP_04-part1)
FC_NETAPP-vm--112--disk--0: 1 dependencies  : (FC_NETAPP_04-part1)
FC_NETAPP-vm--113--disk--0: 1 dependencies  : (FC_NETAPP_03-part1)
FC_NETAPP-vm--114--disk--0: 1 dependencies  : (FC_NETAPP_04-part1)
FC_NETAPP-vm--115--disk--0: 1 dependencies  : (FC_NETAPP_04-part1)
FC_NETAPP-vm--116--disk--0: 1 dependencies  : (FC_NETAPP_03-part1)
FC_NETAPP-vm--117--disk--0: 1 dependencies  : (FC_NETAPP_04-part1)
FC_NETAPP-vm--149--disk--1: 1 dependencies  : (FC_NETAPP_03-part1)
FC_NETAPP-vm--150--disk--0: 1 dependencies  : (FC_NETAPP_01-part1)
FC_NETAPP-vm--1500--cloudinit: 1 dependencies : (FC_NETAPP_01-part1)
FC_NETAPP-vm--1500--disk--0: 1 dependencies : (FC_NETAPP_01-part1)
FC_NETAPP-vm--151--disk--0: 1 dependencies  : (FC_NETAPP_01-part1)
FC_NETAPP-vm--151--disk--1: 1 dependencies  : (FC_NETAPP_03-part1)
FC_NETAPP-vm--151--disk--2: 2 dependencies  : (FC_NETAPP_01-part1) (FC_NETAPP_02-part1)
FC_NETAPP-vm--151--disk--3: 1 dependencies  : (FC_NETAPP_05-part1)
FC_NETAPP-vm--152--disk--0: 1 dependencies  : (FC_NETAPP_01-part1)
FC_NETAPP-vm--1521--cloudinit: 1 dependencies : (FC_NETAPP_01-part1)
FC_NETAPP-vm--1521--disk--0: 1 dependencies : (FC_NETAPP_01-part1)
FC_NETAPP-vm--1522--cloudinit: 1 dependencies : (FC_NETAPP_01-part1)
FC_NETAPP-vm--1522--disk--0: 1 dependencies : (FC_NETAPP_01-part1)
FC_NETAPP-vm--1523--cloudinit: 1 dependencies : (FC_NETAPP_01-part1)
FC_NETAPP-vm--1523--disk--0: 1 dependencies : (FC_NETAPP_01-part1)
FC_NETAPP-vm--153--disk--0: 1 dependencies  : (FC_NETAPP_03-part1)
FC_NETAPP-vm--154--disk--0: 1 dependencies  : (FC_NETAPP_01-part1)
FC_NETAPP-vm--1541--cloudinit: 1 dependencies : (FC_NETAPP_01-part1)
FC_NETAPP-vm--1541--disk--0: 2 dependencies : (FC_NETAPP_04-part1) (FC_NETAPP_01-part1)
FC_NETAPP-vm--155--disk--0: 1 dependencies  : (FC_NETAPP_03-part1)
FC_NETAPP-vm--156--disk--0: 1 dependencies  : (FC_NETAPP_01-part1)
FC_NETAPP-vm--156--disk--1: 1 dependencies  : (FC_NETAPP_03-part1)
.
.
.
FC_NETAPP-vm--203--disk--0: 1 dependencies  : (FC_NETAPP_01-part1)
FC_NETAPP-vm--204--disk--0: 1 dependencies  : (FC_NETAPP_01-part1)
FC_NETAPP-vm--205--disk--0: 2 dependencies  : (FC_NETAPP_03-part1) (FC_NETAPP_01-part1)
FC_NETAPP-vm--207--disk--0: 1 dependencies  : (FC_NETAPP_04-part1)
FC_NETAPP-vm--208--disk--0: 1 dependencies  : (FC_NETAPP_01-part1)
FC_NETAPP-vm--209--disk--0: 1 dependencies  : (FC_NETAPP_01-part1)
FC_NETAPP-vm--210--disk--0: 1 dependencies  : (FC_NETAPP_01-part1)
FC_NETAPP-vm--211--disk--0: 1 dependencies  : (FC_NETAPP_03-part1)
FC_NETAPP-vm--212--disk--0: 1 dependencies  : (FC_NETAPP_01-part1)
FC_NETAPP-vm--213--disk--0: 1 dependencies  : (FC_NETAPP_03-part1)
FC_NETAPP-vm--213--disk--1: 1 dependencies  : (FC_NETAPP_04-part1)
FC_NETAPP-vm--214--disk--0: 2 dependencies  : (FC_NETAPP_04-part1) (FC_NETAPP_03-part1)
FC_NETAPP-vm--215--disk--0: 1 dependencies  : (FC_NETAPP_01-part1)
.
.
.
FC_NETAPP-vm--313--disk--4: 1 dependencies  : (FC_NETAPP_01-part1)
FC_NETAPP-vm--316--disk--0: 1 dependencies  : (FC_NETAPP_01-part1)
FC_NETAPP-vm--317--disk--0: 1 dependencies  : (FC_NETAPP_01-part1)
FC_NETAPP-vm--318--disk--0: 1 dependencies  : (FC_NETAPP_01-part1)
FC_NETAPP-vm--319--disk--0: 1 dependencies  : (FC_NETAPP_03-part1)
FC_NETAPP-vm--320--disk--0: 2 dependencies  : (FC_NETAPP_03-part1) (FC_NETAPP_01-part1)
FC_NETAPP-vm--321--disk--0: 1 dependencies  : (FC_NETAPP_03-part1)
FC_NETAPP-vm--322--disk--0: 1 dependencies  : (FC_NETAPP_03-part1)
FC_NETAPP-vm--322--disk--1: 1 dependencies  : (FC_NETAPP_01-part1)
FC_NETAPP-vm--327--disk--0: 1 dependencies  : (FC_NETAPP_01-part1)
FC_NETAPP-vm--329--disk--0: 1 dependencies  : (FC_NETAPP_01-part1)
.
.
.
FC_NETAPP-vm--90007--disk--0: 1 dependencies  : (FC_NETAPP_01-part1)
FC_NETAPP-vm--90008--disk--0: 1 dependencies  : (FC_NETAPP_03-part1)
FC_NETAPP-vm--90009--disk--0: 1 dependencies  : (FC_NETAPP_04-part1)
FC_NETAPP-vm--9001--disk--0: 1 dependencies : (FC_NETAPP_01-part1)
FC_NETAPP-vm--90010--disk--0: 1 dependencies  : (FC_NETAPP_01-part1)
FC_NETAPP-vm--90011--disk--0: 1 dependencies  : (FC_NETAPP_01-part1)
FC_NETAPP-vm--920--disk--0: 1 dependencies  : (FC_NETAPP_01-part1)
FC_NETAPP-vm--981--disk--0: 1 dependencies  : (FC_NETAPP_04-part1)
FC_NETAPP-vm--984--disk--0: 1 dependencies  : (FC_NETAPP_04-part1)
FC_NETAPP_01: 4 dependencies  : (sdaz) (sdak) (sdas) (sdf)
FC_NETAPP_01-part1: 1 dependencies  : (FC_NETAPP_01)
FC_NETAPP_02: 4 dependencies  : (sdar) (sde) (sday) (sdaj)
FC_NETAPP_02-part1: 1 dependencies  : (FC_NETAPP_02)
FC_NETAPP_03: 4 dependencies  : (sdaq) (sdd) (sdax) (sdai)
FC_NETAPP_03-part1: 1 dependencies  : (FC_NETAPP_03)
FC_NETAPP_04: 4 dependencies  : (sdap) (sdc) (sdaw) (sdah)
FC_NETAPP_04-part1: 1 dependencies  : (FC_NETAPP_04)
FC_NETAPP_05: 4 dependencies  : (sdat) (sdg) (sdba) (sdal)
FC_NETAPP_05-part1: 1 dependencies  : (FC_NETAPP_05)
FC_NETAPP_06: 4 dependencies  : (sdbb) (sdam) (sdau) (sdh)
FC_NETAPP_06-part1: 1 dependencies  : (FC_NETAPP_06)
FC_NETAPP_07: 4 dependencies  : (sdav) (sdi) (sdbc) (sdan)
FC_NETAPP_07-part1: 1 dependencies  : (FC_NETAPP_07)
FC_XROOTD_00: 2 dependencies  : (sdaf) (sdt)
FC_XROOTD_00-part1: 1 dependencies  : (FC_XROOTD_00)
FC_XROOTD_01: 2 dependencies  : (sdu) (sdag)
FC_XROOTD_01-part1: 1 dependencies  : (FC_XROOTD_01)
Huawei_Disk_01-vm--118--disk--0: 1 dependencies : (sdao1)
.
.
.
Huawei_Disk_01-vm--90502--disk--0: 1 dependencies : (sdao1)
vgroot-lvroot: 1 dependencies : (md0)

  • And the output from multipath -v i attached to the file.

    Regards,
    p.




 

Attachments

  • multipath-v3_logs.txt
    165.9 KB · Views: 5
Thanks. So the multipath config is part of the initramfs. But according to the dependencies in the dmsetup deps output, for some PVs LVM still picks up on one of the block devices instead of the multipath device, e.g. the FC_STORAGE_00 LVs directly depend on sdj1:
piotrzu said:
FC_STORAGE_00-vm--130--disk--0: 1 dependencies : (sdj1)
FC_STORAGE_00-vm--130--disk--1: 1 dependencies : (sdj1)
FC_STORAGE_00-vm--130--disk--2: 1 dependencies : (sdj1)
Click to expand...

It seems like /etc/multipath/wwids does not contain all wwids of all LUNs listed in /etc/multipath.conf, e.g. the WWID 360080e5000297d3c000007405b57e407 of the FC_STORAGE_00 LUN is not part of the file. For LVM to detect the multipath components, the WWID must be listed in that file. Could you run multipath -a WWID for the missing WWIDs? It adds the WWID to /etc/multipath/wwids. Could you also try removing the find_multipaths "no" setting from the /etc/multipath.conf?

After the changes, please run apt install --reinstall multipath-tools-boot to regenerate the initramfs and reboot.
If it still does not work, please again attach the output of dmsetup deps -o devname and multipath -v3.
 
  • Like
Reactions: gawron737 and piotrzu
fweber said:
Thanks. So the multipath config is part of the initramfs. But according to the dependencies in the dmsetup deps output, for some PVs LVM still picks up on one of the block devices instead of the multipath device, e.g. the FC_STORAGE_00 LVs directly depend on sdj1:


It seems like /etc/multipath/wwids does not contain all wwids of all LUNs listed in /etc/multipath.conf, e.g. the WWID 360080e5000297d3c000007405b57e407 of the FC_STORAGE_00 LUN is not part of the file. For LVM to detect the multipath components, the WWID must be listed in that file. Could you run multipath -a WWID for the missing WWIDs? It adds the WWID to /etc/multipath/wwids. Could you also try removing the find_multipaths "no" setting from the /etc/multipath.conf?

After the changes, please run apt install --reinstall multipath-tools-boot to regenerate the initramfs and reboot.
If it still does not work, please again attach the output of dmsetup deps -o devname and multipath -v3.
Click to expand...
Hello,
thank You very much! It works, after that operation all started.
All connections have appeared and everything is functioning correctly.

Regards,
p.
 
  • Like
Reactions: gawron737 and fweber
fweber said:
Thanks. If I see correctly, each LUN has a partition table and a single partition, and this single partition is set up as an LVM physical volume. If I set up multipath in this scenario in my test system, I can reproduce the problem that, after a reboot, the multipath device does not appear and multipath -v3 shows errors like the following:
Code: 
15.366065 | libdevmapper: ioctl/libdm-iface.c(1980): device-mapper: reload ioctl on 35566777712345678 (253:6) failed: Device or resource busy

I think the problem might be that since the SAN is connected via FC (and not iSCSI), each LUN is already visible as two block devices in early userspace after boot (when only the initramfs is available). But the multipath device is not set up during early boot, so LVM picks up on one of the block devices (or rather: the PV on top of the partition of one of the block devices) instead. And later, this somehow prevents multipath from setting up the multipath device (hence the error "Device or resource busy").

Can you check whether installing the package multipath-tools-boot helps? By installing multipath-tools-boot, the file /etc/multipath/wwids becomes available in the initramfs. If that file is present and multipath_component_detection is set to 1 (the default), LVM will ignore any (non-multipath) block devices whose WWIDs are listed in /etc/multipath/wwids. This should allow multipath to set up the multipath device later. When that multipath device is available, LVM will pick up on it (or rather, the partition on that multipath device).
Click to expand...
In my setup, this solution helped me!
 
  • Like
Reactions: fweber
fweber said:
Thanks. If I see correctly, each LUN has a partition table and a single partition, and this single partition is set up as an LVM physical volume. If I set up multipath in this scenario in my test system, I can reproduce the problem that, after a reboot, the multipath device does not appear and multipath -v3 shows errors like the following:
Code: 
15.366065 | libdevmapper: ioctl/libdm-iface.c(1980): device-mapper: reload ioctl on 35566777712345678 (253:6) failed: Device or resource busy

I think the problem might be that since the SAN is connected via FC (and not iSCSI), each LUN is already visible as two block devices in early userspace after boot (when only the initramfs is available). But the multipath device is not set up during early boot, so LVM picks up on one of the block devices (or rather: the PV on top of the partition of one of the block devices) instead. And later, this somehow prevents multipath from setting up the multipath device (hence the error "Device or resource busy").

Can you check whether installing the package multipath-tools-boot helps? By installing multipath-tools-boot, the file /etc/multipath/wwids becomes available in the initramfs. If that file is present and multipath_component_detection is set to 1 (the default), LVM will ignore any (non-multipath) block devices whose WWIDs are listed in /etc/multipath/wwids. This should allow multipath to set up the multipath device later. When that multipath device is available, LVM will pick up on it (or rather, the partition on that multipath device).
Click to expand...
Thanks for this, multipath-tools-boot fixed my issue, believe it was exactly as you described, binding to soon after boot preventing multipath from grabbing it.
 
  • Like
Reactions: fweber
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back