Proxmox 4 Multipath starting at boot

[adamb]

It seems that multipath fails to start after a fresh boot or reboot.

root@ccsmiscrit2:~# systemctl status multipath-tools.service
● multipath-tools.service - LSB: multipath daemon
Loaded: loaded (/etc/init.d/multipath-tools)
Active: active (exited) since Tue 2015-12-22 07:26:09 EST; 24s ago
Process: 2737 ExecStart=/etc/init.d/multipath-tools start (code=exited, status=0/SUCCESS)

Dec 22 07:26:09 ccsmiscrit2 multipath-tools[2737]: Starting multipath daemon: multipathd.
Dec 22 07:26:09 ccsmiscrit2 multipathd[2741]: mpatha: ignoring map
Dec 22 07:26:09 ccsmiscrit2 multipathd[2741]: failure during configuration

I can simply do the following to get it going.

root@ccsmiscrit2:~# systemctl restart multipath-tools.service

root@ccsmiscrit2:~# systemctl status multipath-tools.service
● multipath-tools.service - LSB: multipath daemon
Loaded: loaded (/etc/init.d/multipath-tools)
Active: active (running) since Tue 2015-12-22 07:27:21 EST; 1s ago
Process: 3067 ExecStop=/etc/init.d/multipath-tools stop (code=exited, status=0/SUCCESS)
Process: 3077 ExecStart=/etc/init.d/multipath-tools start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/multipath-tools.service
└─3082 /sbin/multipathd

Dec 22 07:27:21 ccsmiscrit2 multipath-tools[3077]: Starting multipath daemon: multipathd.
Dec 22 07:27:21 ccsmiscrit2 multipathd[3082]: mpatha: ignoring map
Dec 22 07:27:22 ccsmiscrit2 multipathd[3082]: mpath0: load table [0 16777216000 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:16 10 8:32 10]
Dec 22 07:27:22 ccsmiscrit2 multipathd[3082]: mpath0: event checker started
Dec 22 07:27:22 ccsmiscrit2 multipathd[3082]: path checkers start up
Dec 22 07:27:22 ccsmiscrit2 multipathd[3082]: dm-3: remove map (uevent)
Dec 22 07:27:22 ccsmiscrit2 multipathd[3082]: dm-3: remove map (uevent)

My best guess is multipath is attempting to start before iscsi disks are logged in. Is anyone else running into this issue?

 

[adamb]

This gets more interesting.

If I don't setup LVM on the multipath device, it comes up with no issues after a reboot.

The below output is directly after a reboot.

root@ccsmiscrit1:~# multipath -ll
mpath0 (2e4a4ad53839783766c9ce900d893772d) dm-3 Nimble,Server
size=7.8T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 3:0:0:0 sdc 8:32 active ready running
`- 4:0:0:0 sdb 8:16 active ready running

Once I put LVM on top the issue comes back.

pvcreate /dev/mapper/mpath0
vgcreate MissionCrit /dev/mapper/mpath0

Once that is done, multipath doesn't come up after a reboot. Very odd.

I can make it come up by doing one of the following.
/etc/init.d/multipath-tools restart
multipath -v2

 

[adamb]

I was starting to think this is a LVM filter issue, but changing that doesn't seem to help either.

filter = [ "a|/dev/disk/by-id/dm-uuid-.*-mpath-.*|", "a|/dev/sda.*|", "r|.*|" ]

 

[adamb]

For now my workaround is to add "multipath -v2" to /etc/rc.local, but im not a huge fan of this. Definitely seems like a bug of some sort.

 

[bnags]

Were you able to solve this issue?
We're also having this problem with multipath iSCSI. I could sort-of get it to work by adding dm-multipath and dm_round_robin to /etc/modules

It's a big issue because if a path goes down it never comes back up to active:

• multipathd doesn't load correctly
• path 1 fails, path 2 active
• path 1 returns
• multipathd - lists path1 as faulty even though it's back up
• path 2 fails, path1 pseudo-active
• all paths down forever

I assume that it has something to do with the config file, because this operation times out:

• #multipathd -k
• multipathd> show config
• **TIMEMOUT**

Maybe a systemd initramfs problem ? I'm just guessing.

 

[LnxBil]

What is your configuration? I'm using multipathed (but FC-based) SAN for at least half a decade now.

 

[bnags]

Thanks for the reply, here is the multipath config:

/etc/multipath.conf

defaults {
polling_interval 2
path_grouping_policy "multibus"
path_selector "round-robin 0"
uid_attribute "ID_SERIAL"
failback "immediate"
rr_min_io 1
rr_weight "uniform"
no_path_retry "240"
user_friendly_names "yes"
missing_uev_wait_timeout 30
}
blacklist {
device {
vendor "HP"
product ".*"
}
}
devices {
device {
vendor "LIO-ORG"
product "IBLOCK"
path_grouping_policy "multibus"
path_selector "round-robin 0"
rr_min_io 1
alias_prefix "iscsi"
}​

/etc/multipath/bindings

iscsia 36001405df8faf5100000000000000000
iscsib 36001405ce2e82e500000000000000000
​

/etc/multipath/wwids

# Valid WWIDs:
/36001405df8faf5100000000000000000/
/36001405ce2e82e500000000000000000/
​

 

[LnxBil]

Does multipath -v2 show any errors?

 

[bnags]

FIXED

I think the issue was zfs swap that I did not blacklist /dev/zd0 and that caused something to break. (zd0: failed to get udev uid: Invalid argument)
It's also worth noting that the /etc/multipath.conf config must be in exactly the right order - defaults,blacklist,blacklist_exceptions,devices. I originally listed the blacklist after the devices section, and that threw an error.

For everyone's reference, there's what works:

Network
bond3 LACP
bond3.15 (path1)
bond3.16 (path2)

/etc/lvm/lvm.conf

global_filter = [ "r|/dev/dm.*|", "r|/dev/sd.*|", "r|/dev/zd.*|", "r|/dev/mapper/pve-.*|", "a|/dev/mapper/iscsi.*|" ]​

/etc/multipath.conf

defaults {
polling_interval 2
path_grouping_policy "multibus"
path_selector "round-robin 0"
uid_attribute "ID_SERIAL"
failback "immediate"
rr_min_io 1
rr_weight "uniform"
no_path_retry "240"
user_friendly_names "yes"
missing_uev_wait_timeout 30
}
blacklist {
device {
vendor "HP"
product ".*"
}
devnode "^zd"
}
devices {
device {
vendor "LIO-ORG"
product "IBLOCK"
path_grouping_policy "multibus"
path_selector "round-robin 0"
rr_min_io 1
alias_prefix "iscsi"
}
​

I'm going to run some more extensive tests now, but I think that's got it.

 

[LnxBil]

Glad it worked out. Does the -v2 does the trick or how did you get your result?

For the record:
There is also the option to blacklist everything and enable only what you need.

 

[bnags]

The command "multipath -v2" does load the paths and populate /dev however, the multipathd.service was still not running, and that service is responsible for reconfiguring the multipath map. Restarting the service with "systemctl restart multipathd" would fix it...however the problem came back after a reboot.
In the end I blacklisted the zfs zvols with with 'devnode "^zd" ' and everything worked. So it looks like multipathd tried to probe /dev/zd0 then failed and crashed.

The key to all of this is to get the configuration spot on. To help, I refered to the live config. "#multipathd -k" to get a prompt, then "show config" This will list a huge config file showing all the defaults, formatted the same way as multipath.conf
There's actually quite a good list of devices blacklisted by default, so I just added 1 extra. (see my earlier post for the full config)

 

[LnxBil]

Thanks for sharing!