Multipath iSCSI /dev/mapper device is not created (Proxmox 6)

[DiegodelaFuente]

I'm testing a Proxmox 6 environment and I'm having issues trying to configure iSCSI with multipath.

I have a Dell Rc710 (4 onboard 1 Gb nics for VMs and management). And 2 10Gb nics for iSCSI.
I have connected these two nics to a Lenovo De2000h storage.

I have created the Multipath and if I test it I can see up and runing.

root@Mercurio:~# systemctl status -l multipath-tools.service
â multipathd.service - Device-Mapper Multipath Device Controller
Loaded: loaded (/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2019-08-25 02:36:11 -03; 22s ago
Process: 15173 ExecStartPre=/sbin/modprobe -a scsi_dh_alua scsi_dh_emc scsi_dh_rdac dm-multipath (code=exited, status=0/SUCCESS)
Main PID: 15180 (multipathd)
Status: "up"
Tasks: 7
Memory: 11.1M
CGroup: /system.slice/multipathd.service
ââ15180 /sbin/multipathd -d -s

Aug 25 02:36:11 Mercurio systemd[1]: Starting Device-Mapper Multipath Device Controller...
Aug 25 02:36:11 Mercurio multipathd[15180]: --------start up--------
Aug 25 02:36:11 Mercurio multipathd[15180]: read /etc/multipath.conf
Aug 25 02:36:11 Mercurio multipathd[15180]: path checkers start up
Aug 25 02:36:11 Mercurio systemd[1]: Started Device-Mapper Multipath Device Controller.


/etc/multipath.conf

defaults {
polling_interval 2
path_selector "round-robin 0"
path_grouping_policy multibus
uid_attribute ID_SERIAL
rr_min_io 100
failback immediate
no_path_retry queue
user_friendly_names yes
}

blacklist_exceptions {
wwid "3600a098000fd5a7c000000c05d5e0f25"
}
multipaths {
multipath {
wwid "3600a098000fd5a7c000000c05d5e0f25"
alias mpath0
}


root@Mercurio:~# ls -l /dev/mapper/
total 0
crw------- 1 root root 10, 236 Aug 25 01:49 control
lrwxrwxrwx 1 root root 7 Aug 25 01:49 pve-data -> ../dm-4
lrwxrwxrwx 1 root root 7 Aug 25 01:49 pve-data_tdata -> ../dm-3
lrwxrwxrwx 1 root root 7 Aug 25 01:49 pve-data_tmeta -> ../dm-2
lrwxrwxrwx 1 root root 7 Aug 25 01:49 pve-root -> ../dm-1
lrwxrwxrwx 1 root root 7 Aug 25 01:49 pve-swap -> ../dm-0

* There is no mpath0 device

multipath -ll
does not show something

root@Mercurio:~# multipath -v3
===== paths list =====
uuid hcil dev dev_t pri dm_st chk_st vend/prod
3600a098000fd5a7c000000c05d5e0f25 7:0:0:1 sdd 8:48 50 undef undef LENOVO,DE
3600a098000fd5a7c000000c05d5e0f25 8:0:0:1 sde 8:64 10 undef undef LENOVO,DE
Aug 25 02:51:18 | libdevmapper version 1.02.155 (2018-12-18)
Aug 25 02:51:18 | DM multipath kernel driver v1.13.0
Aug 25 02:51:18 | sdd: udev property ID_WWN whitelisted
Aug 25 02:51:18 | sdd: wwid 3600a098000fd5a7c000000c05d5e0f25 whitelisted
Aug 25 02:51:18 | wwid 3600a098000fd5a7c000000c05d5e0f25 not in wwids file, skipping sdd
Aug 25 02:51:18 | sdd: orphan path, only one path
Aug 25 02:51:18 | alua prioritizer refcount 2
Aug 25 02:51:18 | sde: udev property ID_WWN whitelisted
Aug 25 02:51:18 | sde: wwid 3600a098000fd5a7c000000c05d5e0f25 whitelisted
Aug 25 02:51:18 | wwid 3600a098000fd5a7c000000c05d5e0f25 not in wwids file, skipping sde
Aug 25 02:51:18 | sde: orphan path, only one path
Aug 25 02:51:18 | alua prioritizer refcount 1
Aug 25 02:51:18 | unloading alua prioritizer
Aug 25 02:51:18 | unloading const prioritizer
Aug 25 02:51:18 | unloading rdac checker
Aug 25 02:51:18 | unloading tur checker



root@Mercurio:~# lsscsi
[0:0:0:0] cd/dvd TEAC DVD-ROM DV-28SW R.2A /dev/sr0
[2:0:0:0] disk SanDisk Cruzer Glide 3.0 1.00 /dev/sda
[4:0:0:0] disk iDRAC LCDRIVE 0323 /dev/sdc
[5:0:0:0] cd/dvd iDRAC Virtual CD 0323 /dev/sr1
[5:0:0:1] disk iDRAC Virtual Floppy 0323 /dev/sdb
[7:0:0:0] disk LENOVO DE_Series 0851 -
[7:0:0:1] disk LENOVO DE_Series 0851 /dev/sdd
[7:0:0:7] disk LENOVO Universal Xport 0851 -
[8:0:0:0] disk LENOVO DE_Series 0851 -
[8:0:0:1] disk LENOVO DE_Series 0851 /dev/sde
[8:0:0:7] disk LENOVO Universal Xport 0851 -


root@Mercurio:~# /lib/udev/scsi_id -g -u -d /dev/sdd
3600a098000fd5a7c000000c05d5e0f25

Regards
Diego

 

[Whatever]

Check your multipath.conf file. Seems one more “}” bracket is missing at the end

 

[DiegodelaFuente]

It was a mistake when copy paste the config file to the post.
Her is the cull config file. I'm still have the issue.


defaults {
polling_interval 2
path_selector "round-robin 0"
path_grouping_policy multibus
uid_attribute ID_SERIAL
rr_min_io 100
failback immediate
no_path_retry queue
user_friendly_names yes
}

blacklist_exceptions {
wwid "3600a098000fd5a7c000000c05d5e0f25"
}
multipaths {
multipath {
wwid "3600a098000fd5a7c000000c05d5e0f25"
alias mpath0
}
}

 

[DiegodelaFuente]

I do some search in the web and i find this post somewere.

https://www.mail-archive.com/pve-user@pve.proxmox.com/msg10079.html

Gilberto seems to have the same error. He sais he has no issues installing proxmox 5.x
I will try the older version of proxmox.

 

[DiegodelaFuente]

I have installed proxmox 5.4 and made the same configurations as in proxmox 6

now it seems to be working.
Gilberto must be right.

root@Mercurio:~# multipath -ll
mpath0 (3600a098000fd5a7c000000c05d5e0f25) dm-5 LENOVO,DE_Series
size=13T features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
`- 7:0:0:1 sdd 8:48 active ready running
3600a098000fd58fa000000205d0274da dm-6 LENOVO,Universal Xport
size=20M features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 7:0:0:7 sde 8:64 active ready running

I don't know if the problem is with new linux kernel, or some new libraries, beacuse multipath is more related to linux than Proxmox itself

I will finish the configuration to test multipath with Proxmox 5.4
I will not test Proxmox 6 until i know if is stable to use in production.

 

[LnxBil]

multipath-tools or the upstream version changed a lot and some stuff is different now. All discussed in Debian Bug #932307

This worked for me - default section was changed from Debian Stretch to Debian Buster (PVE5 -> PVE6):

root@pve ~ > cat /etc/multipath.conf
defaults {
    find_multipaths no
}

blacklist {
     devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
     devnode "^hd[a-z][[0-9]*]"
     devnode "^sda[[0-9]*]"
     devnode "^cciss!c[0-9]d[0-9]*"
}

multipaths {
    multipath {
        wwid    1FreeBSD_iSCSI_DISK_10000001
        alias   asm_iscsi
    }
}

root@pve ~ > multipath -ll
asm_iscsi (1FreeBSD_iSCSI_DISK_10000001) dm-5 FreeBSD,iSCSI DISK
size=128G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 3:0:0:0 sdb 8:16 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 4:0:0:0 sdc 8:32 active ready running

 

[DiegodelaFuente]

Thanks for the help.

find_multipaths no was the key.
Now it's working in Proxmox 6.

Regards

Diego

 

[Area TIC]

Thanks for the help.

find_multipaths no was the key.
Now it's working in Proxmox 6.

Regards

Diego

Hi Diego, finally what was the key? I have a similar issue! Thanks for your reply

 

[Area TIC]

Autoreply, in my case i have well configured:

- multipath-tools installed, service active and running
- /etc/multipath.conf well configured for HPE 3PAR
- 4 HBA path detected by rescan-scsi-bus.sh
- restarting multipath-tools.service
- multipath -ll haven't result
- multipath -v3 respond "sdb orphan path, only one path"
- not /dev/mapper mpath0 device

A lot of test and config tested, however the solutions was;
# sudo multipath -a "uuid asigned to multipath"
# sudo systemctl restart multipath-tools.service

Then the mpath0 device was created and mount the HPE 3PAR in the HA Cluster.

 

[stefanobertoli]

Same for me, if removed the path_selector "round-robin 0" and it work also with find_multipaths yes. Seems than the only way to work with round robin is to say no to find_multipaths.

is it normal than, also if in round-robin, the "multipath -ll" command still indicate for one device status=active and for the other status=enabled?

 

[LnxBil]

is it normal than, also if in round-robin, the "multipath -ll" command still indicate for one device status=active and for the other status=enabled?

That depends on the server presenting the luns. If your server is capable of ALUA and it is enabled, those two groups should merge into one.

 

[stefanobertoli]

I'm trying to figure out how to proper activate and configure in IBM Storewize v7000. It's ALUA compliant, let see...