Problem with iscsi after upgrade proxmox

[Melanxolik]

Hi All.

Maybe somebody can help me with my little problem, but I spend two days and couldn't solve this problem.

As can you see my server connected to remote storage over iscsi protocol.

root@cluster-1-3:~# iscsiadm -m session -o show
tcp: [4] 10.0.37.202:3260,1 iqn.2017-02:storage02.ssd03 (non-flash)
tcp: [9] 10.0.38.202:3260,1 iqn.2017-02:storage02.ssd03 (non-flash)
root@cluster-1-3:~#

After that I installed multipath-tools and:

Spoiler: multipath_log

root@cluster-1-3:~# multipath -ll -v3
Feb 14 22:38:00 | libdevmapper version 1.02.93 (2015-01-30)
Feb 14 22:38:00 | DM multipath kernel driver v1.10.0
Feb 14 22:38:00 | loading /lib/multipath/libcheckdirectio.so checker
Feb 14 22:38:00 | loading /lib/multipath/libprioconst.so prioritizer
Feb 14 22:38:00 | sda: udev property ID_SERIAL whitelisted
Feb 14 22:38:00 | sda: not found in pathvec
Feb 14 22:38:00 | sda: mask = 0x25
Feb 14 22:38:00 | sda: dev_t = 8:0
Feb 14 22:38:00 | sda: size = 234441648
Feb 14 22:38:00 | sda: vendor = ATA
Feb 14 22:38:00 | sda: product = SAMSUNG MZ7LM120
Feb 14 22:38:00 | sda: rev = 003Q
Feb 14 22:38:00 | sda: h:b:t:l = 0:0:0:0
Feb 14 22:38:00 | sda: tgt_node_name = ata-1.00
Feb 14 22:38:00 | sda: path state = running

Feb 14 22:38:00 | sda: get_state
Feb 14 22:38:00 | loading /lib/multipath/libcheckreadsector0.so checker
Feb 14 22:38:00 | sda: path checker = readsector0 (config file default)
Feb 14 22:38:00 | sda: checker timeout = 30 ms (internal default)
Feb 14 22:38:00 | sda: state = up
Feb 14 22:38:00 | sdb: udev property ID_SERIAL whitelisted
Feb 14 22:38:00 | sdb: not found in pathvec
Feb 14 22:38:00 | sdb: mask = 0x25
Feb 14 22:38:00 | sdb: dev_t = 8:16
Feb 14 22:38:00 | sdb: size = 234441648
Feb 14 22:38:00 | sdb: vendor = ATA
Feb 14 22:38:00 | sdb: product = SAMSUNG MZ7LM120
Feb 14 22:38:00 | sdb: rev = 003Q
Feb 14 22:38:00 | sdb: h:b:t:l = 1:0:0:0
Feb 14 22:38:00 | sdb: tgt_node_name = ata-2.00
Feb 14 22:38:00 | sdb: path state = running


Feb 14 22:38:00 | sdb: get_state
Feb 14 22:38:00 | sdb: path checker = readsector0 (config file default)
Feb 14 22:38:00 | sdb: checker timeout = 30 ms (internal default)
Feb 14 22:38:00 | sdb: state = up
Feb 14 22:38:00 | sdd: udev property ID_SERIAL whitelisted
Feb 14 22:38:00 | sdd: not found in pathvec
Feb 14 22:38:00 | sdd: mask = 0x25
Feb 14 22:38:00 | sdd: dev_t = 8:48
Feb 14 22:38:00 | sdd: size = 928120832
Feb 14 22:38:00 | sdd: vendor = SCST_BIO
Feb 14 22:38:00 | sdd: product = fJId8ZLOB1NeaqbU
Feb 14 22:38:00 | sdd: rev = 220
Feb 14 22:38:00 | sdd: h:b:t:l = 14:0:0:0
Feb 14 22:38:00 | sdd: path state = running

Feb 14 22:38:00 | sdd: get_state
Feb 14 22:38:00 | sdd: path checker = readsector0 (config file default)
Feb 14 22:38:00 | sdd: checker timeout = 30 ms (internal default)
Feb 14 22:38:00 | sdd: state = up
Feb 14 22:38:00 | sdc: udev property ID_SERIAL whitelisted
Feb 14 22:38:00 | sdc: not found in pathvec
Feb 14 22:38:00 | sdc: mask = 0x25
Feb 14 22:38:00 | sdc: dev_t = 8:32
Feb 14 22:38:00 | sdc: size = 928120832
Feb 14 22:38:00 | sdc: vendor = SCST_BIO
Feb 14 22:38:00 | sdc: product = fJId8ZLOB1NeaqbU
Feb 14 22:38:00 | sdc: rev = 220
Feb 14 22:38:00 | sdc: h:b:t:l = 9:0:0:0
Feb 14 22:38:00 | sdc: path state = running


Feb 14 22:38:00 | sdc: get_state
Feb 14 22:38:00 | sdc: path checker = readsector0 (config file default)
Feb 14 22:38:00 | sdc: checker timeout = 30 ms (internal default)
Feb 14 22:38:00 | sdc: state = up
Feb 14 22:38:00 | loop0: blacklisted, udev property missing
Feb 14 22:38:00 | loop1: blacklisted, udev property missing
Feb 14 22:38:00 | loop2: blacklisted, udev property missing
Feb 14 22:38:00 | loop3: blacklisted, udev property missing
Feb 14 22:38:00 | loop4: blacklisted, udev property missing
Feb 14 22:38:00 | loop5: blacklisted, udev property missing
Feb 14 22:38:00 | loop6: blacklisted, udev property missing
Feb 14 22:38:00 | loop7: blacklisted, udev property missing
Feb 14 22:38:00 | ram0: blacklisted, udev property missing
Feb 14 22:38:00 | ram1: blacklisted, udev property missing
Feb 14 22:38:00 | ram10: blacklisted, udev property missing
Feb 14 22:38:00 | ram11: blacklisted, udev property missing
Feb 14 22:38:00 | ram12: blacklisted, udev property missing
Feb 14 22:38:00 | ram13: blacklisted, udev property missing
Feb 14 22:38:00 | ram14: blacklisted, udev property missing
Feb 14 22:38:00 | ram15: blacklisted, udev property missing
Feb 14 22:38:00 | ram2: blacklisted, udev property missing
Feb 14 22:38:00 | ram3: blacklisted, udev property missing
Feb 14 22:38:00 | ram4: blacklisted, udev property missing
Feb 14 22:38:00 | ram5: blacklisted, udev property missing
Feb 14 22:38:00 | ram6: blacklisted, udev property missing
Feb 14 22:38:00 | ram7: blacklisted, udev property missing
Feb 14 22:38:00 | ram8: blacklisted, udev property missing
Feb 14 22:38:00 | ram9: blacklisted, udev property missing
Feb 14 22:38:00 | zd0: blacklisted, udev property missing
===== paths list =====
uuid hcil dev dev_t pri dm_st chk_st vend/prod/rev dev_st
0:0:0:0 sda 8:0 -1 undef ready ATA,SAMSUNG MZ7LM120 running
1:0:0:0 sdb 8:16 -1 undef ready ATA,SAMSUNG MZ7LM120 running
14:0:0:0 sdd 8:48 -1 undef ready SCST_BIO,fJId8ZLOB1NeaqbU running
9:0:0:0 sdc 8:32 -1 undef ready SCST_BIO,fJId8ZLOB1NeaqbU running
Feb 14 22:38:00 | readsector0 checker refcount 4
Feb 14 22:38:00 | readsector0 checker refcount 3
Feb 14 22:38:00 | readsector0 checker refcount 2
Feb 14 22:38:00 | readsector0 checker refcount 1
Feb 14 22:38:00 | unloading const prioritizer
Feb 14 22:38:00 | unloading readsector0 checker
Feb 14 22:38:00 | unloading directio checker

Drives SDD and SDC arrived from storage over two network cards with different path.

Spoiler: multipath.conf

root@cluster-1-3:~# cat /etc/multipath.conf
defaults {
user_friendly_names yes
polling_interval 2
path_selector "round-robin 0"
path_grouping_policy multibus
path_checker readsector0
uid_attribute ID_SERIAL
rr_min_io 100
failback immediate
no_path_retry queue
}
blacklist {
wwid .*
}
blacklist_exceptions {
wwid "2664a4964385a4c4f"
property "(ID_SCSI_VPD|ID_WWN|ID_SERIAL)"
}
multipaths {
multipath {
wwid "2664a4964385a4c4f"
alias ssd03
}
}

root@cluster-1-3:~#

root@cluster-1-3:~# /lib/udev/scsi_id -g -u -d /dev/sdd
2664a4964385a4c4f
root@cluster-1-3:~#

root@cluster-1-3:~# ll /dev/mapper/
total 0
crw------- 1 root root 10, 236 Feb 14 22:09 control
root@cluster-1-3:~#

I restarted server several times and I don't understand what is wrong with my configuration.

 

[manu]

Well it seems the multipath daemon cannot read the serial of /dev/sdc.
Is the device node properly created ? What is the output of

lsscsi --transport --scsi_id

(you need to install the lsscsi package for that)

Also what do you see in the multipathd log ( journalctl -u multipath-tools ) ?

 

[Melanxolik]

root@cluster-1-3:~# lsscsi --transport --scsi_id
[0:0:0:0]    disk    sata:                           /dev/sda   -
[1:0:0:0]    disk    sata:                           /dev/sdb   -
[9:0:0:0]    disk    iqn.2017-02:storage02.ssd03,t,0x1  /dev/sdc   -
[14:0:0:0]   disk    iqn.2017-02:storage02.ssd03,t,0x1  /dev/sdd   2664a4964385a4c4f

root@cluster-1-3:~# iscsiadm -m session -o show 
tcp: [4] 10.0.37.202:3260,1 iqn.2017-02:storage02.ssd03 (non-flash) 
tcp: [9] 10.0.38.202:3260,1 iqn.2017-02:storage02.ssd03 (non-flash)

Feb 15 15:09:01 cluster-1-3 systemd[1]: Stopping LSB: multipath daemon... 
Feb 15 15:09:01 cluster-1-3 multipath-tools[25453]: Stopping multipath daemon: multipathd. 
Feb 15 15:09:01 cluster-1-3 multipathd[10764]: --------shut down------- 
Feb 15 15:09:01 cluster-1-3 systemd[1]: Starting LSB: multipath daemon... 
Feb 15 15:09:01 cluster-1-3 multipath-tools[25460]: Starting multipath daemon: multipathd. 
Feb 15 15:09:01 cluster-1-3 systemd[1]: Started LSB: multipath daemon. 
Feb 15 15:09:01 cluster-1-3 multipathd[25464]: ssd03: ignoring map 
Feb 15 15:09:01 cluster-1-3 multipathd[25464]: ssd03: ignoring map 
Feb 15 15:09:01 cluster-1-3 multipathd[25464]: path checkers start up 
Feb 15 15:09:01 cluster-1-3 multipathd[25464]: dm-0: remove map (uevent) 
Feb 15 15:09:01 cluster-1-3 multipathd[25464]: dm-0: remove map (uevent) 
Feb 15 15:09:01 cluster-1-3 multipathd[25464]: dm-0: remove map (uevent) 
Feb 15 15:09:01 cluster-1-3 multipathd[25464]: dm-0: remove map (uevent)

it's see strange, I don't know why it has "remove map"

 

[manu]

[9:0:0:0] disk iqn.2017-02:storage02.ssd03,t,0x1 /dev/sdc -

the lack of ID here would explain to me why the multipath daemon leave the device alone
I noticed in my own iscsi testlab here, that it can take time before the my iscsi block devices get an id, but I haven't yet found out what factors influence that

 

[Melanxolik]

Maybe somebody has other idia what is wrong with that?

 

[danypd69]

try to add this to multipath.conf

blacklist_exceptions {
       property "(ID_WWN|SCSI_IDENT_.*|ID_SERIAL)"
       wwid ....
}

it worked for me.

I found it here: https://elkano.org/blog/set-up-multipath-iscsi-targets-on-debian/