iSCSI Portal single Point of Failure

[felipe]

HI,

we have a Netapp E Series which does not support LACP. HA is done over Multipath. This works as expected (tested deactivating one nic for example)
The problem is that proxmox is querying allways the ip of the portal IP to get the status of the iscasi target(s).
When i deactivate the network card with the ip which i use to configure the iscasi portal in the gui then i cannot see the lun or the lvn on top of the lun. As the cli command for getting all storages also times out i think after a while as usual when some storages are offline all items will get grey because some proxmox tasks hang.
The storage itself is still available and the vm works. But the proxmox GUI for the storage not ( and in the worst case all hosts will get grey)

Is there any solution to get HA for the Portal? (Fallback ip ?)

 

[felipe]

proxmox team any idea?

 

[felipe]

if i deactivate the switch where the iscasci port of the SAN is connected which ip i used for configuring via proxmox gui the storage shows as offline.
so i cannot start any vm?
noby ever tried this? if you have multipath normally you don't use lacp und then if the iscsi interface of the SAN is not available for the ip defined in proxmox the storage is offline. (but still runnung vms can access it via multipath)
but it is impossible to start or migrate machines. this is very bad.

no idea how to fix this?

 

[felipe]

see below iscsid looks bad but multiptah looks good:

Feb 18 12:18:14 node3 kernel: [7940340.269119] connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx 6279954892, last ping 6279956144, now 6279957440
Feb 18 12:18:14 node3 kernel: [7940340.270145] connection2:0: detected conn error (1022)
Feb 18 12:18:15 node3 iscsid: Kernel reported iSCSI connection 2:0 error (1022 - Invalid or unknown error code) state (3)
Feb 18 12:18:19 node3 kernel: [7940345.385151] session2: session recovery timed out after 5 secs
Feb 18 12:18:20 node3 multipathd[6458]: mpathc: sdc - rdac checker reports path is down: inquiry failed
Feb 18 12:18:20 node3 kernel: [7940345.409155] sd 1:0:0:0: rejecting I/O to offline device
Feb 18 12:18:20 node3 kernel: [7940345.409621] blk_update_request: I/O error, dev sdc, sector 7052440 op 0x1WRITE) flags 0xca00 phys_seg 1 prio class 0
Feb 18 12:18:20 node3 kernel: [7940345.409985] sd 1:0:0:0: rejecting I/O to offline device
Feb 18 12:18:20 node3 kernel: [7940345.410569] device-mapper: multipath: Failing path 8:32.
Feb 18 12:18:20 node3 kernel: [7940345.410712] device-mapper: multipath: Failing path 8:112.
Feb 18 12:18:20 node3 multipathd[6458]: checker failed path 8:32 in map mpathc
Feb 18 12:18:20 node3 multipathd[6458]: mpathc: remaining active paths: 3
Feb 18 12:18:20 node3 multipathd[6458]: checker failed path 8:112 in map mpathd
Feb 18 12:18:20 node3 multipathd[6458]: mpathd: remaining active paths: 3

----

Feb 18 12:31:05 node3 iscsid: connect to 192.168.10.231:3260 failed (No route to host)
Feb 18 12:31:07 node3 pvestatd[28175]: storage 'lenovotest' is not online

 

[felipe]

iscsi admin shoes me:

tcp: [2] 192.168.10.231:3260,1 iqn.2002-09.com.lenovo:thinksystem.600a098000e6a260000000005c59d825 (non-flash)
tcp: [3] 192.168.10.232:3260,1 iqn.2002-09.com.lenovo:thinksystem.600a098000e6a260000000005c59d825 (non-flash)
tcp: [4] 192.168.10.234:3260,2 iqn.2002-09.com.lenovo:thinksystem.600a098000e6a260000000005c59d825 (non-flash)
tcp: [5] 192.168.10.233:3260,2 iqn.2002-09.com.lenovo:thinksystem.600a098000e6a260000000005c59d825 (non-flash)

 

[felipe]

nobody any idea?

 

[Stoiko Ivanov]

Multipath is currently not directly included and taken into consideration by the PVE stack (you can configure an iSCSI storage as single path, and an LVM on top, and then configure the multipath.conf according to your SAN's recommendations).

AFAICT the current design idea and goal is that the guests keep running if the primary path is down (which they do), but that it's ok to not have the ability to reconfigure your storage in time the primary path to it is not available.

Things you could try (don't have similar setup to try it out currently):
* simply comment out the iscsi-part of the storage.cfg (openiscsid should login to the sessions and provide the disks + multipath should pick them up - and as long as the LVM is available you should be able to do administrative tasks)

the problem with all storages showing a grey question mark is known (it's a design limitation of the current pvestatd implementation - and we have it on our roadmap to adapt that to be more resilient)

I hope this helps a bit!

 

[felipe]

Thank you very much for the answer. So it is like i suspected. We use CEPH which is really nice integrated into pve- but allways we want to have 2 HA storages in case 1 for whatever reasons badly fails or is not available for some days. SCSCI/FC only nettaps for example cost the half price of netapp filers which support nfs. In VMware iscsi is very good integrated and it hurts a lot that now we will need to go for much more expensive filers instead of block storage netapps.
Do you plan to integrate it in the future with multipath?

 

[Stoiko Ivanov]

Hm? why do you _need_ to go with NFS?
As said - my suggestions were:
* try not configuring the iscsi via the PVE-stack (or comment it from your /etc/pve/storage.cfg, after you're done setting up everything) - that should prevent the hanging pvestatd (with associated problems) when the primary path is down.
* in quite a few scenarios it is ok - not to be able to perform configuration tasks if one of the mulitpath interfaces is down

as for the change to a different SAN/NAS with NFS support - if an NFS mount hangs you get at least as many problems as a hanging path of your multipath config

also I'm a bit confused:
the iscsiadm output seems like it's from a lenovo system and not a netapp?

 

[felipe]

>>also I'm a bit confused:

>>the iscsiadm output seems like it's from a lenovo system and not a netapp?
lenovo is just reselling netapps. its netapp hardware.

>> in quite a few scenarios it is ok - not to be able to perform configuration tasks if one of the mulitpath interfaces is down

maybe. but in case of switch failure it can happen some time to bring up the interface. of course there are workarounds. but i like things straight forward. if other sysadmins in our company have to work with it they have no idea whats wrong or what to do.

>> if an NFS mount hangs you get at least as many problems as a hanging path of your multipath config

thats right. but thats in our case allready a disaster worstcase because all storages are HA. so storage (or all switches are gone). with multiptah just the wrong switch or port needs to hang.

i know it is a special case. but with hundrets of vms its not funny to have no redundancy for maintainance of the vms (because i event cant start any vm on that storage)

 

[Stoiko Ivanov]

did you try simply commenting out the iscsi-definition in the storage.cfg (after you've set everything up)?

 

[felipe]

Did anything change in PVE 7? Or any plans to multipath for the storage config? IN Hyper-V and VMware its possible. As this is an Proxmox issue and not Linux/KVM it should be possible to fix it.