Questions about Proxmox with shared storage, Redundancy and support for SCSI-3 PR

[nvanaert]

We recently started using Proxmox with shared storage and a few nodes.

These nodes use redundant configuration to the SAN using multipath, but what happens in case both of the paths go down, as from what I understood Proxmox does not have support for SCSI-3 PR (while Linux does). How is this handled by Proxmox? As from what I saw (in a test case scenario) if a host has both of it's connections dropped, the virtual machines just seem to hang indefinitely and die eventually and Proxmox does not seem to reboot the host (I'm using Dell iDRAC hardware based fencing).

How is the storage aware quorum handled (like the VMware datastore heartbeat)?
How is the response handled to automatic storage loss (like VMware VMCP)
How is SCSI-3PR handled (like Red Hat Enterprise Virtualization does using Pacemaker with Stonith Block Device)

Is any of this handled at all? As it doesn't seem like it when testing.

For now I've resolved this using a script with a systemd service and systemd timer which runs every 30 seconds, which checks all paths and if all paths are down increments a counter and when 3 are it it will trigger a echo c /proc/sysrq-trigger which causes a kernel panic, so the node will hard reboot via the IPMI watchdog.

I'd like to know if there's a better / official supported way to configure this though, as I couldn't find anything concrete on this while searching online.

 

[bbgeek17]

Hi @nvanaert ,

There's quite a bit to unpack here.

First and foremost, Proxmox VE does not monitor storage or network path health. An All Paths Down (APD) condition will not result in a node fence, nor does it interact with the VM HA subsystem in any way. To directly answer your VMware comparisons, there is no equivalent to VMware's datastore heartbeat or VMCP (VM Component Protection) in Proxmox. These simply don't exist in the platform today.

Proxmox's HA is designed to handle node-level failures. The approach is: if a node is dead, fence it and restart VMs elsewhere. If a node is alive but storage is unhealthy, that's outside HA's scope.

Second, PVE does not use Quorum Disk in its HA mechanism. It can, however, use network based QDevice. As such, there is no support to using SCSI3 PR as part of the HA heartbeat. The Virtual Disk ownership is controlled by higher level (application) lock administered by PVE.

Third, your systemd timer approach is a good step. It is similar to what some have reported implementing in the forum, but it has a critical flaw: if you experience a symmetric APD where all nodes lose storage simultaneously, every node would kernel panic and reboot. That's almost certainly not what you want. To make this approach viable, your script needs to verify that other nodes still have healthy storage paths before triggering a self-fence. In other words, you need to confirm the problem is local before taking destructive action, otherwise you're rebooting the entire cluster for no benefit.

Unfortunately, there is no official or built-in way to handle this in Proxmox today. Your scripted approach is a reasonable workaround given the gap, but I'd recommend adding that "check the other side" logic before the fence trigger.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[nvanaert]

Hi @nvanaert ,

There's quite a bit to unpack here.

First and foremost, Proxmox VE does not monitor storage or network path health. An All Paths Down (APD) condition will not result in a node fence, nor does it interact with the VM HA subsystem in any way. To directly answer your VMware comparisons, there is no equivalent to VMware's datastore heartbeat or VMCP (VM Component Protection) in Proxmox. These simply don't exist in the platform today.

Proxmox's HA is designed to handle node-level failures. The approach is: if a node is dead, fence it and restart VMs elsewhere. If a node is alive but storage is unhealthy, that's outside HA's scope.

Second, PVE does not use Quorum Disk in its HA mechanism. It can, however, use network based QDevice. As such, there is no support to using SCSI3 PR as part of the HA heartbeat. The Virtual Disk ownership is controlled by higher level (application) lock administered by PVE.

Third, your systemd timer approach is a good step. It is similar to what some have reported implementing in the forum, but it has a critical flaw: if you experience a symmetric APD where all nodes lose storage simultaneously, every node would kernel panic and reboot. That's almost certainly not what you want. To make this approach viable, your script needs to verify that other nodes still have healthy storage paths before triggering a self-fence. In other words, you need to confirm the problem is local before taking destructive action, otherwise you're rebooting the entire cluster for no benefit.

Unfortunately, there is no official or built-in way to handle this in Proxmox today. Your scripted approach is a reasonable workaround given the gap, but I'd recommend adding that "check the other side" logic before the fence trigger.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Thanks for your reply and for the idea on the script. I'm thinking of using the Dell EMC Storage API to see what information I could get out of there to see what it states then, as that would see all nodes and could be more reliable.

 

[bbgeek17]

Thanks for your reply and for the idea on the script. I'm thinking of using the Dell EMC Storage API to see what information I could get out of there to see what it states then, as that would see all nodes and could be more reliable.

Good luck. It is a hard problem to solve, especially if there are FC switches involved - the SAN view can become skewed.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox