HA error

[clabo]

Hi everyone

Setup: In a 4-node Proxmox cluster, shared storage via iSCSI is connected to 3 of the 4 nodes simultaneously. The fourth node does not have the shared LUNs and acts purely as a compute node, relying solely on Ceph for storage.

In this setup, we simulated an HA event by hard-powering off one node.

The cluster restarted all the VMs. However, one VM—which had its disks on the shared LUNs—was registered on the compute node that lacked access to those LUNs. Consequently, it could not be started. Getting this VM back into a proper state was quite a hassle.

Is this a bug, given that registration occurred on a node that did not meet the necessary requirements?
Or is there an implicit requirement that all cluster members *must* have the identical storage configuration, simply because no validation check is performed?

Greetings
clabo

 

[JohnnieKF]

By default, Proxmox storage configurations are cluster-wide.

The cluster doesn't automatically know that your iSCSI LUNs are physically restricted to specific nodes. Because of this, all four nodes assume they have access to the storage, meaning the Cluster Resource Manager (CRM) has no reason to exclude your compute node.

You need to explicitly restrict the storage configuration to only the nodes that actually have the LUNs attached [0].

Run this command, replacing the nodeX with your actual node names:
pvesm set <storage-id> --nodes node1,node2,node3

[0] https://pve.proxmox.com/pve-docs/chapter-pvesm.html#_common_storage_properties