How to recover from HW failure on a 3 nodes cluster - VMs are hidden

[francoisd]

Hi,

How should we recover from a lost node due to HW failure on a 3 nodes cluster ?
I can't see the VMs anymore, and have no direct way to restart them on remaining nodes.



Best guess is that I should remove the failing node from the cluster, but unfortunately cluster view do not really help:


Since the cluster do not seem to handle the situation, you should add a section "Recovery" in the default manual : https://pve/pve-docs/index.html

Surprisingly, the pvecm nodes do not display the node 1 while the web UI does

root@pve2:~# pvecm nodes

Membership information
----------------------
    Nodeid      Votes Name
         2          1 pve2 (local)
         3          1 pve3

We can however delete the hidden pve1

root@pve2:~# pvecm delnode pve1
Could not kill node (error = CS_ERR_NOT_EXIST)
Killing node 1

Despite the removal of the pve1 faulty node :

• The VMs are still hidden, and we don't know how to start them on the remaining nodes
• The pve1 is still displayed in the WebUI Datacenter but not in the Cluster panel

A useful link to remove a node: https://forum.proxmox.com/threads/remove-node-from-cluster.98752/

 

[SteveITS]

What sort of storage were you using for the VMs?

 

[francoisd]

RADOS Block Device (Ceph)

 

[fba]

Hi,
have a look at /etc/pve/nodes/pve1/qemu-server/. There should the config files of the vm which were running on pve1 be stored in. Move them to one of your other nodes directories. As long as there were no local storage or other local resource usage configured, they should be accessible in the GUI and can be started at the new node afterwards.

 

[francoisd]

Thanks for the heads up. It works.
Is there a way to handle the loss of a node from the UI ?

 

[fba]

No, there is no way via GUI, only CLI as described here: https://pve.proxmox.com/wiki/Cluster_Manager#_remove_a_cluster_node
If you delete the folder of the node /etc/pve/nodes/pve1 it should disappear from GUI too.