Sudden death to cluster?

[finlaydag33k]

Hii there,
Today, we had a little bit of maintenance on the power grid, so I turned off my small 3 node proxmox cluster.
after the maintenance was done, I booted up the cluster, and poof, the cluster was completely dead.
the cluster is still there (in some sort).

each of the nodes say the other nodes are offline, but for some kind of reason that is unknown to me, all of the nodes are able to work independently (although I can't change any configs, or make/move VMs).
and I get connection errors thrown at my face all the time, but atleast the core VMs that I need are alive...

can anyone guide me trough what to do next?
I think (even though I am a hobbyist with almost no clue what he is doing) that I have to resync the cluster keys.. but I don't know how.
anyone who has suggestions?

Cheers

 

[Christophe Casalegno]

Hi, can you try to ping all the other nodes from each nodes ? Is it working fine ?

 

[udo]

Hii there,
Today, we had a little bit of maintenance on the power grid, so I turned off my small 3 node proxmox cluster.
after the maintenance was done, I booted up the cluster, and poof, the cluster was completely dead.
the cluster is still there (in some sort).

each of the nodes say the other nodes are offline, but for some kind of reason that is unknown to me, all of the nodes are able to work independently (although I can't change any configs, or make/move VMs).
and I get connection errors thrown at my face all the time, but atleast the core VMs that I need are alive...

can anyone guide me trough what to do next?
I think (even though I am a hobbyist with almost no clue what he is doing) that I have to resync the cluster keys.. but I don't know how.
anyone who has suggestions?

Cheers

Hi,
what is the output of

pvecm nodes
pvecm status

what happens if you restart pve-cluster? (if you have quorum).

service pve-cluster restart

on each node (one after one).

Do you have do any changes in hostname, networkconfig or in /etc/host before shutdown/restart?

Udo

 

[finlaydag33k]

Hi,
what is the output of

pvecm nodes
pvecm status

what happens if you restart pve-cluster? (if you have quorum).

service pve-cluster restart

on each node (one after one).

Do you have do any changes in hostname, networkconfig or in /etc/host before shutdown/restart?

Udo

restarting the pve-cluster service worked. quite odd that it didnt do that when I rebooted the server.