[SOLVED] Proxmox Cluster Recovery After Primary Node Failure and Reinstallation

[bytegenius]

Hello,

We are facing an issue with our Proxmox Cluster configuration and would appreciate any assistance. In our system, we have three servers: pve-1, pve-2, and pve-3. All servers were configured with a clean installation. We initially created a cluster on pve-1 and successfully added pve-2 and pve-3 to the cluster. This process worked smoothly, and virtual machines are running on all servers.

However, at one point, pve-1 encountered a failure and had to be removed from the system. Since the cluster was created on pve-1, the cluster structure broke down. We now need a solution to properly manage the cluster in this situation.

Our approach was to reinstall pve-1, create a new cluster, and attempt to re-add pve-2 and pve-3 to the cluster. However, this time, we encountered an issue stating that there are virtual machines running on pve-2 and pve-3, preventing them from being added to the cluster.

In such a scenario, what is the correct way to manage a Proxmox Cluster? What are the appropriate steps to ensure the system continues running smoothly and without interruption?

All PVE Hardware Detail

• Dell R740 16SFF
• 2x Intel Gold 6138
• 1TB DDR4 ECC
• 12x Raid10 DS + 2x Hotspare + 2x OS DS Proxmox VE

Thank you.

 

[leesteken]

However, at one point, pve-1 encountered a failure and had to be removed from the system. Since the cluster was created on pve-1, the cluster structure broke down.

This does not make sense, sorry. No single node is special and pve-2 and pve-3 should just keep working as a cluster (but barely, since two-node clusters are problematic without a QDevice).

We now need a solution to properly manage the cluster in this situation.

Find out why the break-down of pve-1 interfered with the other nodes. Did you try to manage the cluster from pve-2 or pve-3?

Our approach was to reinstall pve-1, create a new cluster, and attempt to re-add pve-2 and pve-3 to the cluster. However, this time, we encountered an issue stating that there are virtual machines running on pve-2 and pve-3, preventing them from being added to the cluster.

It's not a good idea to reuse names and IPs when adding new nodes to a cluster. It's also not a good idea to add pve-2 and pve-3 to a second cluster (while they are still a cluster themselves).

In such a scenario, what is the correct way to manage a Proxmox Cluster? What are the appropriate steps to ensure the system continues running smoothly and without interruption?

Manage your cluster from pve-2 or pve-3. Remove pve-1 and add a new node (possible by reinstalling Proxmox cleanly on the same hardware with a new IP and new name).

 

[bytegenius]

Bu mantıklı değil, üzgünüm. Hiçbir tek düğüm özel değildir ve pve-2 ve pve-3 sadece bir küme olarak çalışmaya devam etmelidir (ama zar zor, çünkü iki düğümlü kümeler bir QDevice olmadan sorunludur).

Pve-1'in bozulmasının diğer düğümlere neden müdahale ettiğini öğrenin. Kümeyi pve-2 veya pve-3'ten yönetmeyi denediniz mi?

Bir kümeye yeni düğümler eklerken adları ve IP'leri yeniden kullanmak iyi bir fikir değildir. Ayrıca, pve-2 ve pve-3'ü ikinci bir kümeye eklemek de iyi bir fikir değildir (kendileri hala bir küme iken).

Kümenizi pve-2 veya pve-3'ten yönetin. Pve-1'i kaldırın ve yeni bir düğüm ekleyin (aynı donanımda yeni bir IP ve yeni bir adla Proxmox'u temiz bir şekilde yeniden yükleyerek mümkün).

Hello,

Thank you for your quick response. We realized that in the web interface of pve-2 and pve-3, under the "Datacenter" section, the "join" option is already available. Therefore, pve-1 can be re-added to the cluster with a clean installation without any issues. Your suggestion was indeed spot on.

I suppose I asked this question unnecessarily due to some fatigue and lack of focus.

Thank you again for your help.