[SOLVED] Accidentally deleted /etc/ in one node

[ahmed.samer]

Hello everyone,
I was planning to remove one node from the cluster, and to do so I need to shut down this node and make sure that it will never come back up with the same configurations.

I managed to delete /boot/, but instead of deleting /etc/pve/corosync.conf I deleted /etc/
and this deleted the content of /etc/pve/ from the whole cluster because of the pmxcfs, Checked If i have any backup on /var/lib/pve-cluster/backup/ but couldn't find any.

Since everything is deleted from the /etc/pve

• The pveproxy complained about the certificates
  Fixed by pvecm updatecerts --force
  
  
• Cluster out of sync
  I still have the /etc/corosync/corosync.conf
  
  
• No VM configured on any of my nodes
  I managed to re-create the configurations of the VMs on each node
  
  
• pvecm status and pvecm nodes showing that everything is okay with one warning
  Corosync config '/etc/pve/corosync.conf' does not exist - is this node part of a cluster?
  
  
• No storage
  Manually created /etc/pve/storage.cfg

In the end, My question will be.

Is there any way to assemble this cluster back together?

Regards

 

[ahmed.samer]

I managed to copy corosync configurations
from /etc/corosync/corosync.conf to /etc/pve/corosync.conf
restart corosync and the cluster was able to comunicate together again

 

[ahmed.samer]

Hello everyone,
May I ask what is the optimal solution to backup the cluster configuration to avoid situations like this in the future?

Regards

 

[Moayad]

Hi,

there is a script in GitHub but we don't support it [0]

However, if you are using Proxmox Backup Server you could create a hook-script and put it in your cron job to make a backup every week example, for example, backup for /etc/ directory [1]

proxmox-backup-client backup etc.pxar:/etc --repository 192.168.1.3:PBS

NOTE: make sure to double check your backup and don't blindly restore the backup of /etc, but specifically, restore individual config files or use them as reference.

[0] https://gist.github.com/mrpeardotnet/6bdc4b504f43ce57fa7eaee96d376edf
[1] https://pbs.proxmox.com/docs/backup-client.html