Help, destroyed my cluster by "rm -rf /etc" while deprovisioning

[adoII]

Today i tried a "new method" of removing a proxmox cluster node out of 4. I thought it was a good idea to issue "rm -rf /boot /etc" to make sure the node never comes back. Well, this was the most stupid idea in the last years.

Unfortunately this deleted most configs in /etc/pve like corosync.conf and vm configs and my cluster now is disfunctional. There are no backups of /etc/pve . Also i cannot login to the proxmox gui anymore.

Do you have an idea how i could recover /etc/pve or recover from the situation ?

 

[ubu]

Ok, what happened is that you deleted /etc which also deleted /etc/pve, which unfortunatly is a clusterwide Filesystem.

Do you by any chance have etckeeper installed on your clusternodes?

Do you have proxmoxbackups of your containers and VMs?

 

[adoII]

Unfortunately i have no etckeeper on this node. I have tons of vmbackups in a proxmox backup server. I think i can restore the vm configs easily, but not the corosync.conf ,storage conf and all the other virtual files. I thought about undeleting in the sqlite database in /var/lib/pve-cluster/config.db . any experience with this ?

 

[adoII]

I managed to recover the cluster while all vms kept running:
1) copy corosync.conf from /etc/corosync/corosync.conf to /etv/pve/corosync.conf
2) recover all vmconfigs from vm backups. i went to the pbs and recovered the newest qemu-server.conf.blob from each vm.
copy them to the /etc/pve/nodes/<server>/qemu-server/ directory for the hosts where the vms are running
3) create new certs with pvecm updatecerts --force
4) systemctl restart pvedaemon pveproxy on all nodes
5) recreate storage.cfg and /etc/pve/priv/storage/<yourpbs>.pwd manually

 

[ubu]

Great

 

[nsc117]

nb