[SOLVED] Accidently deleted /etc

[Dumb]

Hi.
I accidently deleted /etc on the second host in a 2 host proxmox cluster (no HA active).

Short description (from my memory):
PVE1 installed, 2 physical disks, ZFS RAID1
Created cluster
PVE2 installed 2 physical disks, ZFS RAID1, and joined to cluster
4 VMs installed, configured and prepared for production use.
Replication with default settings activated
nfs installed and configured on both hosts and backup job created (VMs on PVE1 backed up to nfs on PVE2 and VMs on PVE2 backed up to nfs on PVE1)
Proxmox backup server installed and backup of all VMs done to external site

Migrated all VMs to PVE1
Deleted /etc on PVE2

Can´t login to proxmox WEB GUI
Can ssh from my workstation to PVE1

Both hosts are now shutted down

To wipe and reinstall PVE2 is fine but how to "revert" PVE1 to a single host, no cluster state again or a single host in the broken cluster again?

If we need to wipe and reinstall both hosts will we be able to use backups from nfs or backup server and restore in to the new cluster?

Kind regards \\Mike

 

[narrateourale]

The remaining node is now basically in this scenario https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_separate_node_without_reinstall

Follow the steps on this node until the pvecm delnode command (which would be on a node still in the cluster)

 

[Dumb]

Thank You narrateourale for quick answer!

So this is the only few commands needed to be able to: login to GUI again, create a new cluster and join newly reinstalled PVE2?

First, stop the corosync and pve-cluster services on the node:
systemctl stop pve-cluster
systemctl stop corosync

Start the cluster file system again in local mode:
pmxcfs -l

Delete the corosync configuration files:
rm /etc/pve/corosync.conf
rm -r /etc/corosync/*

You can now start the file system again as a normal service:
killall pmxcfs
systemctl start pve-cluster

I will try that tomorrow and get back with feedback later

 

[Johannes S]

To get a clean state I would suggest to reinstall everything after recovering the vms. So: As soon as everything is running again backup your vms to an external usb drive, a PBS server or whatever backup media you have available. Afterwards do a clean reinstall of the PVE nodes and restore the vms from the backup. You should test the restore before to ensure that the backups are working.

 

[Dumb]

Ok...
Before I did any changes I noticed that "/etc/pve/corosync.conf" did not exist not "/etc/corosync/" either
So I logged in to GUI and now it worked. Seems like the reboot of the PVE1 did someting... (PVE2 is still shutted down)
The cluster is gone in GUI
The local-zfs is gone in GUI

The four VMs are also gone in GUI but Bulk start VMs and Containers=OK
All four VMs are also started and OK due to the startup log at the bottom
qm list: does not show anything
qm status 100: Says nodes/PVE1/qemu-server/100.conf' does not exist (100 to 103 says the same)

How to get back VMs and local-zfs in GUI?

Will actions in "https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_separate_node_without_reinstall" fix that also?

 

[Dumb]

To get a clean state I would suggest to reinstall everything after recovering the vms. So: As soon as everything is running again backup your vms to an external usb drive, a PBS server or whatever backup media you have available. Afterwards do a clean reinstall of the PVE nodes and restore the vms from the backup. You should test the restore before to ensure that the backups are working.

Sorry "Johannes S" I did'nt see Your answer until I posted the reply 2 min ago.
Mayby You are right... the best thing to do is to reinstall everything.
As I said from the beginning.. I do have a PBS set up...
and now I also have verified that restore from it to a temporary set up PVE is OK and the restored VMs are starting up and seems OK.

 

[Johannes S]

Sorry "Johannes S" I did'nt see Your answer until I posted the reply 2 min ago.
Mayby You are right... the best thing to do is to reinstall everything.
As I said from the beginning.. I do have a PBS set up...
and now I also have verified that restore from it to a temporary set up PVE is OK and the restored VMs are starting up and seems OK.

Sorry I missed that you have a running PBS. Then I personally woudn't bother trying to fix everything, maybe I miss something which will bite me back later. I'm also quite lazy and tend to go the easy route. However: If you see this as a learning oppurturnity (it is!) and want to find out whether you could recover from such a situation I won't stop you Might be interesting to see, whether the procedure in the wiki would help in such a case. I still would reinstall everything afterwards, so I'm again in a clean state.

 

[Dumb]

Sorry I missed that you have a running PBS. Then I personally woudn't bother trying to fix everything, maybe I miss something which will bite me back lite. I'm also quite lazy and tend to go the easy route. However: If you see this as a learning oppurturnity (it is!) and want to find out whether you could recover from such a situation I won't stop you Might be interesting to see, whether the procedure in the wiki would help in such a case. I still would reinstall everything afterwards, so I'm again in a clean state.

>>maybe I miss something which will bite me back lite
I 100% agree and now I have reinstalled everyting, restored the 4 VMs from the PBS and everything is up and running as it was before.
Thank You all for trying helping me out.
Kind regards \\Mike

 

[Johannes S]

For the future you should consider to setup a backup of your host. You could use the proxmox-backup-client to safe the backups on PBS or another backup tool of your choice.