emptied /etc/pve/, recovery possible?

G

glaaf

Member
Jan 23, 2021
7
0
6
32
Hi

Last night I obviously hadn't enough sleep, and while I tried to solve a problem I encountered while adding a new host to a "cluster" (consisting of one host until then) I thought it would solve the issue when I just cleaned the config on the new host, so I cleared out /etc/pve. And since that did not help, restarted a few services on the old and new host. So, I f*cked up the cluster and am now very awake and angry at myself.

The VMs (which were started before) are still running, but nothing else is possible of course. I'm fine with extracting the data and moving to a new host, but I'd really love to get back the VM configs. It would be quite some work to recreate them from scratch.

Is there any automatic backup in factory settings of Proxmox? etckeeper, snapshots, whatever?

Regards
 
Last edited:
glaaf said:
Is there any automatic backup in factory settings of Proxmox? etckeeper, snapshots, whatever?
Click to expand...
When a node joins the luster, the current DB would be backed up and the a new DB is filled with the contents of the luster.

glaaf said:
I just clean the config on the new host, so I cleared out /etc/pve.
Click to expand...
Well, gone is gone. :oops:
 
Well, looks like this is final. I'm now trying to collect the file fragments from disk, if this doesn't work I have to rebuild everything from scratch. I think I will set up a new cluster first and try to implant its configuration into the old one, to get access to the VMs again via Spice, so I can shut them down graceful. Thanks anyway.
 
I managed to scratch all relevant files from disk and rebuild my /etc/pve dir. Was quite some work... Everything seems to work again, nonetheless I will "some day" have to reinstall the hosts and cluster from scratch just to be sure.

And while I am totally aware that deleting /etc/pve was my fault and just mine, I nonetheless would like to leave a related feature request:

As it is absolutely deadly for the cluster when something happens to this dir, and one host can influence all the others and bring them down, an automated backup should be factory-default here. Either by installing etckeeper or something else, but IMHO this is needed as service for the "customer". One day there will be someone REALLY thankful for that.
 
glaaf said:
And while I am totally aware that deleting /etc/pve was my fault and just mine, I nonetheless would like to leave a related feature request:
Click to expand...
Since it's a directory, it can be included in any backup routine. :) A good tool is the Proxmox Backup Server, it can also save the file contents on the node itself.
 
I am aware that one CAN create backups. Point is: This directory is THE single point of failure for the whole cluster, and any cluster member can destroy the whole cluster for whatever reason - so whatever touches this directory, has the complete cluster at its balls. This directory should be defended at all costs, from the very beginning -> as a factory default.
 
in case you run into this again - joining the cluster should have created a dump of the backing sqlite db, which could have been simply restored with sqlite (after stopping all pve services).
 
Thank you for this annotation, but the backup does not contain the state of the cluster, but the state of the standalone host before it gets added to the cluster. And as this was a freshly installed machine, the backup contained just that - an empty shell. The system with the VMs already was in a cluster for a long time, just without other cluster members.
 
glaaf said:
Thank you for this annotation, but the backup does not contain the state of the cluster, but the state of the standalone host before it gets added to the cluster. And as this was a freshly installed machine, the backup contained just that - an empty shell. The system with the VMs already was in a cluster for a long time, just without other cluster members.
Click to expand...
that makes sense. sorry for the noise then - let's hope it helps others that find this thread in the future with similar issues :)
 
glaaf said:
I am aware that one CAN create backups. Point is: This directory is THE single point of failure for the whole cluster, and any cluster member can destroy the whole cluster for whatever reason - so whatever touches this directory, has the complete cluster at its balls. This directory should be defended at all costs, from the very beginning -> as a factory default.
Click to expand...

Anyone with access to a cluster member already has your cluster 'by its balls' in many ways. So, don't work on your cluster without enough sleep, make backups and don't delete /etc/pve ...
 
glaaf said:
I managed to scratch all relevant files from disk and rebuild my /etc/pve dir. Was quite some work... Everything seems to work again, nonetheless I will "some day" have to reinstall the hosts and cluster from scratch just to be sure.

And while I am totally aware that deleting /etc/pve was my fault and just mine, I nonetheless would like to leave a related feature request:

As it is absolutely deadly for the cluster when something happens to this dir, and one host can influence all the others and bring them down, an automated backup should be factory-default here. Either by installing etckeeper or something else, but IMHO this is needed as service for the "customer". One day there will be someone REALLY thankful for that.
Click to expand...
Hi, How did you do That?

I have the same problem..!!

Thanks in advance
 
I talked about the OP's problem, not yours. Which means, you'll need to recreate the configs. Especially, the storage.cfg and if you have backups you can get the VM/CT config from these.
 
You must log in or register to reply here.
Top Bottom