emptied /etc/pve/, recovery possible?

[glaaf]

Hi

Last night I obviously hadn't enough sleep, and while I tried to solve a problem I encountered while adding a new host to a "cluster" (consisting of one host until then) I thought it would solve the issue when I just cleaned the config on the new host, so I cleared out /etc/pve. And since that did not help, restarted a few services on the old and new host. So, I f*cked up the cluster and am now very awake and angry at myself.

The VMs (which were started before) are still running, but nothing else is possible of course. I'm fine with extracting the data and moving to a new host, but I'd really love to get back the VM configs. It would be quite some work to recreate them from scratch.

Is there any automatic backup in factory settings of Proxmox? etckeeper, snapshots, whatever?

Regards

 

[Alwin Antreich]

Is there any automatic backup in factory settings of Proxmox? etckeeper, snapshots, whatever?

When a node joins the luster, the current DB would be backed up and the a new DB is filled with the contents of the luster.

I just clean the config on the new host, so I cleared out /etc/pve.

Well, gone is gone.

 

[glaaf]

Well, looks like this is final. I'm now trying to collect the file fragments from disk, if this doesn't work I have to rebuild everything from scratch. I think I will set up a new cluster first and try to implant its configuration into the old one, to get access to the VMs again via Spice, so I can shut them down graceful. Thanks anyway.

 

[glaaf]

I managed to scratch all relevant files from disk and rebuild my /etc/pve dir. Was quite some work... Everything seems to work again, nonetheless I will "some day" have to reinstall the hosts and cluster from scratch just to be sure.

And while I am totally aware that deleting /etc/pve was my fault and just mine, I nonetheless would like to leave a related feature request:

As it is absolutely deadly for the cluster when something happens to this dir, and one host can influence all the others and bring them down, an automated backup should be factory-default here. Either by installing etckeeper or something else, but IMHO this is needed as service for the "customer". One day there will be someone REALLY thankful for that.

 

[Alwin Antreich]

And while I am totally aware that deleting /etc/pve was my fault and just mine, I nonetheless would like to leave a related feature request:

Since it's a directory, it can be included in any backup routine. A good tool is the Proxmox Backup Server, it can also save the file contents on the node itself.

 

[glaaf]

I am aware that one CAN create backups. Point is: This directory is THE single point of failure for the whole cluster, and any cluster member can destroy the whole cluster for whatever reason - so whatever touches this directory, has the complete cluster at its balls. This directory should be defended at all costs, from the very beginning -> as a factory default.

 

[fabian]

in case you run into this again - joining the cluster should have created a dump of the backing sqlite db, which could have been simply restored with sqlite (after stopping all pve services).

 

[spirit]

"/var/lib/cluster/config.db.bak"

 

[glaaf]

Thank you for this annotation, but the backup does not contain the state of the cluster, but the state of the standalone host before it gets added to the cluster. And as this was a freshly installed machine, the backup contained just that - an empty shell. The system with the VMs already was in a cluster for a long time, just without other cluster members.

 

[fabian]

Thank you for this annotation, but the backup does not contain the state of the cluster, but the state of the standalone host before it gets added to the cluster. And as this was a freshly installed machine, the backup contained just that - an empty shell. The system with the VMs already was in a cluster for a long time, just without other cluster members.

that makes sense. sorry for the noise then - let's hope it helps others that find this thread in the future with similar issues

 

[William Edwards]

I am aware that one CAN create backups. Point is: This directory is THE single point of failure for the whole cluster, and any cluster member can destroy the whole cluster for whatever reason - so whatever touches this directory, has the complete cluster at its balls. This directory should be defended at all costs, from the very beginning -> as a factory default.

Anyone with access to a cluster member already has your cluster 'by its balls' in many ways. So, don't work on your cluster without enough sleep, make backups and don't delete /etc/pve ...

 

[glaaf]

Sure, it always helps when one can avoid doing mistakes.

 

[mlopezbialet]

I managed to scratch all relevant files from disk and rebuild my /etc/pve dir. Was quite some work... Everything seems to work again, nonetheless I will "some day" have to reinstall the hosts and cluster from scratch just to be sure.

And while I am totally aware that deleting /etc/pve was my fault and just mine, I nonetheless would like to leave a related feature request:

As it is absolutely deadly for the cluster when something happens to this dir, and one host can influence all the others and bring them down, an automated backup should be factory-default here. Either by installing etckeeper or something else, but IMHO this is needed as service for the "customer". One day there will be someone REALLY thankful for that.

Hi, How did you do That?

I have the same problem..!!

Thanks in advance

 

[Alwin Antreich]

I have the same problem..!!

I'd assume, he probably re-created the configuration.

 

[mlopezbialet]

I'd assume, he probably re-created the configuration.

Hi..!!
No, this reply is today in April 2024 ..!!
Help please..!!
Regards

 

[Alwin Antreich]

I talked about the OP's problem, not yours. Which means, you'll need to recreate the configs. Especially, the storage.cfg and if you have backups you can get the VM/CT config from these.