[SOLVED] All backup lost after update of pve and pbs

[ralfi31]

Hello,

I am new on Proxmox and was running pve and pbs on 2 separate hardware still a few weeks.
All was going well with pve 6.2 and pbs 1.0.1, even after several updates.

This week-end, I update to pve 6.3.2 and pbs 1.0.5 and reboot both computers
This morning, I entered to the pbs console and see that my datastore usage drops down from about 500Go to only 23Go !!

So I go to Datastore and still see all my backuped VM. I also open a shell and make a "df" which confirms that only 23Go of my pool was used.

I try to download a VM to my PC but have an error message : cant download the file or something like that.

Finally, I delete all VM which still appears in Datastore and relaunch a complete backup.

Is this a known issue or some mistake from me with the update ?

Thanks for help.

Rafael

 

[fabian]

This morning, I entered to the pbs console and see that my datastore usage drops down from about 500Go to only 23Go !!

if there aren't any prune and garbage collection tasks that explain this reduction, then it sounds to me like either before or after the reboot something was not mounted correctly and the backups went to some other part of the disk than you thought..

 

[ralfi31]

I have prune and CG activated but I dont know if it has correct parameters.

 

[janssensm]

Just a hunch, not sure about this, but perhaps I noticed something related.
Could this have anything to do with proxmox-backup-client getting out of beta status for the PVE instance?

Not until I upgraded to pve 6.3 I saw the message about the beta status (subscription repo) disappear.
Also the pbs storage dialog was extended with extra options for encryption key management, so that was the moment I decided to newly generate the encryption key and delete the existing backups I had made and start them from scratch.

The only thing I remember that caught my attention:
After upgrading and after regenerating the key, in dialog storage > PBS > Encryption now I see after Encryption Key: " Active - Fingerprint <my_fingerprint>" while before regenerating I believe it was empty.
I didn't investigate, because the client went out of beta, so my plan was to start over my backups from scratch anyway.

 

[fabian]

I have prune and CG activated but I dont know if it has correct parameters.
View attachment 21622

you can check the task history (Administration -> Tasks) and look for the last few prune and garbage collection tasks, the logs will include if and how much they removed.

 

[fabian]

Just a hunch, not sure about this, but perhaps I noticed something related.
Could this have anything to do with proxmox-backup-client getting out of beta status for the PVE instance?

Not until I upgraded to pve 6.3 I saw the message about the beta status (subscription repo) disappear.
Also the pbs storage dialog was extended with extra options for encryption key management, so that was the moment I decided to newly generate the encryption key and delete the existing backups I had made and start them from scratch.

The only thing I remember that caught my attention:
After upgrading and after regenerating the key, in dialog storage > PBS > Encryption now I see after Encryption Key: " Active - Fingerprint <my_fingerprint>" while before regenerating I believe it was empty.
I didn't investigate, because the client went out of beta, so my plan was to start over my backups from scratch anyway.

no, we of course did not remove any backups as part of the beta -> stable transition the key/storage edit dialogue was revamped, but that alone did also not even change any configuration..

 

[ralfi31]

you are right. Task hystory shows a removed garbage corresponding nearly to all my data.
I suppose that I do not set correct parameters in Prune and CG.

 

[ralfi31]

Hi,

So now that we know that it is the garbage collection that removed my data, what should I do to avoid that ?
The goal is to keep backups, not to remove them.

Moreover, when I go Datastore Content, I always see only the last backup and not the increments. There is something I do not understand.

Regards,
Rafael

 

[che]

Hi,

So now that we know that it is the garbage collection that removed my data, what should I do to avoid that ?
The goal is to keep backups, not to remove them.

Moreover, when I go Datastore Content, I always see only the last backup and not the increments. There is something I do not understand.

Regards,
Rafael

Hi,
the whole point of Prune&GC IS to remove unwanted backups!
So read the docs regarding these features https://pbs.proxmox.com/docs/backup-client.html#pruning-and-removing-backups and check out the prune simulator https://pbs.proxmox.com/docs/prune-simulator/ in order to set it up correctly.
You should see more than the last backup in the Datastore->your-store->Content tab. There should be a tree view showing all the backups for a group (host/VM/CT). If that is not the case, than you probably prune the backup before creating the next one. Try manually performing two consecutive backups and check the tree view.

 

[fiona]

Hi,

Hi,

So now that we know that it is the garbage collection that removed my data, what should I do to avoid that ?
The goal is to keep backups, not to remove them.

Moreover, when I go Datastore Content, I always see only the last backup and not the increments. There is something I do not understand.

Regards,
Rafael

which retention options are configured on the client side, i.e. PVE's storage configuration for the PBS storage? If those are empty, it will default to keeping only the last backup.

Now, when creating a new (PBS) storage via the GUI it should get set to keep all, which in fact means that the server is responsible for the pruning. But maybe the old version didn't set that default yet, when you created the storage. Please ensure that the keep all checkbox is checked on the client side.

 

[ralfi31]

Hello,

I thought that this configuration had to be made on pbs side, not on pve side.

At the time I created the pbs storage on pve side, backup retention was configured with "Keep all backups" checkbox not set, keep-last set to 1 and other keep-xxx not set.

So this could explained the behaviour of my backup retention. I will now set "Keep all backups" and then set the pruning on pbs side. I will see if it works in the next days.

Thank you
Regards,
Rafael

 

[ralfi31]

Hello,
The issue is solved. I confirm that it comes from the (old) initial default value at creation of pbs storage.
Thank you for your help.
Regards,
Rafael

 

[fiona]

Glad to hear that it worked! Please mark the thread as [SOLVED]. This helps others to find solutions more easily.