Datastore Full

[deepcloud]

Hi,

At a customer site the backup datastore became full, we pruned the number of backups from keep 7 to keep 4 and ran prun, we even deleted the backups of 2-3 vm

but still the storage is full at 100%,

Any ideas on how to come out of this situation.

Thanks in advance

 

[deepcloud]

even after deleting backups and reducing the retention from last 7 to 4 and pruning the backups - this remains 0

and zero bytes free any ideas guys

 

[sw-omit]

It probably doesn't have enough room TO prune the data.
I'm trying to find the post again (might edit this message in a bit) but I saw a staff member suggest moving some of the blocks to an external drive, running the prune (ignoring the errors about some blocks not being found), then moving the blocks back and running the prune again.

EDIT: Not staff, but at least 1 source:
https://forum.proxmox.com/threads/how-to-recover-from-100-disk-use.142044/#post-637174

EDIT 2: Found the staff-post:
https://forum.proxmox.com/threads/disk-full-unable-to-run-garbage-collection.81800/#post-371926

 

[VictorSTS]

GC will not remove chunks that have a timestamp under 24h + 5 minutes [1]. You will have to wait at least that time to run GC again so it actually removes expired chunks. I don't recommend setting server datetime to a future time as that will affect the prune and GC behavior during that period (prune will remove snapshots that may had not been removed if time was current, GC will set chunks timestamps to a time in the future) and for a few days once you bring the time back to current (prune will not expire backups and GC will not remove any chunck for some days).

You also have the issue that your datastore uses ZFS, which is copy on write and requires some space to be able to remove data. Thanks @sw-omit for linking my post with a step by step method that may help once the 24h + 5 minutes period has passed.

You really must set quotas on ZFS (or create temp files that you may safely delete to recover some GB) in order to allow ZFS to delete data. Also monitoring free space may had prevented a full filesystem.

[1] https://forum.proxmox.com/threads/garbage-collection.139547/

 

[deepcloud]

Thanks for the tips, in the meanwhile - added some space to the zfs pool by increasing the size of the proxmox backup - datastore disks by resizing them.

and i am able to do the GC now

Thanks a ton again