The space on the backup server is full

[Mixailvk]

Hi.

The space on the backup server is full, I manually deleted the extra backups, but the storage summary indicates that the disk is 100% used, and on the hosts the added backup server disk is displayed as full. How to clean the backup storage from old copies? The garbage collector says there is not enough space:

ProxmoxBackup Server 3.1-5
2024-04-11T11:25:24+03:00: starting garbage collection on store PBS8
2024-04-11T11:25:24+03:00: Start GC phase1 (mark used chunks)
2024-04-11T11:25:24+03:00: TASK ERROR: update atime failed for chunk/file "/mnt/datastore/PBS8/.chunks/f768/f768537bc46de91b577be9369f3a31943f58acf021076175e2a58eb147ad2778" - ENOSPC: No space left on device

Thanks.

 

[ggoller]

Hi!
what kind of filesystem is this? Is this on a separate disk?
If your datastore is on the root disk you can try to delete some other files so that gc will run (try some in /var/log).

 

[Mixailvk]

Hi!
what kind of filesystem is this? Is this on a separate disk?
If your datastore is on the root disk you can try to delete some other files so that gc will run (try some in /var/log).

yes, on a separate disk.

 

[Dunuin]

There is no good way to free up space without buying more disks or losing some random backups.
Only with a GC space will be freed up and for running a GC you need some free space.
One hacky workaround is to move some hidden chunk files to some other storage and symlinking those files back to the original location on the datastore.

Next time set up a quota and monitoring with notifications so you can't fill it to 100% by accident.

At staff:
Would it be possible to check the datastore size before uploading chunks and letting the backup job fail once 99% of the datastore is filled up?
Or some automatism setting the datastores maintainance mode to read-only once 99% is exceeded?
Or at least some default quota set to 99% when creating a datastore.

 

[Mixailvk]

I freed up a few MB by cleaning the indexes and the GC started. you need to stop all backup jobs for this time.

Thanks

 

[Chris]

At staff:
Would it be possible to check the datastore size before uploading chunks and letting the backup job fail once 99% of the datastore is filled up?
Or some automatism setting the datastores maintainance mode to read-only once 99% is exceeded?
Or at least some default quota set to 99% when creating a datastore.

There was a RFC patch series to do some sanity check (which might be used to check the state also before creating new backup snapshots), see https://lists.proxmox.com/pipermail/pbs-devel/2023-December/007518.html

Please feel free to open a bug report at https://bugzilla.proxmox.com/, I might look into this when time allows.

 

[janssensm]

There already is a bug report available that touches this issue : https://bugzilla.proxmox.com/show_bug.cgi?id=5376.
It was closed but asked to reopen and see if some general solution is possible.