Unable to delete backups, No space left on device

[Pito2317]

Hi, so, i noticed that today my BPS server storage just has filled up, i was trying to do some cleaning tasks, but here are results:

2024-02-08T13:23:34+01:00: starting garbage collection on store HDD1
2024-02-08T13:23:34+01:00: Start GC phase1 (mark used chunks)
2024-02-08T13:23:34+01:00: TASK ERROR: update atime failed for chunk/file "/mnt/datastore/HDD1/.chunks/cb4e/cb4ef34b3aa1a14d58b309b5466bd8759f9d6539ac2af48580eaad823b0e6188" - ENOSPC: No space left on device

I want to add that i have only one storage: HDD1. What should i do now?

 

[elmarconi]

Been there.

https://forum.proxmox.com/threads/garbage-collector-error-due-to-datastore-full.136813/#post-607732

You will temporary have to move some chunk dir's in /mnt/datastore/HDD1/.chunks/ to another disk.

Enable maintanance mode on datastore HDD1
mv /mnt/datastore/HDD1/.chunks/ff* /some/other/disk or safer 2 steps cp -a and rm
then delete some backups from the webinterface
The run the garbage collection
check HDD1 disk space with df
mv /some/other/disk/ff* /mnt/datastore/HDD1/.chunks/ or safer 2 steps cp -a and rm
run Verify Job
disable maintanance mode on datastore HDD1

 

[muekno]

moved a VM to another DISK VM is no more visible are the Full Volume but visible at the Volume It has been moved to,
Still can nor run garbage collection or prune.

 

[kph]

Been there.

https://forum.proxmox.com/threads/garbage-collector-error-due-to-datastore-full.136813/#post-607732

You will temporary have to move some chunk dir's in /mnt/datastore/HDD1/.chunks/ to another disk.

Enable maintanance mode on datastore HDD1
...
run Verify Job
disable maintanance mode on datastore HDD1

With the storage in Maintenace Mode (offline), neither GC nor Verify will run. This is with PBS 4.2.1 on PVE 9.2.3.

I just made sure that none of my backup jobs would run while I performed the house-cleaning.

As expected, there were warnings about missing chunks during the garbage collection. But after restoring them, a second GC completed successfully and a subsequent Verify job passed without problem.

My mistake was that I did not uncheck the Backup flag for each storage disk attached to the VM. (VM -> Hardware -> <disk> -> edit --- rookie mistake)