Disk full - unable to run garbage collection

[dff]

Backup Server 1.0-6

Noticed the disk was full. Tried to run garbage collection from the web ui .
unable to start garbage collection job on datastore backups - ENOSPC: No space left on device

Manually deleted snapshots in the web ui and from the command line and rebooted - problem persists.

Ran proxmox-backup-client garbage-collect --repository backups
unable to start garbage collection job on datastore backups - ENOSPC: No space left on device

Ran proxmox-backup-manager garbage-collection start backups
unable to start garbage collection job on datastore backups - ENOSPC: No space left on device

Im happy to provide logs if you tell me which ones are helpful. Thanks in advance

Filesystem      Size  Used Avail Use% Mounted on
udev            2.9G     0  2.9G   0% /dev
tmpfs           595M   16M  580M   3% /run
/dev/sda2       490G  465G     0 100% /
tmpfs           3.0G     0  3.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.0G     0  3.0G   0% /sys/fs/cgroup
/dev/sda1       511M  6.6M  505M   2% /boot/efi
tmpfs           595M     0  595M   0% /run/user/1001

 

[Stefan_R]

This behaviour is certainly not ideal, but currently you need at least *some* free space to run most PBS operations. Try deleting some unnecessary data and try again (e.g. some old (gzipped) log files from /var/log are often a good candidate).

 

[dff]

This behaviour is certainly not ideal, but currently you need at least *some* free space to run most PBS operations. Try deleting some unnecessary data and try again (e.g. some old (gzipped) log files from /var/log are often a good candidate).

Thank you
Removing old log files did the trick. Garbage collection could run again.

 

[AST]

Is this a test machine?
If no, then give the poor thing a respectable hard drive
In addition, I recommend to install an SNMP service on PVE and therefore also on PBS to collect the values from a monitoring system (e.g. PRTG).

 

[robson_ek]

i have the same situation but in ZFS backup drive

HDD                   1.8T  1.8T  3.0M 100% /mnt/datastore/HDD

# proxmox-backup-client garbage-collect --repository HDD
starting garbage collection on store HDD
Start GC phase1 (mark used chunks)
TASK ERROR: update atime failed for chunk/file "/mnt/datastore/HDD/.chunks/bb9f/bb9f8df61474d25e71fa00722318cd387396ca1736605e1248821cc0de3d3af8" - ENOSPC: No space left on device
# proxmox-backup-manager garbage-collection start HDD
starting garbage collection on store HDD
Start GC phase1 (mark used chunks)
TASK ERROR: update atime failed for chunk/file "/mnt/datastore/HDD/.chunks/a1ba/a1ba55b9c10ebc59ceef92b920f443dd542121929e55a7bd85c5160b73d3a859" - ENOSPC: No space left on device

Can I delete some chunks?

 

[Stefan_R]

Can I delete some chunks?

If you want to lose your data, sure Otherwise I'd rather recommend deleting *unused* files (e.g. /var/log/ is a good place to start) first, and then try the GC again. Just a few kilobytes of free space should already be enough to start the operation, and it will then clear up more space as it goes.

 

[robson_ek]

ok, I try, but on main partition I have a lot of free space

Filesystem Size Used Avail Use% Mounted on
udev 2.9G 0 2.9G 0% /dev
tmpfs 595M 8.1M 587M 2% /run
/dev/mapper/pbs-root 5.6G 1.8G 3.5G 34% /
tmpfs 3.0G 0 3.0G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.0G 0 3.0G 0% /sys/fs/cgroup
HDD 1.8T 1.8T 4.2M 100% /mnt/datastore/HDD
tmpfs 595M 0 595M 0% /run/user/0

 

[Stefan_R]

That's a tricky situation then, though it looks like you have 4.2M of free space remaining? You can't remove chunks without deleting data, but you can move a few somewhere else and then run the GC, afterwards move them back. Note that any backup snapshots containing those chunks will be marked bad until you run a manual verification again, and GC will probably complain about those missing chunks as well.

 

[Elliott Partridge]

That's a tricky situation then, though it looks like you have 4.2M of free space remaining? You can't remove chunks without deleting data, but you can move a few somewhere else and then run the GC, afterwards move them back. Note that any backup snapshots containing those chunks will be marked bad until you run a manual verification again, and GC will probably complain about those missing chunks as well.

Thanks, this is the approach I took with my own 100% full backup drive. Seems to work just fine, although I'm still running the verify job after garbage collecting & moving chunks back.

 

[AngryAnt]

Currently running this chunk juggling dance as well. I realize there's obviously a monitoring failure at play here, but it would be pretty nice if pbs would slam the brakes on filling up storage at a buffers distance from 100% - where that buffer is determined sizable enough to successfully run GC.

 

[Etienne Charlier]

Currently running this chunk juggling dance as well. I realize there's obviously a monitoring failure at play here, but it would be pretty nice if pbs would slam the brakes on filling up storage at a buffers distance from 100% - where that buffer is determined sizable enough to successfully run GC.

of just create a "filler" file at repository creation. In case of gc fails with no space.. delete the filler file and restart ...

 

[AngryAnt]

of just create a "filler" file at repository creation. In case of gc fails with no space.. delete the filler file and restart ...

That would indeed be a very viable buffer.

On resolution, like recommended I moved some chunks off, ran gc with warnings, moved the chunks back, re-ran gc cleanly, started validation.