Disk full - unable to run garbage collection

D

dff

Member
Jan 4, 2021
5
0
6
55
Backup Server 1.0-6

Noticed the disk was full. Tried to run garbage collection from the web ui .
unable to start garbage collection job on datastore backups - ENOSPC: No space left on device

Manually deleted snapshots in the web ui and from the command line and rebooted - problem persists.

Ran proxmox-backup-client garbage-collect --repository backups
unable to start garbage collection job on datastore backups - ENOSPC: No space left on device

Ran proxmox-backup-manager garbage-collection start backups
unable to start garbage collection job on datastore backups - ENOSPC: No space left on device

Im happy to provide logs if you tell me which ones are helpful. Thanks in advance

Code: 
Filesystem      Size  Used Avail Use% Mounted on
udev            2.9G     0  2.9G   0% /dev
tmpfs           595M   16M  580M   3% /run
/dev/sda2       490G  465G     0 100% /
tmpfs           3.0G     0  3.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.0G     0  3.0G   0% /sys/fs/cgroup
/dev/sda1       511M  6.6M  505M   2% /boot/efi
tmpfs           595M     0  595M   0% /run/user/1001
 
This behaviour is certainly not ideal, but currently you need at least *some* free space to run most PBS operations. Try deleting some unnecessary data and try again (e.g. some old (gzipped) log files from /var/log are often a good candidate).
 
Stefan_R said:
This behaviour is certainly not ideal, but currently you need at least *some* free space to run most PBS operations. Try deleting some unnecessary data and try again (e.g. some old (gzipped) log files from /var/log are often a good candidate).
Click to expand...
Thank you
Removing old log files did the trick. Garbage collection could run again.
 
Is this a test machine?
If no, then give the poor thing a respectable hard drive :-)
In addition, I recommend to install an SNMP service on PVE and therefore also on PBS to collect the values from a monitoring system (e.g. PRTG).
 
  • Like
Reactions: Stefan_R
i have the same situation but in ZFS backup drive
Code: 
HDD                   1.8T  1.8T  3.0M 100% /mnt/datastore/HDD

Code: 
# proxmox-backup-client garbage-collect --repository HDD
starting garbage collection on store HDD
Start GC phase1 (mark used chunks)
TASK ERROR: update atime failed for chunk/file "/mnt/datastore/HDD/.chunks/bb9f/bb9f8df61474d25e71fa00722318cd387396ca1736605e1248821cc0de3d3af8" - ENOSPC: No space left on device
# proxmox-backup-manager garbage-collection start HDD
starting garbage collection on store HDD
Start GC phase1 (mark used chunks)
TASK ERROR: update atime failed for chunk/file "/mnt/datastore/HDD/.chunks/a1ba/a1ba55b9c10ebc59ceef92b920f443dd542121929e55a7bd85c5160b73d3a859" - ENOSPC: No space left on device
Can I delete some chunks?
 
robson_ek said:
Can I delete some chunks?
Click to expand...
If you want to lose your data, sure ;) Otherwise I'd rather recommend deleting *unused* files (e.g. /var/log/ is a good place to start) first, and then try the GC again. Just a few kilobytes of free space should already be enough to start the operation, and it will then clear up more space as it goes.
 
ok, I try, but on main partition I have a lot of free space

Filesystem Size Used Avail Use% Mounted on
udev 2.9G 0 2.9G 0% /dev
tmpfs 595M 8.1M 587M 2% /run
/dev/mapper/pbs-root 5.6G 1.8G 3.5G 34% /
tmpfs 3.0G 0 3.0G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.0G 0 3.0G 0% /sys/fs/cgroup
HDD 1.8T 1.8T 4.2M 100% /mnt/datastore/HDD
tmpfs 595M 0 595M 0% /run/user/0
Click to expand...
 
That's a tricky situation then, though it looks like you have 4.2M of free space remaining? You can't remove chunks without deleting data, but you can move a few somewhere else and then run the GC, afterwards move them back. Note that any backup snapshots containing those chunks will be marked bad until you run a manual verification again, and GC will probably complain about those missing chunks as well.
 
  • Like
Reactions: geffin, _gabriel and Elliott Partridge
Stefan_R said:
That's a tricky situation then, though it looks like you have 4.2M of free space remaining? You can't remove chunks without deleting data, but you can move a few somewhere else and then run the GC, afterwards move them back. Note that any backup snapshots containing those chunks will be marked bad until you run a manual verification again, and GC will probably complain about those missing chunks as well.
Click to expand...
Thanks, this is the approach I took with my own 100% full backup drive. Seems to work just fine, although I'm still running the verify job after garbage collecting & moving chunks back.
 
Currently running this chunk juggling dance as well. I realize there's obviously a monitoring failure at play here, but it would be pretty nice if pbs would slam the brakes on filling up storage at a buffers distance from 100% - where that buffer is determined sizable enough to successfully run GC.
 
  • Like
Reactions: DerDanilo
AngryAnt said:
Currently running this chunk juggling dance as well. I realize there's obviously a monitoring failure at play here, but it would be pretty nice if pbs would slam the brakes on filling up storage at a buffers distance from 100% - where that buffer is determined sizable enough to successfully run GC.
Click to expand...
of just create a "filler" file at repository creation. In case of gc fails with no space.. delete the filler file and restart ...
 
  • Like
Reactions: Bent and AngryAnt
You must log in or register to reply here.
Top Bottom