Disk full - unable to run garbage collection

dff

Member
Jan 4, 2021
3
0
6
54
Backup Server 1.0-6

Noticed the disk was full. Tried to run garbage collection from the web ui .
unable to start garbage collection job on datastore backups - ENOSPC: No space left on device

Manually deleted snapshots in the web ui and from the command line and rebooted - problem persists.

Ran proxmox-backup-client garbage-collect --repository backups
unable to start garbage collection job on datastore backups - ENOSPC: No space left on device

Ran proxmox-backup-manager garbage-collection start backups
unable to start garbage collection job on datastore backups - ENOSPC: No space left on device

Im happy to provide logs if you tell me which ones are helpful. Thanks in advance

Code:
Filesystem      Size  Used Avail Use% Mounted on
udev            2.9G     0  2.9G   0% /dev
tmpfs           595M   16M  580M   3% /run
/dev/sda2       490G  465G     0 100% /
tmpfs           3.0G     0  3.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.0G     0  3.0G   0% /sys/fs/cgroup
/dev/sda1       511M  6.6M  505M   2% /boot/efi
tmpfs           595M     0  595M   0% /run/user/1001
 
This behaviour is certainly not ideal, but currently you need at least *some* free space to run most PBS operations. Try deleting some unnecessary data and try again (e.g. some old (gzipped) log files from /var/log are often a good candidate).
 
This behaviour is certainly not ideal, but currently you need at least *some* free space to run most PBS operations. Try deleting some unnecessary data and try again (e.g. some old (gzipped) log files from /var/log are often a good candidate).
Thank you
Removing old log files did the trick. Garbage collection could run again.
 
Is this a test machine?
If no, then give the poor thing a respectable hard drive :)
In addition, I recommend to install an SNMP service on PVE and therefore also on PBS to collect the values from a monitoring system (e.g. PRTG).
 
  • Like
Reactions: Stefan_R
i have the same situation but in ZFS backup drive
Code:
HDD                   1.8T  1.8T  3.0M 100% /mnt/datastore/HDD

Code:
# proxmox-backup-client garbage-collect --repository HDD
starting garbage collection on store HDD
Start GC phase1 (mark used chunks)
TASK ERROR: update atime failed for chunk/file "/mnt/datastore/HDD/.chunks/bb9f/bb9f8df61474d25e71fa00722318cd387396ca1736605e1248821cc0de3d3af8" - ENOSPC: No space left on device
# proxmox-backup-manager garbage-collection start HDD
starting garbage collection on store HDD
Start GC phase1 (mark used chunks)
TASK ERROR: update atime failed for chunk/file "/mnt/datastore/HDD/.chunks/a1ba/a1ba55b9c10ebc59ceef92b920f443dd542121929e55a7bd85c5160b73d3a859" - ENOSPC: No space left on device
Can I delete some chunks?
 
Can I delete some chunks?
If you want to lose your data, sure ;) Otherwise I'd rather recommend deleting *unused* files (e.g. /var/log/ is a good place to start) first, and then try the GC again. Just a few kilobytes of free space should already be enough to start the operation, and it will then clear up more space as it goes.
 
ok, I try, but on main partition I have a lot of free space

Filesystem Size Used Avail Use% Mounted on
udev 2.9G 0 2.9G 0% /dev
tmpfs 595M 8.1M 587M 2% /run
/dev/mapper/pbs-root 5.6G 1.8G 3.5G 34% /
tmpfs 3.0G 0 3.0G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.0G 0 3.0G 0% /sys/fs/cgroup
HDD 1.8T 1.8T 4.2M 100% /mnt/datastore/HDD
tmpfs 595M 0 595M 0% /run/user/0
 
That's a tricky situation then, though it looks like you have 4.2M of free space remaining? You can't remove chunks without deleting data, but you can move a few somewhere else and then run the GC, afterwards move them back. Note that any backup snapshots containing those chunks will be marked bad until you run a manual verification again, and GC will probably complain about those missing chunks as well.
 
That's a tricky situation then, though it looks like you have 4.2M of free space remaining? You can't remove chunks without deleting data, but you can move a few somewhere else and then run the GC, afterwards move them back. Note that any backup snapshots containing those chunks will be marked bad until you run a manual verification again, and GC will probably complain about those missing chunks as well.
Thanks, this is the approach I took with my own 100% full backup drive. Seems to work just fine, although I'm still running the verify job after garbage collecting & moving chunks back.
 
Currently running this chunk juggling dance as well. I realize there's obviously a monitoring failure at play here, but it would be pretty nice if pbs would slam the brakes on filling up storage at a buffers distance from 100% - where that buffer is determined sizable enough to successfully run GC.
 
  • Like
Reactions: DerDanilo
Currently running this chunk juggling dance as well. I realize there's obviously a monitoring failure at play here, but it would be pretty nice if pbs would slam the brakes on filling up storage at a buffers distance from 100% - where that buffer is determined sizable enough to successfully run GC.
of just create a "filler" file at repository creation. In case of gc fails with no space.. delete the filler file and restart ...
 
  • Like
Reactions: Bent and AngryAnt

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!