I ran out of space ...

[Nhoague]

Welp, I did it. I maxed out of my PBS, version 4.1.0. I can't run prune or GC now, and I get this error:

2026-05-15T09:35:42-06:00: TASK ERROR: atime safety check failed: update atime failed for chunk/file "/mnt/datastore/LKG02/.chunks/bb9f/bb9f8df61474d25e71fa00722318cd387396ca1736605e1248821cc0de3d3af8" - ENOSPC: No space left on device

I had a prune schedule to keep it maintained, but not sure if its correct:



And I may need to tone that down depending, but how can I clear space manually? I tried removing old snapshots from PBS, but without being able to run garbage collection how do I free it up?

Thanks much everyone!

 

[Impact]

Can you tell us a bit about the datastore? What file system? Local? df -hT and lsblk -o+FSTYPE would be interesting.

 

[Nhoague]

Yes! PBS runs the OS on SSD. ZFS RAIDZ1 for the datastore. Consisting of 8 x 2 TB spinners. *Apologize, how do you paste results so its looks nice n pretty?



root@LKG02:~# df -hT
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 119G 0 119G 0% /dev
tmpfs tmpfs 24G 1.5M 24G 1% /run
/dev/mapper/pbs-root ext4 433G 5.0G 406G 2% /
tmpfs tmpfs 119G 0 119G 0% /dev/shm
efivarfs efivarfs 128K 112K 12K 91% /sys/firmware/efi/efivars
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs tmpfs 1.0M 0 1.0M 0% /run/credentials/systemd-journald.service
tmpfs tmpfs 119G 0 119G 0% /tmp
tmpfs tmpfs 119G 380K 119G 1% /run/proxmox-backup
/dev/sdi2 vfat 1022M 8.8M 1014M 1% /boot/efi
LKG02 zfs 13T 13T 0 100% /mnt/datastore/LKG02
tmpfs tmpfs 1.0M 0 1.0M 0% /run/credentials/getty@tty1.service
tmpfs tmpfs 24G 4.0K 24G 1% /run/user/0
root@LKG02:~#

root@LKG02:~# lsblk -o+FSTYPE
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS FSTYPE
sda 8:0 0 1.8T 0 disk
├─sda1 8:1 0 1.8T 0 part zfs_member
└─sda9 8:9 0 8M 0 part
sdb 8:16 0 1.8T 0 disk
├─sdb1 8:17 0 1.8T 0 part zfs_member
└─sdb9 8:25 0 8M 0 part
sdc 8:32 0 1.8T 0 disk
├─sdc1 8:33 0 1.8T 0 part zfs_member
└─sdc9 8:41 0 8M 0 part
sdd 8:48 0 1.8T 0 disk
├─sdd1 8:49 0 1.8T 0 part zfs_member
└─sdd9 8:57 0 8M 0 part
sde 8:64 0 1.8T 0 disk
├─sde1 8:65 0 1.8T 0 part zfs_member
└─sde9 8:73 0 8M 0 part
sdf 8:80 0 1.8T 0 disk
├─sdf1 8:81 0 1.8T 0 part zfs_member
└─sdf9 8:89 0 8M 0 part
sdg 8:96 0 1.8T 0 disk
├─sdg1 8:97 0 1.8T 0 part zfs_member
└─sdg9 8:105 0 8M 0 part
sdh 8:112 0 1.8T 0 disk
├─sdh1 8:113 0 1.8T 0 part zfs_member
└─sdh9 8:121 0 8M 0 part
sdi 8:128 0 465.8G 0 disk
├─sdi1 8:129 0 1007K 0 part
├─sdi2 8:130 0 1G 0 part /boot/efi vfat
└─sdi3 8:131 0 464.8G 0 part LVM2_member
├─pbs-swap 252:0 0 8G 0 lvm [SWAP] swap
└─pbs-root 252:1 0 440.8G 0 lvm / ext4
root@LKG02:~#

 

[Nhoague]

I've tried changing the prune, and running some local commands I found:

find /path2pbs-datastore/.chunks -type f -print0 | xargs -0 touch -d "-2 days"

But, when I run this command, because there is no space left it fails. Pretty much loops and repeats the same error message from the PBS log.

 

[Impact]

That's hard to read without code blocks. For ZFS see here: https://forum.proxmox.com/threads/p...ot-become-completely-full.166768/#post-774198

 

[Nhoague]

Amazing! I had to go up to level 8 before I could start a garbage collect! It's running now ... thank you!

*Code blocks. My bad haha

 

[Onslow]

how do you paste results so its looks nice n pretty?

Use CODE blocks. This </> icon in the composing menu.