[SOLVED] PBS Root Disk Full can't sync, backup or run garbage collection.

[Jarvar]

A few days ago, my Proxmox Backup Server filled up in the root folder. I managed to delete some large log files then and get functional.
Last night during a remote sync it filled up and my job failed.
Could somebody please help me diagnose the issue?
Thanks, This is out of a 100 GB drive.

this is what I found so far.

root@pbs1:~# du -sh /* | sort -hr
du: cannot access '/proc/1177/task/1177/fd/4': No such file or directory
du: cannot access '/proc/1177/task/1177/fdinfo/4': No such file or directory
du: cannot access '/proc/1177/fd/3': No such file or directory
du: cannot access '/proc/1177/fdinfo/3': No such file or directory
1.7T    /backup
3.9G    /usr
1.1G    /var
574M    /boot
4.0M    /etc
740K    /run
80K     /root
40K     /tmp
16K     /lost+found
12K     /mnt
4.0K    /srv
4.0K    /opt
4.0K    /media
4.0K    /home
4.0K    /disk3
0       /sys
0       /sbin
0       /proc
0       /libx32
0       /lib64
0       /lib32
0       /lib
0       /dev
0       /bin
root@pbs1:~#

 

[datschlatscher]

A bit more information about your storage topology is needed here. Do you have any hidden folders in your root directory for some reason? du -sh /.* | sort -hr

Is /backup mounted directly or do you have subfolders mounted in there? What does lsblk say?

You should be able to get a hold of where most of your data by using something like baobab (Disk Usage Analyzer) if you are running a Desktop environment. If you only have terminal access consider a tool like ncdu. Both being really easy to use and great to quickly tell where most of your disk space goes.

 

[Jarvar]

root@pbs1:/backup# du -sh /.* | sort -hr du: cannot access '/./proc/1298/task/1298/fd/4': No such file or directory du: cannot access '/./proc/1298/task/1298/fdinfo/4': No such file or directory du: cannot access '/./proc/1298/fd/3': No such file or directory du: cannot access '/./proc/1298/fdinfo/3': No such file or directory 1.9T /. root@pbs1:/backup#

I didn't do anything specific to hide anything in the root directory.
/backup was mounted to an external USB drive for the datastore.

At this moment I don't have enough free space to install ncdu\

lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 100G 0 disk ├─sda1 8:1 0 1007K 0 part ├─sda2 8:2 0 512M 0 part └─sda3 8:3 0 99.5G 0 part ├─pbs-swap 253:0 0 3.9G 0 lvm [SWAP] └─pbs-root 253:1 0 95.6G 0 lvm / sdb 8:16 0 3.6T 0 disk └─sdb1 8:17 0 3.6T 0 part /mnt/usb-drive01/disk04 sr0 11:0 1 1024M 0 rom root@pbs1:/backup#

 

[Jarvar]

I think I solved my own issue. There was time where something didn't mount correctly. I mounted an NFS as well to backup the contents of my external hard drive to rsync things over as an added backup. When it failed, it filled up.
So I edited it out in the fstab, rebooted, deleted the contents of the folder which was about 89G and I'm cleaning up old mounts and will try it again.
df -h brings up only 7% drive use now which is good for me now.
I'll apt install ncdu -y now.
Thank you so much!

 

[Jarvar]

Any recommendations on which files to delete to free up space? This occurred with with a new remote which was supposed to be my remote backup server but became full. I ran garbage collection several times, but I need to wait 24 hours and 5 minutes for it to free up?
Any help would be much appreciated.

 

[datschlatscher]

I ran garbage collection several times, but I need to wait 24 hours and 5 minutes for it to free up?

Garbage collection will only delete unused chunks if they haven't been touched for 24 hours and 5 minutes. E.g., after pruning, you need to wait for 1 day before running GC for it to free up any space.

Any recommendations on which files to delete to free up space?

Usual suspects are journal log files which can be configured to be cleaned up regularly via the --vacuum option and sometimes the apt package cache.

However, in my experience, ~100GB is usually sufficient. Are you doing anything particular with your setup? Other things to look out for would be leftover build directories from compiled programs (if that is relevant for you), big iso files and old backups saved to your local disk storage.

 

[Jarvar]

Hello @datschlatscher I have proxmox backup server running at another place without a separate root drive. The Data and the OS are on the same drive. I think atleast with ZFS there is an ability to set quota.

 

[datschlatscher]

If I understand you correctly: Are you looking for setting a quota on your block devices created in ZFS? If so, here should be what you are looking for [Setting ZFS Quotas and Reservations].

Please keep in mind, I'm not that familiar with ZFS in general, so please don't hesitate if I got you wrong here.

 

[coldwater]

My Proxmox Backup Server (Backup Server 2.4-1) has 109.31 GB HD space root that ended up with 99.98% fillrate and the two main logs I blame for that is 46G ./daemon.log and 46G ./syslog which I have now deleted (though it seems I'll have to wait 24 hours before df shows any different than 99.98% full on /..
This is why Proxmox doesn't get picked for enterprise solutions (even nowadays when VMware seemingly doesn't want regular customers anymore)..

 

[Ernst T.]

Log files are deleted immediately! The 24 hours only apply to pbs-chunks!

 

[tcabernoch]

People frequently misunderstand what PBS is and does.
As Ernst indicates, you can purge log files the regular way, but the chunks system is more akin to a database of file parts.
You have to manage it as such.

Do you want to screw up your backup system? Here. Try this.
This command essentially purges ALL of your backups and replications.
It marks all backups as 2 days old, and eligible for Garbage Collection.

find /path2pbs-datastore/.chunks -type f -print0 | xargs -0 touch -d "-2 days"

Have fun with that.

 

[coldwater]

They seem to be gone from the folder after deleting them, yes, but df -h still showed 99.98% full after deleting two logs that were both 46 GB (on a volume that is 109.31 GB). The only datastores on this Proxmox Backup Server are on separate drives (small 128 GB NVMe for PBS system drive, one 2TB SATA SSD for one datastore and one 5 TB HDD used as a secondary datastore), so datastore chunks are not what's filling up the root drive, but somehow I'm not getting more space in / after deleting the 46 GB syslog and 46 GB daemon.log file (which frankly never should have been allowed to grow this large anyways).
24 hours later I now have 89% full / and in /var/log the new syslog is 1.3 M and damon.log is also 1.3 M. The biggest files by a long shot in /var/log now is proxmox-backup at 4.1 G and journal at 4.0 G (so I have no idea what's taking up 89% of 109 GB root now that the two 46 GB logs are deleted.. It should have opened up 92 GB, not just 11 GB..)

 

[Ernst T.]

You can just wait until the next night. Debian-based systems run an automatic fstrim in the night from Sunday to Monday.

Or run the command manually: fstrim -av