[SOLVED] PBS Root Disk Full can't sync, backup or run garbage collection.

Jarvar

Member
Aug 27, 2019
242
9
23
A few days ago, my Proxmox Backup Server filled up in the root folder. I managed to delete some large log files then and get functional.
Last night during a remote sync it filled up and my job failed.
Could somebody please help me diagnose the issue?
Thanks, This is out of a 100 GB drive.

this is what I found so far.
Code:
root@pbs1:~# du -sh /* | sort -hr
du: cannot access '/proc/1177/task/1177/fd/4': No such file or directory
du: cannot access '/proc/1177/task/1177/fdinfo/4': No such file or directory
du: cannot access '/proc/1177/fd/3': No such file or directory
du: cannot access '/proc/1177/fdinfo/3': No such file or directory
1.7T    /backup
3.9G    /usr
1.1G    /var
574M    /boot
4.0M    /etc
740K    /run
80K     /root
40K     /tmp
16K     /lost+found
12K     /mnt
4.0K    /srv
4.0K    /opt
4.0K    /media
4.0K    /home
4.0K    /disk3
0       /sys
0       /sbin
0       /proc
0       /libx32
0       /lib64
0       /lib32
0       /lib
0       /dev
0       /bin
root@pbs1:~#
 
Last edited:

datschlatscher

Proxmox Retired Staff
Retired Staff
Mar 14, 2022
190
60
28
A bit more information about your storage topology is needed here. Do you have any hidden folders in your root directory for some reason? du -sh /.* | sort -hr

Is /backup mounted directly or do you have subfolders mounted in there? What does lsblk say?

You should be able to get a hold of where most of your data by using something like baobab (Disk Usage Analyzer) if you are running a Desktop environment. If you only have terminal access consider a tool like ncdu. Both being really easy to use and great to quickly tell where most of your disk space goes.
 

Jarvar

Member
Aug 27, 2019
242
9
23
root@pbs1:/backup# du -sh /.* | sort -hr du: cannot access '/./proc/1298/task/1298/fd/4': No such file or directory du: cannot access '/./proc/1298/task/1298/fdinfo/4': No such file or directory du: cannot access '/./proc/1298/fd/3': No such file or directory du: cannot access '/./proc/1298/fdinfo/3': No such file or directory 1.9T /. root@pbs1:/backup#

I didn't do anything specific to hide anything in the root directory.
/backup was mounted to an external USB drive for the datastore.

At this moment I don't have enough free space to install ncdu\

lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 100G 0 disk ├─sda1 8:1 0 1007K 0 part ├─sda2 8:2 0 512M 0 part └─sda3 8:3 0 99.5G 0 part ├─pbs-swap 253:0 0 3.9G 0 lvm [SWAP] └─pbs-root 253:1 0 95.6G 0 lvm / sdb 8:16 0 3.6T 0 disk └─sdb1 8:17 0 3.6T 0 part /mnt/usb-drive01/disk04 sr0 11:0 1 1024M 0 rom root@pbs1:/backup#
 

Jarvar

Member
Aug 27, 2019
242
9
23
I think I solved my own issue. There was time where something didn't mount correctly. I mounted an NFS as well to backup the contents of my external hard drive to rsync things over as an added backup. When it failed, it filled up.
So I edited it out in the fstab, rebooted, deleted the contents of the folder which was about 89G and I'm cleaning up old mounts and will try it again.
df -h brings up only 7% drive use now which is good for me now.
I'll apt install ncdu -y now.
Thank you so much!
 
  • Like
Reactions: datschlatscher

Jarvar

Member
Aug 27, 2019
242
9
23
Any recommendations on which files to delete to free up space? This occurred with with a new remote which was supposed to be my remote backup server but became full. I ran garbage collection several times, but I need to wait 24 hours and 5 minutes for it to free up?
Any help would be much appreciated.
 

datschlatscher

Proxmox Retired Staff
Retired Staff
Mar 14, 2022
190
60
28
I ran garbage collection several times, but I need to wait 24 hours and 5 minutes for it to free up?
Garbage collection will only delete unused chunks if they haven't been touched for 24 hours and 5 minutes. E.g., after pruning, you need to wait for 1 day before running GC for it to free up any space.

Any recommendations on which files to delete to free up space?
Usual suspects are journal log files which can be configured to be cleaned up regularly via the --vacuum option and sometimes the apt package cache.

However, in my experience, ~100GB is usually sufficient. Are you doing anything particular with your setup? Other things to look out for would be leftover build directories from compiled programs (if that is relevant for you), big iso files and old backups saved to your local disk storage.
 

Jarvar

Member
Aug 27, 2019
242
9
23
Hello @datschlatscher I have proxmox backup server running at another place without a separate root drive. The Data and the OS are on the same drive. I think atleast with ZFS there is an ability to set quota.
 

datschlatscher

Proxmox Retired Staff
Retired Staff
Mar 14, 2022
190
60
28

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!