pve root full, can't find consumtion

B

B1G_WH1T3

New Member
Jul 12, 2024
6
1
3
hi

So quick back story, my JBODd was busy &$% the bed, so I had to do a reboot of my PVE node, after the reboot I started everything and without checking one of my LXC's disks didn't mount correctly and it looks like it consumed all the space on /dev/mapper/pve-root.

Now, the node was unusable since its disk space was 0. I eventually got a small amount of space available and extended the LVM to 77 GB. But now the issue is I can't find the space that is being consumed at all. Any tips or locations I should look for?

Df output
Filesystem Size Used Avail Use% Mounted on
udev 44G 0 44G 0% /dev
tmpfs 8.7G 5.5M 8.7G 1% /run
/dev/mapper/pve-root 77G 58G 15G 80% /
tmpfs 44G 34M 44G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
efivarfs 128K 106K 18K 87% /sys/firmware/efi/efivars
/dev/sdb1 469G 207G 239G 47% /mnt/pve/Fastboi
/dev/sdc2 1022M 344K 1022M 1% /boot/efi
/dev/fuse 128M 24K 128M 1% /etc/pve
tmpfs 8.7G 0 8.7G 0% /run/user/0

du -h -x -d1 output

root@pve:/# du -h -x -d1
60K ./mnt
4.0K ./home
4.0K ./media
4.0K ./srv
4.0K ./opt
1.1G ./var
733M ./boot
7.4G ./usr
7.0M ./etc
104K ./root
40K ./tmp
16K ./lost+found
9.1G

am I just missing something or am I blind?
 
Please put the command outputs above in code blocks so it's readable. Also share lsblk -o+FSTYPE and lvs.
 
Last edited:
There are 2 option for the 58 vs 9 GB problem: Once you still had a running process which holds data even if it's deleted and it isn't seen anymore by du (which resolves just by another reboot) or you have - maybe accidently - written data to a path which is not yet accessible because there is a mount onto and so even cannot be counted by du yet (maybe eg /mnt/pve/Fastboi).
 
  • Like
Reactions: leesteken
Impact said:
Please put the command outputs above in code blocks so it's readable. Also share lsblk -o+FSTYPE and lvs.
Click to expand...
apologies for not using the code block my bad, see below output :

Code: 
root@pve:~# lsblk -o+FSTYPE
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS      FSTYPE
loop0          7:0    0   200G  0 loop                  ext4
loop1          7:1    0    40G  0 loop                  ext4
loop2          7:2    0     8G  0 loop                  ext4
loop3          7:3    0    60G  0 loop                  ext4
loop4          7:4    0    24G  0 loop                  ext4
loop5          7:5    0    23G  0 loop                  ext4
loop6          7:6    0     8G  0 loop                  ext4
sdb            8:16   0 476.9G  0 disk                 
└─sdb1         8:17   0 476.9G  0 part /mnt/pve/Fastboi ext4
sdc            8:32   0 111.8G  0 disk                 
├─sdc1         8:33   0  1007K  0 part                 
├─sdc2         8:34   0     1G  0 part /boot/efi        vfat
└─sdc3         8:35   0 110.8G  0 part                  LVM2_member
  ├─pve-swap 252:0    0     8G  0 lvm  [SWAP]           swap
  └─pve-root 252:1    0  77.7G  0 lvm  /                ext4
root@pve:~# lvs
  LV   VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root pve -wi-ao---- <77.70g                                                   
  swap pve -wi-ao----   8.00g
 
Please try to disable and unmount fanboi and then run this to find out what the space is used by
Bash: 
apt install gdu
gdu /
 
Last edited:
Impact said:
Please try to didable and unmount fanboi and then run this to find out what the space is used by
Bash: 
apt install gdu
gdu /
Click to expand...
Thanks will try that when I get downtime on my PVE host again.

as for gdu looks the same as ncdu, but here is the output without unmounting


1751824271657.png
1751824281112.png
1751824295790.png
1751824308340.png

I don't see anything standing out
 
"chattr +i" is nice but will even nicely forgotten while a used script with "mountpoint -q" in root home or /usr/local/bin could be easily backup'ed and even copied to a "next" disk or next host ... no one of us itself is a computer to remember whatever one has done ... ;)
 
  • Like
Reactions: B1G_WH1T3
Hi all

Finally got some time to respond and provide my findings.

So this is what I suspect happened, because the one mount didn't start correctly with PVE boot my deluge instance defaulted to a mount point location on PVE /mnt/downloads and for some reason never showed that it consumed space.

After disabling the fastboi mount and searching, I found the folder with the "linux iso's" cleaned it up and the issue has been fixed since.

appreciate the assistance and input from everyone.
 
  • Like
Reactions: Impact
You must log in or register to reply here.
Top Bottom