Proxhost root disk filling up, can't find culprit

P

paddyirishman

New Member
Jun 22, 2023
3
1
3
Hello,

My proxhost has been purring nicely for some time so I haven't been needing to do much of anything. Just working on the docker/portainer host nested on there for container fun, using plex and so on. I came to update the host today and noticed the root disk (a single nvme 1tb ssd) had 96% usage. Disk is way above whats actually registering as being used.

Ive ran a systemfile check, updates, cleared temp folders, caches, removed stale dump files. I've run lengthy scans to reveal the largest files in / along with ncdu -x / output shown below. There should be approx <=300gb total usage on this drive. Rebooted also of course.

Other notes:

I don't use snaps shots
Its single node setup using a mini PC connected to a QNAP NAS
I have the 6.5.13 kernel pinned for transcoding on the N305 chip via Plex
I have 2 daily backup jobs running 1 to NAS-SSD and 1 local. Retention is set to keep daily 1 on both.

I am really lost now at this point. See the screenshot here showing the /var/lib/vz usage jumping up essentially from 300 - 600 then 600 - 900... 300gb jumps.. Any help here is greatly appreciated chaps.

Thanks.
A confused Irishman :)


Screenshot 2024-10-27 at 14.31.35.pngScreenshot 2024-10-27 at 14.58.44.png


Code: 
ncdu shows

--- / --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  197.2 GiB [##################################] /var
    4.5 GiB [                                  ] /usr
  405.8 MiB [                                  ] /root
  276.0 MiB [                                  ] /boot
    5.2 MiB [                                  ] /etc
   36.0 KiB [                                  ] /tmp
   20.0 KiB [                                  ] /mnt
e  16.0 KiB [                                  ] /lost+found
e   4.0 KiB [                                  ] /srv
e   4.0 KiB [                                  ] /opt
e   4.0 KiB [                                  ] /media
e   4.0 KiB [                                  ] /home
@   0.0   B [                                  ]  lib64
@   0.0   B [                                  ]  sbin
@   0.0   B [                                  ]  lib
@   0.0   B [                                  ]  bin
>   0.0   B [                                  ] /sys
>   0.0   B [                                  ] /run
>   0.0   B [                                  ] /proc
>   0.0   B [                                  ] /dev

contents of mount

root@pve:~# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=16297924k,nr_inodes=4074481,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=3266328k,mode=755,inode64)
/dev/mapper/pve-root on / type ext4 (rw,relatime,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=24588)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
/dev/nvme0n1p2 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
10.242.30.10:/ssd on /mnt/pve/thegaf-ssd type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.242.30.11,local_lock=none,addr=10.242.30.10)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
10.242.30.10:/Multimedia on /mnt/pve/thegaf-nas type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.242.30.11,local_lock=none,addr=10.242.30.10)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=3266324k,nr_inodes=816581,mode=700,inode64)

storage.cfg file

dir: local
        path /var/lib/vz
        content rootdir,iso,vztmpl,snippets,images,backup
        prune-backups keep-last=1
        shared 0

nfs: thegaf-nas
        export /Multimedia
        path /mnt/pve/thegaf-nas
        server 10.242.30.10
        content backup
        prune-backups keep-weekly=1

nfs: thegaf-ssd
        export /ssd
        path /mnt/pve/thegaf-ssd
        server 10.242.30.10
        content backup
        prune-backups keep-weekly=1


root@pve:~# df -h



Filesystem                Size  Used Avail Use% Mounted on

udev                       16G     0   16G   0% /dev

tmpfs                     3.2G  2.1M  3.2G   1% /run

/dev/mapper/pve-root      908G  804G   66G  93% /

tmpfs                      16G   34M   16G   1% /dev/shm

tmpfs                     5.0M     0  5.0M   0% /run/lock

efivarfs                  192K   87K  101K  47% /sys/firmware/efi/efivars

/dev/nvme0n1p2           1022M   12M 1011M   2% /boot/efi

/dev/fuse                 128M   24K  128M   1% /etc/pve

10.242.30.10:/Multimedia   22T   17T  5.0T  77% /mnt/pve/thegaf-nas

10.242.30.10:/ssd         1.8T   74G  1.7T   5% /mnt/pve/thegaf-ssd

tmpfs                     3.2G     0  3.2G   0% /run/user/0



root@pve:~# du -hxd 1 /

4.0K    /media

4.0K    /home

5.2M    /etc

16K     /lost+found

20K     /mnt

36K     /tmp

276M    /boot

4.0K    /srv

4.6G    /usr

406M    /root

198G    /var

4.0K    /opt

203G    /

root@pve:~# fdisk -l

Disk /dev/loop0: 50 GiB, 53687091200 bytes, 104857600 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes





Disk /dev/loop1: 52 GiB, 55834574848 bytes, 109051904 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes





Disk /dev/loop2: 50 GiB, 53687091200 bytes, 104857600 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes





Disk /dev/loop3: 10 GiB, 10737418240 bytes, 20971520 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes





Disk /dev/nvme0n1: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors

Disk model: CT1000P3PSSD8                      

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: gpt

Disk identifier: CB20CEAB-EDAC-4B5A-855B-4398D145DFC7



Device           Start        End    Sectors   Size Type

/dev/nvme0n1p1      34       2047       2014  1007K BIOS boot

/dev/nvme0n1p2    2048    2099199    2097152     1G EFI System

/dev/nvme0n1p3 2099200 1953525134 1951425935 930.5G Linux LVM





Disk /dev/mapper/pve-swap: 8 GiB, 8589934592 bytes, 16777216 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes





Disk /dev/mapper/pve-root: 922.51 GiB, 990535221248 bytes, 1934639104 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes
 
Last edited:
After some more digging I found the issue... My NAS went down and writes were still going to the /mnt/nas folders but locally as drives weren't mounted.

Solved with umount then rm then remounting.

yay.
 
  • Like
Reactions: waltar
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back