[SOLVED] LXC suddenly at full bootdisk size from nowhere

papatikka

Member
Jun 26, 2023
33
1
8
Hello,

one of my container went suddenly full from like 1GB to 19.50 (19.52) GB.


Code:
pveversion --verbose
proxmox-ve: 8.0.1 (running kernel: 6.2.16-3-pve)
pve-manager: 8.0.3 (running version: 8.0.3/bbf3993334bfa916)
pve-kernel-6.2: 8.0.2
pve-kernel-5.15: 7.4-4
pve-kernel-6.2.16-3-pve: 6.2.16-3
pve-kernel-5.15.108-1-pve: 5.15.108-1
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-3
libknet1: 1.25-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.0
libpve-access-control: 8.0.3
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.5
libpve-guest-common-perl: 5.0.3
libpve-http-server-perl: 5.0.3
libpve-rs-perl: 0.8.3
libpve-storage-perl: 8.0.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.1-1
proxmox-backup-file-restore: 3.0.1-1
proxmox-kernel-helper: 8.0.2
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.0.5
pve-cluster: 8.0.1
pve-container: 5.0.4
pve-docs: 8.0.4
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.2
pve-firmware: 3.7-1
pve-ha-manager: 4.0.2
pve-i18n: 3.0.4
pve-qemu-kvm: 8.0.2-3
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.6
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.12-pve1

When running du -cha --max-depth=2 --exclude=/mnt / 2>/dev/null

Code:
32K     /usr/include
4.0K    /usr/lib64
15M     /usr/sbin
4.0K    /usr/src
4.4M    /usr/libexec
142M    /usr/share
4.0K    /usr/libx32
4.0K    /usr/lib32
76K     /usr/local
328M    /usr/lib
79M     /usr/bin
4.0K    /usr/games
567M    /usr
0       /dev/hugepages
0       /dev/log
0       /dev/initctl
0       /dev/core
0       /dev/shm
4.0K    /dev/.lxc-boot-id
0       /dev/stderr
0       /dev/stdout
0       /dev/stdin
0       /dev/fd
0       /dev/tty2
0       /dev/tty1
0       /dev/console
0       /dev/lxc
0       /dev/ptmx
0       /dev/zero
0       /dev/urandom
0       /dev/tty
0       /dev/random
0       /dev/null
0       /dev/full
0       /dev/.lxc
0       /dev/mqueue
0       /dev/pts
4.0K    /dev
0       /sys/kernel
0       /sys/power
0       /sys/class
0       /sys/devices
0       /sys/dev
0       /sys/hypervisor
0       /sys/fs
0       /sys/bus
0       /sys/firmware
0       /sys/block
0       /sys/module
0       /sys
4.0K    /etc/passwd-
4.0K    /etc/passwd
4.0K    /etc/request-key.conf
0       /etc/vtrgb
24K     /etc/samba
4.0K    /etc/gshadow
0       /etc/subuid
16K     /etc/udev
36K     /etc/sysctl.d
4.0K    /etc/crontab
4.0K    /etc/magic.mime
4.0K    /etc/rc1.d
20K     /etc/network
8.0K    /etc/ca-certificates.conf
8.0K    /etc/newt
4.0K    /etc/adduser.conf
8.0K    /etc/sudo.conf
4.0K    /etc/rc5.d
20K     /etc/X11
56K     /etc/security
4.0K    /etc/gshadow-
4.0K    /etc/legal
8.0K    /etc/manpath.config
8.0K    /etc/cron.yearly
0       /etc/.pwd.lock
4.0K    /etc/protocols
8.0K    /etc/ldap
20K     /etc/ppp
4.0K    /etc/nsswitch.conf
4.0K    /etc/issue
4.0K    /etc/group-
4.0K    /etc/issue.net
632K    /etc/ssl
8.0K    /etc/selinux
4.0K    /etc/rc2.d
4.0K    /etc/inputrc
56K     /etc/iproute2
4.0K    /etc/netplan
4.0K    /etc/rc3.d
4.0K    /etc/logrotate.conf
4.0K    /etc/gai.conf
12K     /etc/vim
4.0K    /etc/machine-id
28K     /etc/cron.daily
28K     /etc/update-motd.d
4.0K    /etc/libaudit.conf
4.0K    /etc/fstab
4.0K    /etc/nftables.conf
4.0K    /etc/rsyslog.conf
4.0K    /etc/sgml
0       /etc/rmt
4.0K    /etc/group
16K     /etc/skel
4.0K    /etc/debconf.conf
4.0K    /etc/hosts
4.0K    /etc/bash.bashrc
4.0K    /etc/nfs.conf.d
12K     /etc/ld.so.conf.d
4.0K    /etc/networks
12K     /etc/nanorc
16K     /etc/rsyslog.d
12K     /etc/kernel
8.0K    /etc/cron.monthly
4.0K    /etc/aliases
8.0K    /etc/wgetrc
616K    /etc/ssh
4.0K    /etc/appliance.info
4.0K    /etc/ethertypes
4.0K    /etc/host.conf
4.0K    /etc/modules
4.0K    /etc/zsh_command_not_found
44K     /etc/dhcp
12K     /etc/aliases.db
8.0K    /etc/depmod.d
4.0K    /etc/ld.so.conf
4.0K    /etc/shadow-
4.0K    /etc/e2scrub.conf
200K    /etc/console-setup
16K     /etc/request-key.d
4.0K    /etc/nfs.conf
20K     /etc/update-manager
8.0K    /etc/cron.hourly
4.0K    /etc/shells
12K     /etc/profile.d
4.0K    /etc/xattr.conf
40K     /etc/logrotate.d
4.0K    /etc/resolv.conf
8.0K    /etc/sudoers.d
4.0K    /etc/lsb-release
4.0K    /etc/deluser.conf
4.0K    /etc/profile
4.0K    /etc/hosts.deny
16K     /etc/apparmor.d
4.0K    /etc/rc0.d
4.0K    /etc/ucf.conf
0       /etc/subgid
16K     /etc/ubuntu-advantage
28K     /etc/modprobe.d
16K     /etc/services
8.0K    /etc/terminfo
12K     /etc/apparmor
4.0K    /etc/magic
4.0K    /etc/hosts.allow
4.0K    /etc/locale.alias
8.0K    /etc/gss
4.0K    /etc/tmpfiles.d
4.0K    /etc/modules-load.d
4.0K    /etc/opt
12K     /etc/groff
4.0K    /etc/timezone
4.0K    /etc/hostname
24K     /etc/dpkg
124K    /etc/postfix
4.0K    /etc/fuse.conf
24K     /etc/xdg
12K     /etc/insserv.conf.d
72K     /etc/libibverbs.d
8.0K    /etc/ca-certificates
12K     /etc/dbus-1
4.0K    /etc/rc6.d
4.0K    /etc/binfmt.d
12K     /etc/sudo_logsrvd.conf
4.0K    /etc/rpc
28K     /etc/networkd-dispatcher
8.0K    /etc/python3
4.0K    /etc/bash_completion
0       /etc/localtime
16K     /etc/ld.so.cache
64K     /etc/ufw
96K     /etc/init.d
0       /etc/.resolv.conf.systemd-resolved.bak
12K     /etc/libnl-3
12K     /etc/resolvconf
4.0K    /etc/rc4.d
12K     /etc/locale.gen
16K     /etc/logcheck
4.0K    /etc/netconfig
4.0K    /etc/mke2fs.conf
8.0K    /etc/wireguard
120K    /etc/systemd
8.0K    /etc/alternatives
4.0K    /etc/cifs-utils
4.0K    /etc/debian_version
0       /etc/mtab
28K     /etc/unpackerr
4.0K    /etc/shadow
4.0K    /etc/bindresvport.blacklist
4.0K    /etc/environment
4.0K    /etc/pam.conf
4.0K    /etc/keyutils
8.0K    /etc/python3.11
4.0K    /etc/idmapd.conf
80K     /etc/apt
60K     /etc/default
92K     /etc/pam.d
8.0K    /etc/hdparm.conf
4.0K    /etc/sudoers
12K     /etc/cron.weekly
16K     /etc/login.defs
0       /etc/os-release
4.0K    /etc/sysctl.conf
76K     /etc/mime.types
12K     /etc/cron.d
4.0K    /etc/rcS.d
3.6M    /etc
8.0K    /tmp/systemd-private-351553b6fe6a42fea74687d748ecbe18-systemd-logind.service-cwOCj5
4.0K    /tmp/.XIM-unix
4.0K    /tmp/.ICE-unix
4.0K    /tmp/.X11-unix
4.0K    /tmp/.font-unix
8.0K    /tmp/systemd-private-351553b6fe6a42fea74687d748ecbe18-systemd-resolved.service-BtaDz0
36K     /tmp
4.0K    /srv
0       /proc/fb
0       /proc/fs
0       /proc/bus
0       /proc/dma
0       /proc/irq
0       /proc/net
0       /proc/spl
4.0K    /proc/sys
0       /proc/tty
0       /proc/acpi
0       /proc/keys
0       /proc/kmsg
0       /proc/misc
0       /proc/mtrr
0       /proc/scsi
0       /proc/stat
0       /proc/iomem
0       /proc/kcore
0       /proc/locks
0       /proc/swaps
0       /proc/asound
0       /proc/crypto
0       /proc/driver
0       /proc/mdstat
0       /proc/mounts
0       /proc/uptime
0       /proc/vmstat
0       /proc/cgroups
0       /proc/cmdline
0       /proc/cpuinfo
0       /proc/devices
0       /proc/ioports
0       /proc/loadavg
0       /proc/meminfo
0       /proc/modules
0       /proc/sysvipc
0       /proc/version
0       /proc/consoles
0       /proc/kallsyms
0       /proc/pressure
0       /proc/slabinfo
0       /proc/softirqs
0       /proc/zoneinfo
0       /proc/buddyinfo
0       /proc/diskstats
0       /proc/key-users
0       /proc/schedstat
0       /proc/bootconfig
0       /proc/interrupts
0       /proc/kpagecount
0       /proc/kpageflags
0       /proc/partitions
0       /proc/timer_list
0       /proc/execdomains
0       /proc/filesystems
0       /proc/kpagecgroup
0       /proc/vmallocinfo
0       /proc/pagetypeinfo
0       /proc/dynamic_debug
0       /proc/sysrq-trigger
0       /proc/version_signature
0       /proc/self
0       /proc/thread-self
0       /proc/1
0       /proc/48
0       /proc/87
0       /proc/93
0       /proc/94
0       /proc/97
0       /proc/99
0       /proc/106
0       /proc/107
0       /proc/108
0       /proc/109
0       /proc/178
0       /proc/221
0       /proc/234
0       /proc/236
0       /proc/241
0       /proc/245
0       /proc/246
0       /proc/247
0       /proc/325
0       /proc/326
0       /proc/327
0       /proc/373
0       /proc/410
0       /proc/411
0       /proc/417
0       /proc/471
4.0K    /proc
0       /lib64
0       /sbin
50M     /var/log
20K     /var/tmp
780K    /var/spool
856K    /var/backups
4.0K    /var/opt
4.0K    /var/local
89M     /var/cache
0       /var/lock
0       /var/run
210M    /var/lib
4.0K    /var/mail
349M    /var
4.0K    /home
4.0K    /boot
4.0K    /media
16K     /lost+found
0       /libx32
4.0K    /opt
0       /lib32
4.0K    /.cache/qBittorrent
8.0K    /.cache
4.0K    /run/motd.dynamic
0       /run/agetty.reload
0       /run/crond.reboot
4.0K    /run/crond.pid
3.1M    /run/samba
4.0K    /run/sm-notify.pid
4.0K    /run/sshd.pid
0       /run/uuidd
0       /run/rpcbind.lock
0       /run/rpc_pipefs
4.0K    /run/utmp
0       /run/user
0       /run/sudo
0       /run/rpcbind
0       /run/sshd
0       /run/sendsigs.omit.d
0       /run/shm
0       /run/dbus
0       /run/console-setup
8.0M    /run/log
0       /run/mount
0       /run/initctl
0       /run/rpcbind.sock
0       /run/credentials
140K    /run/systemd
0       /run/lock
12M     /run
0       /lib
0       /bin
15M     /.local/share
15M     /.local
28K     /.config/qBittorrent
32K     /.config
4.0K    /root/.Donkeycredentials
16K     /root/.bash_history
4.0K    /root/.ssh
4.0K    /root/.lesshst
4.0K    /root/.HQcredentials
4.0K    /root/.bashrc
8.0K    /root/.cache
52K     /root/.local
28K     /root/.config
4.0K    /root/.profile
132K    /root
946M    /
946M    total
root@TorrentContainer:~#

It says only 1 GB is occupied right?

df -h

Code:
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/pve-vm--102--disk--0   20G   20G     0 100% /
none                              492K  4.0K  488K   1% /dev
tmpfs                             7.8G     0  7.8G   0% /dev/shm
tmpfs                             3.1G   12M  3.1G   1% /run
tmpfs                             5.0M     0  5.0M   0% /run/lock

lsof / | grep deleted
yields nothing!

what can I do?
 
I cannot run sudo du -a / | sort -n -r | head -n 20

because i get permission denied at /dev/.lxc/sys/foo and /dev/.lxc/proc/foo2 level

id yields
uid=0(root) gid=0(root) groups=0(root)

only until sys and proc I can read, nothing further than that
 
I am sorry, this is the 4th time that I start a thread and solve it by myself. Found the culprit, it was by human error because I ticked off a option in a program, of which, started to put incompleted files into /mnt in a hidden folder.
 
Can you tell me specifically how you solved this. I think I am running into exactly the same problem and beating my head over it.
 
Can you tell me specifically how you solved this. I think I am running into exactly the same problem and beating my head over it.

Went into qbittorrent-nox and changed path of temporary files from LXC to fileserver where they were before, I misclicked when configuring qbit-nox.