[SOLVED] LXC suddenly at full bootdisk size from nowhere

papatikka

Member
Jun 26, 2023
31
0
6
Hello,

one of my container went suddenly full from like 1GB to 19.50 (19.52) GB.


Code:
pveversion --verbose
proxmox-ve: 8.0.1 (running kernel: 6.2.16-3-pve)
pve-manager: 8.0.3 (running version: 8.0.3/bbf3993334bfa916)
pve-kernel-6.2: 8.0.2
pve-kernel-5.15: 7.4-4
pve-kernel-6.2.16-3-pve: 6.2.16-3
pve-kernel-5.15.108-1-pve: 5.15.108-1
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-3
libknet1: 1.25-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.0
libpve-access-control: 8.0.3
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.5
libpve-guest-common-perl: 5.0.3
libpve-http-server-perl: 5.0.3
libpve-rs-perl: 0.8.3
libpve-storage-perl: 8.0.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.1-1
proxmox-backup-file-restore: 3.0.1-1
proxmox-kernel-helper: 8.0.2
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.0.5
pve-cluster: 8.0.1
pve-container: 5.0.4
pve-docs: 8.0.4
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.2
pve-firmware: 3.7-1
pve-ha-manager: 4.0.2
pve-i18n: 3.0.4
pve-qemu-kvm: 8.0.2-3
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.6
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.12-pve1

When running du -cha --max-depth=2 --exclude=/mnt / 2>/dev/null

Code:
32K     /usr/include
4.0K    /usr/lib64
15M     /usr/sbin
4.0K    /usr/src
4.4M    /usr/libexec
142M    /usr/share
4.0K    /usr/libx32
4.0K    /usr/lib32
76K     /usr/local
328M    /usr/lib
79M     /usr/bin
4.0K    /usr/games
567M    /usr
0       /dev/hugepages
0       /dev/log
0       /dev/initctl
0       /dev/core
0       /dev/shm
4.0K    /dev/.lxc-boot-id
0       /dev/stderr
0       /dev/stdout
0       /dev/stdin
0       /dev/fd
0       /dev/tty2
0       /dev/tty1
0       /dev/console
0       /dev/lxc
0       /dev/ptmx
0       /dev/zero
0       /dev/urandom
0       /dev/tty
0       /dev/random
0       /dev/null
0       /dev/full
0       /dev/.lxc
0       /dev/mqueue
0       /dev/pts
4.0K    /dev
0       /sys/kernel
0       /sys/power
0       /sys/class
0       /sys/devices
0       /sys/dev
0       /sys/hypervisor
0       /sys/fs
0       /sys/bus
0       /sys/firmware
0       /sys/block
0       /sys/module
0       /sys
4.0K    /etc/passwd-
4.0K    /etc/passwd
4.0K    /etc/request-key.conf
0       /etc/vtrgb
24K     /etc/samba
4.0K    /etc/gshadow
0       /etc/subuid
16K     /etc/udev
36K     /etc/sysctl.d
4.0K    /etc/crontab
4.0K    /etc/magic.mime
4.0K    /etc/rc1.d
20K     /etc/network
8.0K    /etc/ca-certificates.conf
8.0K    /etc/newt
4.0K    /etc/adduser.conf
8.0K    /etc/sudo.conf
4.0K    /etc/rc5.d
20K     /etc/X11
56K     /etc/security
4.0K    /etc/gshadow-
4.0K    /etc/legal
8.0K    /etc/manpath.config
8.0K    /etc/cron.yearly
0       /etc/.pwd.lock
4.0K    /etc/protocols
8.0K    /etc/ldap
20K     /etc/ppp
4.0K    /etc/nsswitch.conf
4.0K    /etc/issue
4.0K    /etc/group-
4.0K    /etc/issue.net
632K    /etc/ssl
8.0K    /etc/selinux
4.0K    /etc/rc2.d
4.0K    /etc/inputrc
56K     /etc/iproute2
4.0K    /etc/netplan
4.0K    /etc/rc3.d
4.0K    /etc/logrotate.conf
4.0K    /etc/gai.conf
12K     /etc/vim
4.0K    /etc/machine-id
28K     /etc/cron.daily
28K     /etc/update-motd.d
4.0K    /etc/libaudit.conf
4.0K    /etc/fstab
4.0K    /etc/nftables.conf
4.0K    /etc/rsyslog.conf
4.0K    /etc/sgml
0       /etc/rmt
4.0K    /etc/group
16K     /etc/skel
4.0K    /etc/debconf.conf
4.0K    /etc/hosts
4.0K    /etc/bash.bashrc
4.0K    /etc/nfs.conf.d
12K     /etc/ld.so.conf.d
4.0K    /etc/networks
12K     /etc/nanorc
16K     /etc/rsyslog.d
12K     /etc/kernel
8.0K    /etc/cron.monthly
4.0K    /etc/aliases
8.0K    /etc/wgetrc
616K    /etc/ssh
4.0K    /etc/appliance.info
4.0K    /etc/ethertypes
4.0K    /etc/host.conf
4.0K    /etc/modules
4.0K    /etc/zsh_command_not_found
44K     /etc/dhcp
12K     /etc/aliases.db
8.0K    /etc/depmod.d
4.0K    /etc/ld.so.conf
4.0K    /etc/shadow-
4.0K    /etc/e2scrub.conf
200K    /etc/console-setup
16K     /etc/request-key.d
4.0K    /etc/nfs.conf
20K     /etc/update-manager
8.0K    /etc/cron.hourly
4.0K    /etc/shells
12K     /etc/profile.d
4.0K    /etc/xattr.conf
40K     /etc/logrotate.d
4.0K    /etc/resolv.conf
8.0K    /etc/sudoers.d
4.0K    /etc/lsb-release
4.0K    /etc/deluser.conf
4.0K    /etc/profile
4.0K    /etc/hosts.deny
16K     /etc/apparmor.d
4.0K    /etc/rc0.d
4.0K    /etc/ucf.conf
0       /etc/subgid
16K     /etc/ubuntu-advantage
28K     /etc/modprobe.d
16K     /etc/services
8.0K    /etc/terminfo
12K     /etc/apparmor
4.0K    /etc/magic
4.0K    /etc/hosts.allow
4.0K    /etc/locale.alias
8.0K    /etc/gss
4.0K    /etc/tmpfiles.d
4.0K    /etc/modules-load.d
4.0K    /etc/opt
12K     /etc/groff
4.0K    /etc/timezone
4.0K    /etc/hostname
24K     /etc/dpkg
124K    /etc/postfix
4.0K    /etc/fuse.conf
24K     /etc/xdg
12K     /etc/insserv.conf.d
72K     /etc/libibverbs.d
8.0K    /etc/ca-certificates
12K     /etc/dbus-1
4.0K    /etc/rc6.d
4.0K    /etc/binfmt.d
12K     /etc/sudo_logsrvd.conf
4.0K    /etc/rpc
28K     /etc/networkd-dispatcher
8.0K    /etc/python3
4.0K    /etc/bash_completion
0       /etc/localtime
16K     /etc/ld.so.cache
64K     /etc/ufw
96K     /etc/init.d
0       /etc/.resolv.conf.systemd-resolved.bak
12K     /etc/libnl-3
12K     /etc/resolvconf
4.0K    /etc/rc4.d
12K     /etc/locale.gen
16K     /etc/logcheck
4.0K    /etc/netconfig
4.0K    /etc/mke2fs.conf
8.0K    /etc/wireguard
120K    /etc/systemd
8.0K    /etc/alternatives
4.0K    /etc/cifs-utils
4.0K    /etc/debian_version
0       /etc/mtab
28K     /etc/unpackerr
4.0K    /etc/shadow
4.0K    /etc/bindresvport.blacklist
4.0K    /etc/environment
4.0K    /etc/pam.conf
4.0K    /etc/keyutils
8.0K    /etc/python3.11
4.0K    /etc/idmapd.conf
80K     /etc/apt
60K     /etc/default
92K     /etc/pam.d
8.0K    /etc/hdparm.conf
4.0K    /etc/sudoers
12K     /etc/cron.weekly
16K     /etc/login.defs
0       /etc/os-release
4.0K    /etc/sysctl.conf
76K     /etc/mime.types
12K     /etc/cron.d
4.0K    /etc/rcS.d
3.6M    /etc
8.0K    /tmp/systemd-private-351553b6fe6a42fea74687d748ecbe18-systemd-logind.service-cwOCj5
4.0K    /tmp/.XIM-unix
4.0K    /tmp/.ICE-unix
4.0K    /tmp/.X11-unix
4.0K    /tmp/.font-unix
8.0K    /tmp/systemd-private-351553b6fe6a42fea74687d748ecbe18-systemd-resolved.service-BtaDz0
36K     /tmp
4.0K    /srv
0       /proc/fb
0       /proc/fs
0       /proc/bus
0       /proc/dma
0       /proc/irq
0       /proc/net
0       /proc/spl
4.0K    /proc/sys
0       /proc/tty
0       /proc/acpi
0       /proc/keys
0       /proc/kmsg
0       /proc/misc
0       /proc/mtrr
0       /proc/scsi
0       /proc/stat
0       /proc/iomem
0       /proc/kcore
0       /proc/locks
0       /proc/swaps
0       /proc/asound
0       /proc/crypto
0       /proc/driver
0       /proc/mdstat
0       /proc/mounts
0       /proc/uptime
0       /proc/vmstat
0       /proc/cgroups
0       /proc/cmdline
0       /proc/cpuinfo
0       /proc/devices
0       /proc/ioports
0       /proc/loadavg
0       /proc/meminfo
0       /proc/modules
0       /proc/sysvipc
0       /proc/version
0       /proc/consoles
0       /proc/kallsyms
0       /proc/pressure
0       /proc/slabinfo
0       /proc/softirqs
0       /proc/zoneinfo
0       /proc/buddyinfo
0       /proc/diskstats
0       /proc/key-users
0       /proc/schedstat
0       /proc/bootconfig
0       /proc/interrupts
0       /proc/kpagecount
0       /proc/kpageflags
0       /proc/partitions
0       /proc/timer_list
0       /proc/execdomains
0       /proc/filesystems
0       /proc/kpagecgroup
0       /proc/vmallocinfo
0       /proc/pagetypeinfo
0       /proc/dynamic_debug
0       /proc/sysrq-trigger
0       /proc/version_signature
0       /proc/self
0       /proc/thread-self
0       /proc/1
0       /proc/48
0       /proc/87
0       /proc/93
0       /proc/94
0       /proc/97
0       /proc/99
0       /proc/106
0       /proc/107
0       /proc/108
0       /proc/109
0       /proc/178
0       /proc/221
0       /proc/234
0       /proc/236
0       /proc/241
0       /proc/245
0       /proc/246
0       /proc/247
0       /proc/325
0       /proc/326
0       /proc/327
0       /proc/373
0       /proc/410
0       /proc/411
0       /proc/417
0       /proc/471
4.0K    /proc
0       /lib64
0       /sbin
50M     /var/log
20K     /var/tmp
780K    /var/spool
856K    /var/backups
4.0K    /var/opt
4.0K    /var/local
89M     /var/cache
0       /var/lock
0       /var/run
210M    /var/lib
4.0K    /var/mail
349M    /var
4.0K    /home
4.0K    /boot
4.0K    /media
16K     /lost+found
0       /libx32
4.0K    /opt
0       /lib32
4.0K    /.cache/qBittorrent
8.0K    /.cache
4.0K    /run/motd.dynamic
0       /run/agetty.reload
0       /run/crond.reboot
4.0K    /run/crond.pid
3.1M    /run/samba
4.0K    /run/sm-notify.pid
4.0K    /run/sshd.pid
0       /run/uuidd
0       /run/rpcbind.lock
0       /run/rpc_pipefs
4.0K    /run/utmp
0       /run/user
0       /run/sudo
0       /run/rpcbind
0       /run/sshd
0       /run/sendsigs.omit.d
0       /run/shm
0       /run/dbus
0       /run/console-setup
8.0M    /run/log
0       /run/mount
0       /run/initctl
0       /run/rpcbind.sock
0       /run/credentials
140K    /run/systemd
0       /run/lock
12M     /run
0       /lib
0       /bin
15M     /.local/share
15M     /.local
28K     /.config/qBittorrent
32K     /.config
4.0K    /root/.Donkeycredentials
16K     /root/.bash_history
4.0K    /root/.ssh
4.0K    /root/.lesshst
4.0K    /root/.HQcredentials
4.0K    /root/.bashrc
8.0K    /root/.cache
52K     /root/.local
28K     /root/.config
4.0K    /root/.profile
132K    /root
946M    /
946M    total
root@TorrentContainer:~#

It says only 1 GB is occupied right?

df -h

Code:
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/pve-vm--102--disk--0   20G   20G     0 100% /
none                              492K  4.0K  488K   1% /dev
tmpfs                             7.8G     0  7.8G   0% /dev/shm
tmpfs                             3.1G   12M  3.1G   1% /run
tmpfs                             5.0M     0  5.0M   0% /run/lock

lsof / | grep deleted
yields nothing!

what can I do?
 
I cannot run sudo du -a / | sort -n -r | head -n 20

because i get permission denied at /dev/.lxc/sys/foo and /dev/.lxc/proc/foo2 level

id yields
uid=0(root) gid=0(root) groups=0(root)

only until sys and proc I can read, nothing further than that
 
I am sorry, this is the 4th time that I start a thread and solve it by myself. Found the culprit, it was by human error because I ticked off a option in a program, of which, started to put incompleted files into /mnt in a hidden folder.
 
Can you tell me specifically how you solved this. I think I am running into exactly the same problem and beating my head over it.
 
Can you tell me specifically how you solved this. I think I am running into exactly the same problem and beating my head over it.

Went into qbittorrent-nox and changed path of temporary files from LXC to fileserver where they were before, I misclicked when configuring qbit-nox.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!