PVE 7: /var/lib/lxcfs/cgroup Input/output error and pve-root filled at 100%

[thommie]

A few days after upgrade from PVE 6.4 to 7.0-11 I see a strange message in the shell:

root@tokoeka ~ # du -hs / | sort -rh | head -5
du: cannot access '/var/lib/lxcfs/cgroup': Input/output error
du: cannot access '/proc/77889': No such file or directory
du: cannot access '/proc/77948/task/77948/fd/3': No such file or directory
du: cannot access '/proc/77948/task/77948/fdinfo/3': No such file or directory
du: cannot access '/proc/77948/fd/4': No such file or directory
du: cannot access '/proc/77948/fdinfo/4': No such file or directory

vms (both lxc and kvm) are running, but pve-root claims to be nearly full:

root@tokoeka ~ # df -h
Filesystem Size Used Avail Use% Mounted on
udev 32G 0 32G 0% /dev
tmpfs 6.3G 2.6M 6.3G 1% /run
/dev/mapper/pve-root 196G 187G 0 100% /
tmpfs 32G 37M 32G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/md0 488M 255M 203M 56% /boot
/dev/mapper/pve-backup 414G 73M 393G 1% /mnt/backup
/dev/mapper/pve-data 3.0T 386G 2.5T 14% /mnt/data
/dev/fuse 128M 44K 128M 1% /etc/pve
tmpfs 6.3G 0 6.3G 0% /run/user/1000

But there is still enough space on the host side. I dont remember the situation for 6.4, but I am pretty shure that the pve-root mountpoint was NOT filled at 100% ... The PVE is a single-node installation (without CEPH storage yet). Any ideas?

Thx, Thommie

 

[oguz]

hi,

I dont remember the situation for 6.4, but I am pretty shure that the pve-root mountpoint was NOT filled at 100%

that can still happen, files/disks/backups etc. being saved can take up space over time.

you can try like this to get an overview:

du -h / 2>/dev/null | sort -rh | head -n 20

and then inspect the directories to remove whatever you don't need. (make sure it's in the root filesystem of the machine and not in /mnt or so, pve-root won't free any space if you delete something that's mounted)

 

[thommie]

Hi Oguz, thanks for the hint, and I think i found the problem. For one of the lxc machines, the separate data partition is accidently located on the pve-root "local" storage instead of the separate mount-point for "data" on /mnt/data which is much bigger ;-(. Anyway, in the middle run, time for thinking about CEPH ;-)