fuse: error creating thread: No space left on device

[dmitryf]

Hi,

My Proxmox 5.2 installation works really strange.
DELL PowerEdge™ R730 DX291 in Hetzner
18 LXC containers

I see a lot of "No space left on device" errors in syslog, like:

lxcfs[1584]: fuse: error creating thread: No space left on device
pvestatd[2161]: command 'lxc-info -n 114 -p' failed: open3: fork failed: No space left on device at /usr/share/perl5/PVE/Tools.pm line 429.

Of course there is a lot of free space on the disks:

Spoiler: df -h

Filesystem Size Used Avail Use% Mounted on
udev 63G 0 63G 0% /dev
tmpfs 13G 44M 13G 1% /run
/dev/mapper/pve-root 94G 8.3G 81G 10% /
tmpfs 63G 40M 63G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 63G 0 63G 0% /sys/fs/cgroup
/dev/sda1 1.8T 129G 1.6T 8% /var/lib/vz
/dev/fuse 30M 32K 30M 1% /etc/pve
tmpfs 13G 0 13G 0% /run/user/0

VM's and Proxmox works very unstable. For example Cassandra in one lxc container goes down with "java.lang.OutOfMemoryError: unable to create new native thread" error.

As far as I understand it's some "limits" issue but I can't figure out the real reason. After reboot all works well about a few hours and then "no space left on device" error come back.
I will be very grateful for the help
Some additional info:

Spoiler: ulimit -a

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 515190
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 655350
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 655350
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Spoiler: pveversion --verbose

proxmox-ve: 5.2-2 (running kernel: 4.15.18-5-pve)
pve-manager: 5.2-9 (running version: 5.2-9/4b30e8f9)
pve-kernel-4.15: 5.2-8
pve-kernel-4.15.18-5-pve: 4.15.18-24
pve-kernel-4.15.18-1-pve: 4.15.18-19
pve-kernel-4.15.17-3-pve: 4.15.17-14
pve-kernel-4.13.13-2-pve: 4.13.13-33
corosync: 2.4.2-pve5
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-40
libpve-guest-common-perl: 2.0-18
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-29
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-2
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-20
pve-cluster: 5.0-30
pve-container: 2.0-27
pve-docs: 5.2-8
pve-firewall: 3.0-14
pve-firmware: 2.0-5
pve-ha-manager: 2.0-5
pve-i18n: 1.0-6
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.2-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-36
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.11-pve1~bpo1

 

[Knuuut]

Maybe you're running out of inodes.

Check this:

df -i

 

[dmitryf]

Yeah forgot to write about it:

Spoiler: df -i

Filesystem Inodes IUsed IFree IUse% Mounted on
udev 16486094 627 16485467 1% /dev
tmpfs 16491330 1326 16490004 1% /run
/dev/mapper/pve-root 6291456 73077 6218379 2% /
tmpfs 16491330 79 16491251 1% /dev/shm
tmpfs 16491330 32 16491298 1% /run/lock
tmpfs 16491330 17 16491313 1% /sys/fs/cgroup
/dev/sda1 122101760 118 122101642 1% /var/lib/vz
/dev/fuse 10000 49 9951 1% /etc/pve
tmpfs 16491330 11 16491319 1% /run/user/0