Memory Hotplug Breaking threads-max in VM

[mike p]

Hi All.

We Have a Debian 10 installation with 8GiB of RAM. Memory Hot-plugging is enabled as explained in wiki (Both udev rules and kernel parameter are set). The Server is Used as an Ingress in Kubernetes cluster and uses nginx. Under High Load, we discovered nginx is failing to fork new threads for new connections due to reaching max allowed threads (Our Post regarding the threads-max). Digging deeper, we found that while booting the kernel is discovering only 1GiB of Memory and therefor setting a very Low amount of max allowed threads (7055):

root@deb:~>>> dmesg | grep Memory:

[    0.282335] Memory: 903820K/1048032K available (10252K kernel code, 1242K rwdata, 3328K rodata, 1600K init, 2260K bss, 144212K reserved, 0K cma-reserved)
root@deb:~>>> cat /proc/sys/kernel/threads-max
7055

However the Memory is recognized completely by VM when booted and Hot-Plugging is working seamlessly.

Disabling the Memory Hotplug Solves the problem all together:

root@deb:~>>> dmesg | grep Memory:
[    1.156223] Memory: 8129300K/8388064K available (10252K kernel code, 1242K rwdata, 3328K rodata, 1600K init, 2260K bss, 258764K reserved, 0K cma-reserved)

root@deb:~>>> cat /proc/sys/kernel/threads-max
63510

Any Help is super Welcomed

 

[Telencephalon]

Hi! I observed the exact same issue with Ubuntu 22.04 as the VM guest. In my case, this low value of max-threads had the effect that certain parallel computation backends such as Matlab's parpool / parcluster feature failed to start up a large number of parallel workers. Disabling memory hotplug and restarting the VMs fixed this issue. We don't need memory hotplug, but it would nevertheless be good to fix this issue. I'd be happy to provide more debug information.

pveversion -v
proxmox-ve: 7.4-1 (running kernel: 5.15.104-1-pve)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.15: 7.4-1
pve-kernel-5.4: 6.4-20
pve-kernel-5.0: 6.0-11
pve-kernel-5.15.104-1-pve: 5.15.104-1
pve-kernel-5.4.203-1-pve: 5.4.203-1
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph: 15.2.17-pve1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
gfs2-utils: 3.3.0-2
glusterfs-client: 9.2-1
ifupdown: 0.8.36+pve2
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-4
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-1
libpve-rs-perl: 0.7.5
libpve-storage-perl: 7.4-2
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.1-1
proxmox-backup-file-restore: 2.4.1-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.1-1
proxmox-widget-toolkit: 3.6.5
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-2
pve-firewall: 4.3-1
pve-firmware: 3.6-4
pve-ha-manager: 3.6.0
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.4-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1

 

[jaredhendrickson13]

Running into the same issue on Ubuntu 24.04 with memory hot plugging enabled. No matter the amount of memory assigned, the kernel only sees ~1GB during boot throwing off the threads-max. Thought maybe it had to do with ballooning, but disabling the ballooning device had no effect. Disabling memory hot plugging is the only thing I've found that fixes the issue altogether.

For now, I was able to work around this by manually setting kernel.threads-max sysctl to a higher value and storing that value in an /etc/sysctl.d/ file on my VMs so it persists across reboots. This works for the most part but certainly isn't ideal. Really hoping to find a solution for this!