Host hangs up after 12-24 hours: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior.

[siheikki]

Hello!

I'm a new Proxmox VE user. I moved my Home assistant (latest HA OS) to the Intel NUC 11 hardware and Promox VE platform (latest) a moment ago. I have a Zigbee ZZH! and a Google Coral AI USB sticks attached to the hardware.

The home assistant is the only vm running currently and it hangs up in a short period of time. I tried to google and found that SCSI device async=threads could help but it did not.

Already tested without help:

1. threads=1 (was already) async=threads setting for SCSI.
2. Disabling memory balloon.

Here is console screenshot. I'm not actually not sure what is relevant in the screenshot.

 

[Moayad]

Hello,

Did you see anything interesting in the Syslog during the VM get hangs from the Proxmox VE side?

Can you please provide us the pveversion -v output and the VM config (qm config <VMID>) as well?

 

[siheikki]

Hello!

Latest full hang happen an hour ago. I attached few screenshots from that also. Some syslog printing also, did not see those before I think.

pveversion -v

root@intelnuc:~# pveversion -v
proxmox-ve: 7.3-1 (running kernel: 5.15.83-1-pve)
pve-manager: 7.3-4 (running version: 7.3-4/d69b70d4)
pve-kernel-helper: 7.3-3
pve-kernel-5.15: 7.3-1
pve-kernel-5.15.83-1-pve: 5.15.83-1
pve-kernel-5.15.74-1-pve: 5.15.74-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.3
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.3-1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-2
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.3-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-1
lxcfs: 5.0.3-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.3.2-1
proxmox-backup-file-restore: 2.3.2-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.3
pve-cluster: 7.3-2
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.6-3
pve-ha-manager: 3.5.1
pve-i18n: 2.8-2
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1
root@intelnuc:~#

qm config 100

root@intelnuc:~# qm config 100
agent: 1
balloon: 2000
bios: ovmf
boot: order=scsi0
cores: 4
efidisk0: local-lvm:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
memory: 8000
meta: creation-qemu=7.1.0,ctime=1675441284
name: homeassistant
net0: virtio=AE:AF:06:3E:1D:12,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
parent: after_hd_change
scsi0: local-lvm:vm-100-disk-1,size=300G
scsihw: virtio-scsi-pci
smbios1: uuid=aa23275f-b9aa-47d9-95d3-a01739b6cdbe
sockets: 1
usb2: host=2-2
usb3: host=1-1
usb4: host=1-8
vmgenid: 2b64a334-69e6-4ce2-9775-acdf4df7d0c7
root@intelnuc:~#

 

[Moayad]

Hello,

Thank you for the output!

From the output, the first thing I noticed is that there is an OOM, I would give the VM fewer resources and test the behavior.

 

[siheikki]

Do you mean fewer memory or fewer USB devices?

Only less than 2GB of memory is used, the rest is used for VM's OS cache.

 

[Moayad]

I mean give the VM less memory RAM instead of 8 GB try to give it 6 GB e.g.

 

[siheikki]

This seems to be issue with Proxmox kernel versions etc. https://github.com/home-assistant/operating-system/issues/2342 .

Testing now updating the kernel to 6.X version and configuring host SCSI to single thread mode.

 

[rcrack2k]

If using Default (LSI 53C895A) as SCSI Controller Type, i also get the error message on trying to install DEBIAN v11 via NetInstall.

ISO-Image: https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.6.0-amd64-netinst.iso