VM with old kernel 2.6.26 hangs on PVE 8.1.4

[Kurgan]

I'm trying to debug this issue: I have an old Debian VM with kernel 2.6.26 (i386) than has been converted from physical to virtual on a PVE 7.x temporary installation. It worked flawlessly on that PVE until it was moved to a new machine with pve-manager/8.1.4/ec5affc9e41f1d79 (running kernel: 6.5.11-7-pve).

The VM uses 4 GB of RAM, standard vCPU configuration and an LSI 53C895A (again standard) emulated SCSI controller because the VM kernel does not support Virtio-SCSI (it supports the older Virtio controller).

On the new PVE 8.1, the VM hangs with lots of these errors:

BUG: soft lockup - CPU#1 stuck for 96s! [apache2:14807]
BUG: soft lockup - CPU#0 stuck for 96s! [swapper:0]
BUG: soft lockup - CPU#3 stuck for 96s! [rsyslogd:14853]

When it happens, it seems that it's the disk i/o that dies. No more i/o AT ALL. Cannot read, cannot write. Power cycle the VM and it works again fine.

Since it's something that happens with the virtual disk (at least in my opinion) I have tried switching from LSI 53C895A to Virtio (not virtio-scsi) but the issue persists.

The virtual disk is configured as default: no cache, iothread enabled, Async IO is default (io_uring).

I/O on the host seems to be fine, no errors on the host log, the other VMs (modern Debian) work just fine. The disks are 2 NVME disks in raid1 (mdadm) with LVM on top

The issues are not clearly related to anything (not to backups, for example). I/O on the whole host is quite low, iodelay is under 5% even at the (rare) peaks, and it's usually under 0,5%.

Any ideas?

Thanks a lot.

 

[fiona]

Hi,
please make sure you are running the latest updates. There was a fix for a similar-sounding issue addressed in pve-qemu-kvm >= 8.1.5-2

 

[Kurgan]

Hi,
please make sure you are running the latest updates. There was a fix for a similar-sounding issue addressed in pve-qemu-kvm >= 8.1.5-2

I have pve-qemu-kvm Version: 8.1.5-5 on my problematic server, so it should not be that specific issue.

I have an older 8.1.2-4 on another server that has no issues at all...

 

[fiona]

I have pve-qemu-kvm Version: 8.1.5-5 on my problematic server, so it should not be that specific issue.

Please share the VM configuration file qm config <ID>. You could try turning off iothread, to see if the issue is related to that.

I have an older 8.1.2-4 on another server that has no issues at all...

With the same VM?

 

[Kurgan]

No, with different VMs. The same VM was fine on a 7.x server which I cannot reach right now (it's switced off and not here).

This is the config of the problematic VM. I have just switched off iothread, I will see if the issue is still there as soon as it crashes again. It was enabled until right now.

root@pve01 ~ # qm config  100
balloon: 0
boot: order=virtio0;ide2
cores: 4
cpu: x86-64-v2-AES
ide2: none,media=cdrom
memory: 4096
meta: creation-qemu=8.1.2,ctime=1705511420
name: anu
net0: virtio=00:50:56:15:14:18,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsihw: lsi
smbios1: uuid=6986a6f7-dffd-475f-a87b-d2de6a8e292e
sockets: 1
tablet: 0
virtio0: local-lvm:vm-100-disk-0,size=235G
vmgenid: d33f2eba-ce10-42cd-9642-1fce7ce9b8cc