VM Won't Start After Upgrade to 9.0.3

[agarlow]

making new thread for comment here: https://forum.proxmox.com/threads/proxmox-virtual-environment-9-0-released.169258/post-788981 @fiona and @dcsapak

I just upgraded to 9.0.3 from 8.4 and its fine except for one troublesome Fedora VM.

Sometimes it refuses to start with "TASK ERROR: timeout waiting on systemd" and sometimes it appears to start, but trying to get into the console I get "error: failed to run vncproxy" and I can't ssh into the VM, none of the containers on the VM start, and it's pinned at 100% memory usage.

The problem VM has a PCIe device passed through but so is another perfectly functional VM on the node, regardless I tried not passing through any devices and that didn't help. I've toggled memory ballooning, changed the display type, restored from a backup, but so far no success.

VM config:
balloon: 0
bios: ovmf
boot: order=scsi0
cores: 14
cpu: host
efidisk0: ProxZFS:vm-101-disk-2,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:06:00
machine: q35
memory: 32768
meta: creation-qemu=9.2.0,ctime=1752219122
name: wharf
net0: virtio=BC:24:11:99:4F:88,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: ProxZFS:vm-101-disk-3,iothread=1,size=80G
scsihw: virtio-scsi-single
smbios1: uuid=2c5ce4da-81ed-42ea-a2e4-c7b5ad62672a
sockets: 1
vmgenid: 4c4f921b-c489-4d97-a15b-268eb0c373b4


journal output of trying to start:

1:26:39 proxmox2 pvedaemon[343903]: VM 101 started with PID 343915.
Aug 06 21:26:40 proxmox2 pvedaemon[248749]: <root@pam> end task UPIDroxmox2:00053F5F:0066ABF1:689400CA:qmstart:101:root@pam: OK
Aug 06 21:26:57 proxmox2 pvestatd[2627]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeo>
Aug 06 21:26:57 proxmox2 pvedaemon[248751]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - ti>
Aug 06 21:27:02 proxmox2 pvedaemon[248749]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - ti>
Aug 06 21:27:02 proxmox2 pvedaemon[344235]: starting vnc proxy UPIDroxmox2:000540AB:0066B6F7:689400E6:vncproxy:101:root@pam:
Aug 06 21:26:35 proxmox2 pvedaemon[248749]: readline() on closed filehandle $smaps_fh at /usr/share/perl5/PVE/QemuServer.pm line 2433.
Aug 06 21:26:39 proxmox2 kernel: vfio-pci 0000:06:00.0: resetting
Aug 06 21:26:39 proxmox2 kernel: vfio-pci 0000:06:00.0: reset done
Aug 06 21:26:39 proxmox2 kernel: vfio-pci 0000:06:00.0: resetting
Aug 06 21:26:39 proxmox2 kernel: vfio-pci 0000:06:00.0: reset done
Aug 06 21:26:39 proxmox2 pvedaemon[343903]: VM 101 started with PID 343915.
Aug 06 21:26:40 proxmox2 pvedaemon[248749]: <root@pam> end task UPIDroxmox2:00053F5F:0066ABF1:689400CA:qmstart:101:root@pam: OK
Aug 06 21:26:57 proxmox2 pvestatd[2627]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 51 retries
Aug 06 21:26:57 proxmox2 pvedaemon[248751]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 51 retries
Aug 06 21:27:02 proxmox2 pvedaemon[248749]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 51 retries
Aug 06 21:27:02 proxmox2 pvedaemon[344235]: starting vnc proxy UPIDroxmox2:000540AB:0066B6F7:689400E6:vncproxy:101:root@pam:
Aug 06 21:27:02 proxmox2 pvedaemon[248749]: <root@pam> starting task UPIDroxmox2:000540AB:0066B6F7:689400E6:vncproxy:101:root@pam:
Aug 06 21:27:02 proxmox2 pvestatd[2627]: status update time (13.452 seconds)
Aug 06 21:27:08 proxmox2 qm[344237]: VM 101 qmp command failed - VM 101 qmp command 'set_password' failed - unable to connect to VM 101 qmp socket - timeout after 51 retries
Aug 06 21:27:08 proxmox2 pvedaemon[344235]: Failed to run vncproxy.
Aug 06 21:27:08 proxmox2 pvedaemon[248749]: <root@pam> end task UPIDroxmox2:000540AB:0066B6F7:689400E6:vncproxy:101:root@pam: Failed to run vncproxy.
Aug 06 21:27:11 proxmox2 pvestatd[2627]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 51 retries
Aug 06 21:27:15 proxmox2 pvestatd[2627]: status update time (12.932 seconds)
Aug 06 21:27:24 proxmox2 pvedaemon[248749]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 51 retries
Aug 06 21:27:24 proxmox2 pvestatd[2627]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 51 retries
Aug 06 21:27:28 proxmox2 pvedaemon[344353]: stop VM 101: UPIDroxmox2:00054121:0066C0ED:68940100:qmstop:101:root@pam:
Aug 06 21:27:28 proxmox2 pvedaemon[248749]: <root@pam> starting task UPIDroxmox2:00054121:0066C0ED:68940100:qmstop:101:root@pam:
Aug 06 21:27:28 proxmox2 pvestatd[2627]: status update time (13.131 seconds)
Aug 06 21:27:33 proxmox2 pvedaemon[344353]: VM 101 qmp command failed - VM 101 qmp command 'quit' failed - unable to connect to VM 101 qmp socket - timeout after 51 retries
Aug 06 21:27:33 proxmox2 pvedaemon[344353]: VM quit/powerdown failed - terminating now with SIGTERM
Aug 06 21:27:37 proxmox2 pvestatd[2627]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 51 retries
Aug 06 21:27:42 proxmox2 pvestatd[2627]: status update time (13.229 seconds)
Aug 06 21:27:43 proxmox2 pvedaemon[344353]: VM still running - terminating now with SIGKILL
Aug 06 21:27:44 proxmox2 pvedaemon[248749]: <root@pam> end task UPIDroxmox2:00054121:0066C0ED:68940100:qmstop:101:root@pam: OK
Aug 06 21:27:44 proxmox2 pvestatd[2627]: VM 101 qmp command failed - VM 101 not running
Aug 06 21:27:44 proxmox2 pvedaemon[248751]: VM 101 qmp command failed - VM 101 not running
Aug 06 21:27:48 proxmox2 pvestatd[2627]: status update time (6.015 seconds)
Aug 06 21:27:54 proxmox2 pvedaemon[344510]: start VM 101: UPIDroxmox2:000541BE:0066CB29:6894011A:qmstart:101:root@pam:
Aug 06 21:27:54 proxmox2 pvedaemon[248749]: <root@pam> starting task UPIDroxmox2:000541BE:0066CB29:6894011A:qmstart:101:root@pam:
Aug 06 21:27:54 proxmox2 kernel: vfio-pci 0000:06:00.0: resetting
Aug 06 21:27:54 proxmox2 kernel: vfio-pci 0000:06:00.0: reset done
Aug 06 21:27:54 proxmox2 systemd[1]: 101.scope: Deactivated successfully.
Aug 06 21:27:54 proxmox2 systemd[1]: Stopped 101.scope.
Aug 06 21:27:54 proxmox2 systemd[1]: 101.scope: Consumed 1min 19.550s CPU time, 32.1G memory peak.
Aug 06 21:28:14 proxmox2 pvedaemon[344510]: timeout waiting on systemd
Aug 06 21:28:14 proxmox2 pvedaemon[248749]: <root@pam> end task UPIDroxmox2:000541BE:0066CB29:6894011A:qmstart:101:root@pam: timeout waiting on systemd

 

[fiona]

Hi,
how does the memory usage on your host look like? Since you are using ZFS, please also see: https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysadmin_zfs_limit_memory_usage

What does zpool status -v say? What does pvesm status say?

Please run apt install pve-qemu-kvm-dbgsym gdb and then, when the VM is in the stuck state,
gdb --batch --ex 't a a bt' -p $(cat /var/run/qemu-server/101.pid) to obtain a backtrace. What does the subtree in ps faxl for the VM look like when it's in the stuck state?

 

[agarlow]

I moronically tried rebooting.
Zpool status now says "ZFS modules cannot be auto-loaded"
Modprobe ZFS says "Fatal: module ZFS not found in /lib/modules/6.14.8-2-pve despite me looking at a folder named ZFS in that directory. Rebooting into an old kernel pre-update didn't help.

Better news, one of those now inoperable VMs is pfSense so I'm relegated to whatever I can download on a phone and offload to a thumb drive. No apt installing anything for now.

I tried to restore just the pfsense VM from a backup, forcing it to use local-lvm instead of Zfs, and now I'm getting "KVM Virtualization configured but not available. Either disable in VM configuration (I have no idea how to do this) or enable in BIOS (I checked, virtualization and IOMMU are enabled)

 

[fiona]

Please share the full system journal for the current boot journalctl -b > /tmp/boot.txt.

 

[fiona]

Do you have the latest BIOS updates/CPU microcode installed? https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysadmin_firmware_cpu

Please also share the output of lscpu on the host.

Rebooting into an old kernel pre-update didn't help.

What kernel version was this?

 

[ZeratuLx]

first try to download 6.8.12-13-pve, it helped me