Some VMs aren't booting after upgrade to 7.1

[rramalho]

Hi,

I'm using proxmox on my homelab for almost a year now. Yesterday I upgraded my two small hosts to 7.1-5 from 7.0-x.

A few minutes after the upgrade, I noticed that some of my VMs didn't respond. I checked the logs on both machines and got this:

Nov 19 23:33:37 bear pvestatd[1193]: VM 105 qmp command failed - VM 105 qmp command 'query-proxmox-support' failed - unable to connect to VM 105 qmp socket - timeout after 31 retries

This happens on both nodes and various types of VMs.

Can anyone help me? Thank you!

 

[rzirzi]

I have the same problem after Proxmox update to 7.1-5:
Nov 20 06:06:20 proxmox pvestatd[3419]: VM 104 qmp command failed - VM 104 qmp command 'query-proxmox-support' failed - unable to connect to VM 104 qmp socket - timeout after 31 retries Nov 20 06:06:20 proxmox pvestatd[3419]: status update time (6.045 seconds)
VM 104 is Windows 11 that was working well before update to 7.1-5 (it was 7.0.x).

 

[Polyphemus]

Same for me... If I reduce the number of vCPU's to one in my case, the VM boots...

 

[Supergamer]

I have exactly the same problem after Proxmox update to 7.1-5 with Windows 10 VMs, no backup works, no console, I can't even shut down the VMs cleanly. Only the Linux containers and the Linux VMs work normally

-- Journal begins at Fri 2021-10-15 17:58:48 CEST. --
Nov 20 10:33:16 pve1 pvestatd[1483]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 31 retries
Nov 20 10:33:19 pve1 pvestatd[1483]: VM 103 qmp command failed - VM 103 qmp command 'query-proxmox-support' failed - unable to connect to VM 103 qmp socket - timeout after 31 retries
Nov 20 10:33:19 pve1 pvestatd[1483]: status update time (12.077 seconds)
Nov 20 10:33:28 pve1 pvestatd[1483]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 31 retries
Nov 20 10:33:31 pve1 pvestatd[1483]: VM 103 qmp command failed - VM 103 qmp command 'query-proxmox-support' failed - unable to connect to VM 103 qmp socket - timeout after 31 retries
Nov 20 10:33:31 pve1 pvestatd[1483]: status update time (12.078 seconds)
Nov 20 10:33:32 pve1 pvedaemon[1512]: VM 103 qmp command failed - VM 103 qmp command 'query-proxmox-support' failed - unable to connect to VM 103 qmp socket - timeout after 31 retries
Nov 20 10:33:40 pve1 pvestatd[1483]: VM 103 qmp command failed - VM 103 qmp command 'query-proxmox-support' failed - unable to connect to VM 103 qmp socket - timeout after 31 retries
Nov 20 10:33:43 pve1 pvestatd[1483]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 31 retries
Nov 20 10:33:43 pve1 pvestatd[1483]: status update time (12.076 seconds)

Here is the config of one of the VMs that no longer works

root@pve1 ~ # qm config 101
boot: order=sata0;ide2;net0
cores: 4
ide2: local:iso/virtio-win.iso,media=cdrom,size=543390K
machine: pc-i440fx-5.2
memory: 16384
name: Win10VM1
net0: e1000=7A:AE:71:9A:AB:84,bridge=vmbr2,firewall=1
numa: 0
onboot: 1
ostype: win10
sata0: local:101/vm-101-disk-0.qcow2,size=128G
scsihw: virtio-scsi-pci
smbios1: uuid=1d869f3f-a5bd-4cba-9860-7a877058b0ee
sockets: 1
vga: virtio

vmgenid: afc79b2e-2be5-426a-9c33-25f391e45275

Does anyone have an idea how I can solve this problem or can I go back to the old Proxmox version or maybe reinstall the kernel?

Thank You

Best Regards
Pascal

 

[Supergamer]

Same for me... If I reduce the number of vCPU's to one in my case, the VM boots...

yes that worked for me for a short time. But now just close proxmox and then open it again and then try to connect to the console of the vm. That only works once when starting the VM and when it is running and you want to connect again later, vncproxy error occurs again

 

[Polyphemus]

Booting into the lastest kernel of Proxmox 7.0 solves it for me, but I do not know if there are dependencies. I don't know how to make that older kernel the default boot kernel...

 

[tom]

sata0: local:101/vm-101-disk-0.qcow2,size=128G

For sata, you Just change the "Async IO" from the new "io_uring" back to "threads"
(go the Hardware Tab of your VM and edit the hard disk setting, do not forget to enable the Advanced settings)

Generally, we do not recommend sata, use virtio scsi.

 

[rzirzi]

For sata, you Just change the "Async IO" from the new "io_uring" back to "threads"
(go the Hardware Tab of your VM and edit the hard disk setting, do not forget to enable the Advanced settings)
Generally, we do not recommend sata, use virtio scsi.

THANK YOU! I could confirm that resolves problem with Win11 VM freezing. So "io_uring" is not recommended for Win11? Why it's default Async IO now?

 

[Polyphemus]

That seems to solve my problem too

 

[Supergamer]

For sata, you Just change the "Async IO" from the new "io_uring" back to "threads"
(go the Hardware Tab of your VM and edit the hard disk setting, do not forget to enable the Advanced settings)

Generally, we do not recommend sata, use virtio scsi.

Thank you very much all errors are gone

 

[rramalho]

hi all again,

To be fair, we all should read the release annoucements carefully. To quote the release annoucement:

• VMs with SATA as disk bus may show issues when used with io_uring
  • For now this can be worked around by switching the disk's Async IO mode to threads (for cache = write back/through) or native (for cache = off, none or direct sync), respectively, in the Advanced options of the disk edit dialogue.
  • Note that SATA is really not the best choice for a VM's disk bus in general, rather use (VirtIO-) SCSI for best performance and feature set. For Windows VirtIO support is available through the Windows VirtIO Drivers

 

[sebcbien]

THANK YOU! I could confirm that resolves problem with Win11 VM freezing. So "io_uring" is not recommended for Win11? Why it's default Async IO now?

For sata, you Just change the "Async IO" from the new "io_uring" back to "threads"
(go the Hardware Tab of your VM and edit the hard disk setting, do not forget to enable the Advanced settings)

Generally, we do not recommend sata, use virtio scsi.

THIS solved my 2 days of nightmare of my win 2019 srv crashing after some hours of running.

I changed this on a virtio passtrough to a sata drive.

• virtio0: local-lvm:vm-107-disk-0,aio=threads,size=32G
• virtio2: /dev/disk/by-id/ata-ST8000DM004-2XXXXXXXXXX032,aio=threads,backup=0,size=7814026584K

thanks !

 

[sebcbien]

For sata, you Just change the "Async IO" from the new "io_uring" back to "threads"
(go the Hardware Tab of your VM and edit the hard disk setting, do not forget to enable the Advanced settings)

Generally, we do not recommend sata, use virtio scsi.

@tom Thanks for the tip of recommending virtio scsi, it's what I do, but I use passthrough for one sata disk to my Windows VM.
At the beginning I didn't find that it was that drive not mounting that freezed the (rarely used video file server) windows 2019 srv.
The whole server was crashing.




I guess the way I did is the only one that I can use ?
Without specifying Async IO with proxmox 6.x it was very stable.

 

[jacorall]

A solution is to set Grub to boot with an earlier version of the kernel (kernel 5.11.22-7-pve). I've tested this on a number of Proxmox installations:

Ensure the the kernel 5.11.22-7-pve is present, by running update-grub, and see if it is present in the output on the screen.

Edit the /etc/default/grub:
nano /etc/default/grub

Comment the line:
#GRUB_DEFAULT=0

Add the line:
GRUB_DEFAULT="Advanced options for Proxmox VE GNU/Linux>Proxmox VE GNU/Linux, with Linux 5.11.22-7-pve"

Save the file.

Run grub-update config

Reboot the server.

The Proxmox server will run on a slightly older kernel.

 

[jacorall]

Kernel 5.15 seems to have solved the issue. You can install the latest version of kernel 5.15:

apt install pve-kernel-5.15

make sure that the following line is in the /etc/default/grub file:
GRUB_DEFAULT=0

Run: grub-update config

reboot the server