All VMs locking up after latest PVE update

I am also affected by this.

5 nodes cluster, all fully updated to latest versions. Storage is on NFS.
Only one node seems to be affected.

# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 40
On-line CPU(s) list: 0-39
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6210U CPU @ 2.50GHz
Stepping: 7
CPU MHz: 1000.117
BogoMIPS: 5000.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 28160K
NUMA node0 CPU(s): 0-39
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke avx512_vnni md_clear flush_l1d arch_capabilities
Click to expand...


# dmidecode -t bios
# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 3.2.0 present.

Handle 0x0000, DMI type 0, 26 bytes
BIOS Information
Vendor: Dell Inc.
Version: 2.10.0
Release Date: 11/12/2020
Address: 0xF0000
Runtime Size: 64 kB
ROM Size: 32 MB
Characteristics:
ISA is supported
PCI is supported
PNP is supported
BIOS is upgradeable
BIOS shadowing is allowed
Boot from CD is supported
Selectable boot is supported
EDD is supported
Japanese floppy for Toshiba 1.2 MB is supported (int 13h)
5.25"/360 kB floppy services are supported (int 13h)
5.25"/1.2 MB floppy services are supported (int 13h)
3.5"/720 kB floppy services are supported (int 13h)
8042 keyboard services are supported (int 9h)
Serial services are supported (int 14h)
CGA/mono video services are supported (int 10h)
ACPI is supported
USB legacy is supported
BIOS boot specification is supported
Function key-initiated network boot is supported
Targeted content distribution is supported
UEFI is supported
BIOS Revision: 2.10

Handle 0x0D00, DMI type 13, 22 bytes
BIOS Language Information
Language Description Format: Long
Installable Languages: 1
en|US|iso8859-1
Currently Installed Language: en|US|iso8859-1
Click to expand...

# pveversion -v
proxmox-ve: 6.3-1 (running kernel: 5.4.103-1-pve)
pve-manager: 6.3-6 (running version: 6.3-6/2184247e)
pve-kernel-5.4: 6.3-7
pve-kernel-helper: 6.3-7
pve-kernel-5.4.103-1-pve: 5.4.103-1
pve-kernel-5.4.101-1-pve: 5.4.101-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.0-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.7
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-5
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-7
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
openvswitch-switch: 2.12.3-1
proxmox-backup-client: 1.0.10-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-6
pve-cluster: 6.2-1
pve-container: 3.3-4
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.2.0-3
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-8
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.3-pve2
Click to expand...

# dpkg -l | grep -E "pve-kernel-5.4 |pve-qemu-kvm|libproxmox-backup|pve-manager"
ii libproxmox-backup-qemu0 1.0.3-1 amd64 Proxmox Backup Server client library for QEMU
ii pve-kernel-5.4 6.3-7 all Latest Proxmox VE Kernel Image
ii pve-manager 6.3-6 amd64 Proxmox Virtual Environment Management Tools
ii pve-qemu-kvm 5.2.0-3 amd64 Full virtualization on x86 hardware
ii pve-qemu-kvm-dbg 5.2.0-3 amd64 pve qemu debugging symbols
Click to expand...

# uname -a
Linux node1 5.4.103-1-pve #1 SMP PVE 5.4.103-1 (Sun, 07 Mar 2021 15:55:09 +0100) x86_64 GNU/Linux
Click to expand...

The VMs are freezed, no VNC-able, no network, no clean shutdown capable. I was able to hard-stop (not great...) and migrate them to another node.

The problem appeared after the last cluster dist-upgrade. The backup task kick-in and immediately after messages like those ones in the system's log (for several VMs, _stopped_ and running) and then the concerned VMs crashes:

pvestatd[1946]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - got timeout
pvestatd[1946]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 31 retries
Click to expand...

The VMs are all linux (centos, debian, archlinux).
 
Last edited:
LazyGatto said:
No. In my case it may happens in a completely random way with random VM while backup/restore, i.e.
Click to expand...

Hi! :D

I understand those VMs don't use iothreads? We don't use Promxox backup or restore so I don't know if those actions could still be a trigger.

I mean this problem seems to be related because we get almost the same error messages and it only happened after upgrading to the latest version:

pvedaemon[48698]: <root@pam> update VM 139: resize --disk scsi2 --size +16G
kernel: rbd2: detected capacity change from 34359738368 to 51539607552
pvedaemon[48698]: VM 139 qmp command failed - VM 139 qmp command 'block_resize' failed - got timeout
pvedaemon[34945]: VM 139 qmp command failed - VM 139 qmp command 'query-proxmox-support' failed - got timeout
pvestatd[4874]: VM 139 qmp command failed - VM 139 qmp command 'query-proxmox-support' failed - got timeout
pvestatd[4874]: status update time (6.291 seconds)
pvestatd[4874]: VM 139 qmp command failed - VM 139 qmp command 'query-proxmox-support' failed - got timeout

And we lose access to the affected virtual machine.

Today by testing I confirmed that disabling the iothreads no longer reproduces our problem so I understand that they may be the cause.

Regards
 
Last edited:
Hi,

same problem here. 9 node cluster with NFS backend. Several VMs get this qmp command 'query-proxmox-support' failed - Error after doing snapshots. Need to kill them on the host to get them online again. No network, no console. We don't have iothreads enabled but see this problem, too. Disabling snapshots solved it for us so far.

Regards.
 
Thank you very much for all of the information provided! I was finally able to reproduce the issue, it appears to be a race-condition in the QMP monitor code introduced in QEMU 5.2.

I have sent a potential fix to the upstream qemu-devel mailing list, since this touches a rather critical part of the code we want to discuss on there if this is the right idea before shipping it in our pve-qemu-kvm builds.
 
  • Like
Reactions: DerDanilo, meichthys, vaschthestampede and 4 others
Thank you for the follow up.
What is the recommended temporary workaround please ?
 
kfx said:
What is the recommended temporary workaround please ?
Click to expand...
As stated before, downgrading pve-qemu-kvm for the time being:

apt install pve-qemu-kvm=5.1.0-8 libproxmox-backup-qemu0=1.0.2-1
 
We also hit this bug and i downgraded kvm and libproxmox-backup qemu some days ago, no problems since then.
Do you think I can update libproxmox-backup to a current version and just leave pve-qemu-kvm downgraded ?
I ask because there was another bug fixed a few days ago that makes incremental backups a lot smaller in some conditions. I would like to take advantage of the bugfix in proxmox-backup.
 
Stefan_R said:
As stated before, downgrading pve-qemu-kvm for the time being:

apt install pve-qemu-kvm=5.1.0-8 libproxmox-backup-qemu0=1.0.2-1
Click to expand...

OK thanks.
Is there a way to do so without the need to reboot all the VMs (as we can't live migrate to a node with a downgraded qemu) ?
 
adoII said:
Do you think I can update libproxmox-backup to a current version and just leave pve-qemu-kvm downgraded ?
Click to expand...
No, that's a hard dependency. AFAIK the bug you mention only affected containers though.

kfx said:
Is there a way to do so without the need to reboot all the VMs (as we can't live migrate to a node with a downgraded qemu) ?
Click to expand...
Suspend/resume could work, but not guaranteed for a downgrade (depends on your machine version). I'd rather just restart the VMs.
 
adoII said:
I was talking about this bug where data was backed up again although it did not change.
https://forum.proxmox.com/threads/p...ackup-time-after-some-days.85703/#post-376286
Is it possible to change some apt packages to get this bugfix and still keep kvm on 5.1 and libproxmox-backup on 1.0.2 ?
Click to expand...
Yes, this is a bug in proxmox-backup-client and thus totally unrelated to pve-qemu-kvm and libproxmox-backup0 - it only affects containers and host backups, and you will get the fix with the next backup client version.
 
We also encountered this problem on our test cluster today after running some test PBS backups and now downgraded and rebooted all VMs.
Thank you for debugging and investigating!
 
Hi,
Great to hear that you have been able to reproduce the error.

I Don't know if you still need these backtraces but here are few that I was able to take today.

http://shell.systec.cloud/proxmox-debug/

.txt file name includes a small description. Those VMs kept freezing even without the backup process.
 
We also had to downgrade two clusters due this issue. I hope you can release a patch asap.

Regards
 
We were affected too. 5 node cluster running ceph on servers about 18 months old. 600TB of spinners.

I've disabled Proxmox backups to NFS and a PBS until the issue is resolved as it appeared to have been trigger during a backup.
 
Looks like I ran into the same problem. But it only occured when doing snapshots. Storage is ZFS and looks like downgrading fixed this too.

I described it here.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back