[Proxmox 7.2-3 - CEPH 16.2.7] Migrating VMs hangs them (kernel panic on Linux, freeze on Windows)

[qXt69WEV2a7fgbET]

I haven't debugged the issue, but from what other users mention, I'm afraid that is the case.


If you don't have any problems with the new kernel, I'd recommend staying on that one. While 5.15 is the default kernel right now, the opt-in 5.19 also gets a similar amount of attention.

Thank you very much! My problem has been resolved by installing the newer kernel.
The systems are running without any problems for some weeks now.

 

[bodo]

I can also confirm live migration is working between nodes running kernel 5.19.7-1
live migration is also working between nodes running kernel 5.15.39-4
live migration is working fine if we migrate a VM from a node running 5.19.7-1 to a node running 5.15.39-4

However, live migration does not work (VM freezes and eats CPU cycles) if we migrate a VM from a node running 5.15.39-4 to a node running 5.19.7-1

Is there a safe way to upgrade a cluster from 5.15.39-4 to 5.19.x without VM freezes/downtime?

BTW: we have tested both, local storage and ceph storage.

We were able to live migrate all VMs from nodes running 5.15.39-4 to nodes running 5.19.17-1. However, it is a one way ticket, we haven't been able to live migrate back to 5.15.39-4 - but we also don't want to

With kernel 5.19.17-1 live migration also works between nodes with different CPU types (Gold, Xeon v4, ...), no matter if the VM had been started on Gold or Xeon v4.

Thanks for the good work!

 

[Tomasz Jeliński]

Hello,
Problem with live migration and VM hangs (100% CPU utilization) still exist in OPT-IN pve-kernel-6.2 (pve-kernel-6.2.6-1-pve) and pve-kernel-5.19 (tested on 5.19.17-2-pve and 5.19.17-1-pve. On pve-kernel-6.2 similar call trace sometimes doesn't appear in dmesg log but also kvm kernel module is the reason of hang.

Call trace #pve-kernel 5.19.17-1-pve

[  259.910077] ------------[ cut here ]------------
[  259.912265] WARNING: CPU: 26 PID: 4468 at arch/x86/kvm/x86.c:10627 kvm_arch_vcpu_ioctl_run+0x1cb0/0x1df0 [kvm]
[  259.914604] Modules linked in: rpcsec_gss_krb5 nfsv4 nfsv3 nfs fscache netfs ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter sctp ip6_udp_tunnel udp_tunnel ses enclosure dm_round_robin scsi_transport_sas nf_tables nfnetlink_cttimeout openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c softdog nfnetlink_log nfnetlink dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common isst_if_common skx_edac nfit x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd mgag200 ipmi_ssif drm_shmem_helper drm_kms_helper fb_sys_fops dell_wmi ledtrig_audio cmdlinepart rapl syscopyarea sparse_keymap spi_nor sysfillrect video cdc_ether mei_me sysimgblt dell_smbios usbnet mii dcdbas mei wmi_bmof dell_wmi_descriptor intel_cstate pcspkr mtd efi_pstore
[  259.914802]  intel_pch_thermal acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler mac_hid acpi_power_meter zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost vhost_iotlb tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd drm nfs_acl lockd auth_rpcgss grace sunrpc ip_tables x_tables autofs4 simplefb crc32_pclmul ixgbe igb i2c_algo_bit xhci_pci xfrm_algo xhci_pci_renesas i2c_i801 spi_intel_pci dca bnxt_en mdio ahci megaraid_sas tg3 lpc_ich spi_intel i2c_smbus libahci xhci_hcd wmi
[  259.930718] CPU: 26 PID: 4468 Comm: CPU 1/KVM Tainted: P          IO      5.19.17-1-pve #1
[  259.931390] Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 2.17.1 11/17/2022
[  259.932040] RIP: 0010:kvm_arch_vcpu_ioctl_run+0x1cb0/0x1df0 [kvm]
[  259.932737] Code: 48 8b 78 08 4c 89 e6 e8 5e a9 fc ff 65 ff 0d 57 4f df 3e 0f 85 a2 ec ff ff 0f 1f 44 00 00 e9 d0 e9 ff ff 0f 0b e9 4f ed ff ff <0f> 0b e9 2c ed ff ff 0f 1f 44 00 00 e9 23 f5 ff ff 48 89 de 4c 89
[  259.934061] RSP: 0018:ffffb8f48f687d00 EFLAGS: 00010202
[  259.934713] RAX: 0000000000000001 RBX: 000000000000ae80 RCX: 0000000000000000
[  259.935360] RDX: 0000380fbee154c0 RSI: 00000000fffffe01 RDI: ffffa0c60dd54000
[  259.935993] RBP: ffffb8f48f687da0 R08: 0000000000000001 R09: 000000000000001a
[  259.936602] R10: 0000000000000000 R11: 0000000000000000 R12: ffffa0c60dd54000
[  259.937232] R13: ffffa0c59dff5000 R14: ffffa0c60dd54048 R15: ffffa0c5cd7c0500
[  259.937857] FS:  00007f2b8ffff700(0000) GS:ffffa0d4bff40000(0000) knlGS:0000000000000000
[  259.938514] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  259.939131] CR2: 0000561e7bc98b10 CR3: 000000018e6c6006 CR4: 00000000007726e0
[  259.939744] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  259.940354] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  259.940976] PKRU: 55555554
[  259.941583] Call Trace:
[  259.942166]  <TASK>
[  259.942743]  ? kvm_arch_vcpu_put+0x1a9/0x200 [kvm]
[  259.943375]  ? vcpu_put+0x22/0x50 [kvm]
[  259.943961]  ? kvm_arch_vcpu_ioctl_run+0x721/0x1df0 [kvm]
[  259.944578]  ? wake_up_q+0x50/0x90
[  259.945162]  kvm_vcpu_ioctl+0x24f/0x6d0 [kvm]
[  259.945783]  ? kvm_vcpu_ioctl+0x2b8/0x6d0 [kvm]
[  259.946394]  ? __fget_light.part.0+0x8c/0xd0
[  259.946966]  __x64_sys_ioctl+0x92/0xd0
[  259.947494]  do_syscall_64+0x59/0x90
[  259.948004]  ? syscall_exit_to_user_mode+0x26/0x50
[  259.948501]  ? do_syscall_64+0x69/0x90
[  259.948990]  ? irqentry_exit_to_user_mode+0x9/0x20
[  259.949524]  ? irqentry_exit+0x3b/0x50
[  259.950035]  ? exc_page_fault+0x87/0x180
[  259.950523]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  259.950980] RIP: 0033:0x7f2ba08a35f7
[  259.951435] Code: 00 00 00 48 8b 05 99 c8 0d 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 69 c8 0d 00 f7 d8 64 89 01 48
[  259.952308] RSP: 002b:00007f2b8fffa408 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[  259.952741] RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f2ba08a35f7
[  259.953214] RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000020
[  259.953683] RBP: 0000561e7c0c2750 R08: 0000561e7a42d240 R09: 0000561e7ab31940
[  259.954143] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[  259.954601] R13: 0000561e7ab38020 R14: 00007f2b8fffa6c0 R15: 0000000000802000
[  259.955012]  </TASK>
[  259.955395] ---[ end trace 0000000000000000 ]---

CPU installed on host

processor    : 39
vendor_id    : GenuineIntel
cpu family    : 6
model        : 85
model name    : Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
stepping    : 4
microcode    : 0x2006e05
cpu MHz        : 2200.000
cache size    : 14080 KB
physical id    : 1
siblings    : 20
core id        : 10
cpu cores    : 10
apicid        : 53
initial apicid    : 53
fpu        : yes
fpu_exception    : yes
cpuid level    : 22
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke md_clear flush_l1d arch_capabilities
vmx flags    : vnmi preemption_timer posted_intr invvpid ept_x_only ept_ad ept_1gb flexpriority apicv tsc_offset vtpr mtf vapic ept vpid unrestricted_guest vapic_reg vid ple shadow_vmcs pml ept_mode_based_exec tsc_scaling
bugs        : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit mmio_stale_data retbleed
bogomips    : 4401.72
clflush size    : 64
cache_alignment    : 64
address sizes    : 46 bits physical, 48 bits virtual
power management:

From my personal tests:
* Migrating VM from host2 (pve-kernel-5.15) to host1 (pve-kernel-6.2) - VM HANGS (100%CPU/ dmesg kvm call trace)
* After stop/start VM (hang) on pve-kernel-6.2 host1 and live migration from host1 (pve-kernel-6.2) to host2 (pve-kernel-5.15) - OK
* Migrating VM host3(pve-kernel-6.2, cpu without tsc_scaling) -> host4 (pve-kernel-5.15/ pve-kernel-6.2 without tsc_scaling) and reverse - OK

PVE hosts DELL Poweredge R440 are up-to-date (FW/ BIOS etc). CPUs installed on hosts have tsc_scaling flag.
Migration between hosts with installed pve-kernel-5.15.102-1-pve works properly. Problem appear only if I migrate from host with pve-kernel-5.15 to pve-kernel-6.2/5.19.
Problem with live migration doesn't exist on host with CPU without tsc_scaling flag and installed new OPT-IN kernel 6.2 and any combination with old pve-kernel-5.15. That was tested on old legacy servers and CPUs without tsc_scaling (HP Gen DL380P Gen 8)

I would like to upgrade kernel 5.15 to new OPT-IN kernel 6.2 (new features) without downtime on production. Unfortunately I can't because only my production hosts affected by this bug sharing external VM storage (direct connection).
I can't live migrate VM to old hosts without tsc_scaling flag
Is this a regression in new PVE kernels (5.19/6.2) ?

Regards
--
Tomasz

 

[qXt69WEV2a7fgbET]

Thank you for your post, i will stay on 5.19 for now then.
Have you upgraded to the newer opt in kernel for any reason? (eg. this migration issue also occurring on 5.19)

 

[Tomasz Jeliński]

Thank you for your post, i will stay on 5.19 for now then.
Have you upgraded to the newer opt in kernel for any reason? (eg. this migration issue also occurring on 5.19)

I don't have possibility to check behavior on pve-kernel-5.19.x hosts with tsc_scaling CPUs without downtime. Maybe live migration from hosts on pve-kernel-5.19 to pve-kernel-6.2 is not affected and live migration between pve-kernel-6.2 hosts also. I'm "stuck" on pve-kernel-5.15 line.

PS.
Reason to upgrade are performance improvements in 6.1/6.2 on btrfs/ext4 and Skylake Xeon CPU, kernels compiled with DYNAMIC_PREEAMPTION, better support for hardware.