VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - got timeout

[gavinxiaozu]

VM 101 is DSM synology, which on high loading casue pve panic.
I got this issue on pve 7.2.3 then I update it to 7.4, the issue still be here.


Jun 01 01:30:16 pve pvestatd[1977]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - got timeout Jun 01 01:30:16 pve kernel: ------------[ cut here ]------------ Jun 01 01:30:16 pve kernel: NETDEV WATCHDOG: enp3s0 (igc): transmit queue 0 timed out Jun 01 01:30:16 pve kernel: WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:477 dev_watchdog+0x277/0x280 Jun 01 01:30:16 pve kernel: Modules linked in: vfio_pci vfio_pci_core vfio_virqfd vfio_iommu_type1 vfio veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel nf_tables softdog bonding tls nfnetlink_log nfnetlink intel_rapl_msr mei_hdcp intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp snd_sof_pci_intel_icl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda kvm_intel kvm snd_sof_pci snd_sof_xtensa_dsp irqbypass snd_sof snd_soc_hdac_hda snd_hda_ext_core crct10dif_pclmul ghash_clmulni_intel aesni_intel snd_soc_acpi_intel_match crypto_simd cryptd snd_soc_acpi snd_hda_codec_hdmi soundwire_bus ledtrig_audio snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine intel_cstate snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec efi_pstore Jun 01 01:30:16 pve kernel: wmi_bmof pcspkr snd_hda_core ee1004 snd_hwdep snd_pcm snd_timer snd soundcore mei_me mei acpi_pad acpi_tad mac_hid zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost vhost_iotlb tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi msr parport_pc ppdev lp parport sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic xor zstd_compress raid6_pq simplefb dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c nvme i2c_i801 crc32_pclmul i2c_smbus i915 nvme_core i2c_algo_bit ttm igc drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci cec rc_core libahci xhci_pci xhci_pci_renesas sdhci_pci intel_lpss_pci cqhci intel_lpss drm xhci_hcd sdhci idma64 wmi video pinctrl_jasperlake Jun 01 01:30:16 pve kernel: CPU: 2 PID: 0 Comm: swapper/2 Tainted: P O 5.15.107-2-pve #1 Jun 01 01:30:16 pve kernel: Hardware name: maiyunda www.maiyunda.com/www.maiyunda.com, BIOS JK4LV102 07/03/2022 Jun 01 01:30:16 pve kernel: RIP: 0010:dev_watchdog+0x277/0x280 Jun 01 01:30:16 pve kernel: Code: eb 97 48 8b 5d d0 c6 05 e8 b6 4c 01 01 48 89 df e8 ae 50 f9 ff 44 89 e1 48 89 de 48 c7 c7 f0 ed 2a ab 48 89 c2 e8 1e 91 1c 00 <0f> 0b eb 80 e9 09 ea 25 00 0f 1f 44 00 00 55 49 89 ca 48 89 e5 41 Jun 01 01:30:16 pve kernel: RSP: 0018:ffffb4870019ce70 EFLAGS: 00010282 Jun 01 01:30:16 pve kernel: RAX: 0000000000000000 RBX: ffff8fa316100000 RCX: ffff8fa670120588 Jun 01 01:30:16 pve kernel: RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff8fa670120580 Jun 01 01:30:16 pve kernel: RBP: ffffb4870019cea8 R08: 0000000000000003 R09: 0000000000000001 Jun 01 01:30:16 pve kernel: R10: 0000000000ffff0a R11: 0000000000000001 R12: 0000000000000000 Jun 01 01:30:16 pve kernel: R13: ffff8fa31610a440 R14: 0000000000000004 R15: ffff8fa3161004c0 Jun 01 01:30:16 pve kernel: FS: 0000000000000000(0000) GS:ffff8fa670100000(0000) knlGS:0000000000000000 Jun 01 01:30:16 pve kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jun 01 01:30:16 pve kernel: CR2: 00007f0fe03706c8 CR3: 0000000111e1a000 CR4: 0000000000352ee0 Jun 01 01:30:16 pve kernel: Call Trace: Jun 01 01:30:16 pve kernel: <IRQ> Jun 01 01:30:16 pve kernel: ? pfifo_fast_enqueue+0x160/0x160 Jun 01 01:30:16 pve kernel: call_timer_fn+0x29/0x120 Jun 01 01:30:16 pve kernel: __run_timers.part.0+0x1e1/0x270 Jun 01 01:30:16 pve kernel: ? ktime_get+0x43/0xc0 Jun 01 01:30:16 pve kernel: ? lapic_next_deadline+0x2c/0x40 Jun 01 01:30:16 pve kernel: ? clockevents_program_event+0xa8/0x130 Jun 01 01:30:16 pve kernel: run_timer_softirq+0x2a/0x60 Jun 01 01:30:16 pve kernel: __do_softirq+0xd6/0x2ea Jun 01 01:30:16 pve kernel: irq_exit_rcu+0x94/0xc0 Jun 01 01:30:16 pve kernel: sysvec_apic_timer_interrupt+0x80/0x90 Jun 01 01:30:16 pve kernel: </IRQ> Jun 01 01:30:16 pve kernel: <TASK> Jun 01 01:30:16 pve kernel: asm_sysvec_apic_timer_interrupt+0x1b/0x20 Jun 01 01:30:16 pve kernel: RIP: 0010:cpuidle_enter_state+0xd9/0x620 Jun 01 01:30:16 pve kernel: Code: 3d 64 6c 9e 55 e8 f7 2a 6d ff 49 89 c7 0f 1f 44 00 00 31 ff e8 38 38 6d ff 80 7d d0 00 0f 85 5e 01 00 00 fb 66 0f 1f 44 00 00 <45> 85 f6 0f 88 6a 01 00 00 4d 63 ee 49 83 fd 09 0f 87 e5 03 00 00 Jun 01 01:30:16 pve kernel: RSP: 0018:ffffb48700123e38 EFLAGS: 00000246 Jun 01 01:30:16 pve kernel: RAX: ffff8fa670130bc0 RBX: ffffd486ffd27100 RCX: 0000000000000000 Jun 01 01:30:16 pve kernel: RDX: 000000000000008a RSI: 00000000401a41a4 RDI: 0000000000000000 Jun 01 01:30:16 pve kernel: RBP: ffffb48700123e88 R08: 000001120e681e14 R09: 00000000002ff940 Jun 01 01:30:16 pve kernel: R10: 0000000000000004 R11: 071c71c71c71c71c R12: ffffffffabad4d60 Jun 01 01:30:16 pve kernel: R13: 0000000000000002 R14: 0000000000000002 R15: 000001120e681e14 Jun 01 01:30:16 pve kernel: ? cpuidle_enter_state+0xc8/0x620 Jun 01 01:30:16 pve kernel: cpuidle_enter+0x2e/0x50 Jun 01 01:30:16 pve kernel: do_idle+0x20d/0x2b0 Jun 01 01:30:16 pve kernel: cpu_startup_entry+0x20/0x30 Jun 01 01:30:16 pve kernel: start_secondary+0x12a/0x180 Jun 01 01:30:16 pve kernel: secondary_startup_64_no_verify+0xc2/0xcb Jun 01 01:30:16 pve kernel: </TASK> Jun 01 01:30:16 pve kernel: ---[ end trace c75db5bb8e7f4f09 ]--- Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: Register Dump Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: Register Name Value Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: CTRL 181c0641 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: STATUS 40780683 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: CTRL_EXT 10000040 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: MDIC 1805dde1 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: ICR 000000c1 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: RCTL 0440803a Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: RDLEN[0-3] 00001000 00001000 00001000 00001000 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: RDH[0-3] 000000a8 00000084 0000006a 000000b8 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: RDT[0-3] 000000a8 00000083 00000069 000000b7 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: RXDCTL[0-3] 02040808 02040808 02040808 02040808 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: RDBAL[0-3] ffffb000 ffffa000 ffff9000 ffff8000 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: RDBAH[0-3] 00000000 00000000 00000000 00000000 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: TCTL a503f0fa Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: TDBAL[0-3] fffff000 ffffe000 ffffd000 ffffc000 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: TDBAH[0-3] 00000000 00000000 00000000 00000000 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: TDLEN[0-3] 00001000 00001000 00001000 00001000 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: TDH[0-3] 00000066 000000ac 0000002d 0000006b Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: TDT[0-3] 00000066 000000ac 0000002d 0000006b Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: TXDCTL[0-3] 02100108 02100108 02100108 02100108 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: Reset adapter Jun 01 01:30:17 pve kernel: igc 0000:03:00.0 enp3s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX Jun 01 01:30:28 pve kernel: watchdog: BUG: soft lockup - CPU#3 stuck for 26s! [kvm:3409] Jun 01 01:30:28 pve kernel: Modules linked in: vfio_pci vfio_pci_core vfio_virqfd vfio_iommu_type1 vfio veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel nf_tables softdog bonding tls nfnetlink_log nfnetlink intel_rapl_msr mei_hdcp intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp snd_sof_pci_intel_icl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda kvm_intel kvm snd_sof_pci snd_sof_xtensa_dsp irqbypass snd_sof snd_soc_hdac_hda snd_hda_ext_core crct10dif_pclmul ghash_clmulni_intel aesni_intel snd_soc_acpi_intel_match crypto_simd cryptd snd_soc_acpi snd_hda_codec_hdmi soundwire_bus ledtrig_audio snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine intel_cstate snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec efi_pstore Jun 01 01:30:28 pve kernel: wmi_bmof pcspkr snd_hda_core ee1004 snd_hwdep snd_pcm snd_timer snd soundcore mei_me mei acpi_pad acpi_tad mac_hid zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost vhost_iotlb tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi msr parport_pc ppdev lp parport sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic xor zstd_compress raid6_pq simplefb dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c nvme i2c_i801 crc32_pclmul i2c_smbus i915 nvme_core i2c_algo_bit ttm igc drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci cec rc_core libahci xhci_pci xhci_pci_renesas sdhci_pci intel_lpss_pci cqhci intel_lpss drm xhci_hcd sdhci idma64 wmi video pinctrl_jasperlake Jun 01 01:30:28 pve kernel: CPU: 3 PID: 3409 Comm: kvm Tainted: P W O 5.15.107-2-pve #1 Jun 01 01:30:28 pve kernel: Hardware name: maiyunda www.maiyunda.com/www.maiyunda.com, BIOS JK4LV102 07/03/2022 Jun 01 01:30:28 pve kernel: RIP: 0010:__synchronize_hardirq+0xbb/0xd0 Jun 01 01:30:28 pve kernel: Code: 41 83 e6 01 75 9f 48 8b 45 d0 65 48 2b 04 25 28 00 00 00 75 1a 48 83 c4 10 5b 41 5c 41 5d 41 5e 41 5f 5d c3 cc cc cc cc f3 90 <e9> 76 ff ff ff e8 5b 53 c4 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f Jun 01 01:30:28 pve kernel: RSP: 0018:ffffb48701743cb0 EFLAGS: 00000206 Jun 01 01:30:28 pve kernel: RAX: 000000003744a200 RBX: ffff8fa3001f82a4 RCX: ffffffffab1f6c35 Jun 01 01:30:28 pve kernel: RDX: 0000000000000000 RSI: 0000000000000246 RDI: ffff8fa3001f82a4 Jun 01 01:30:28 pve kernel: RBP: ffffb48701743ce8 R08: 0000000000000001 R09: 0000000000000001 Jun 01 01:30:28 pve kernel: R10: ffff8fa316051300 R11: ffff8fa309ab4cc0 R12: 0000000000000001 Jun 01 01:30:28 pve kernel: R13: ffff8fa3001f8228 R14: 0000000000000001 R15: ffff8fa3001f8200 Jun 01 01:30:28 pve kernel: FS: 00007f2e247631c0(0000) GS:ffff8fa670180000(0000) knlGS:0000000000000000 Jun 01 01:30:28 pve kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jun 01 01:30:28 pve kernel: CR2: 00007fa3b3d3fafc CR3: 000000010a59e000 CR4: 0000000000352ee0 Jun 01 01:30:28 pve kernel: Call Trace: Jun 01 01:30:28 pve kernel: <TASK> Jun 01 01:30:28 pve kernel: free_irq+0x127/0x370 Jun 01 01:30:28 pve kernel: vfio_pci_set_intx_trigger+0x1e7/0x330 [vfio_pci_core] Jun 01 01:30:28 pve kernel: ? vfio_pci_get_irq_count+0xab/0x120 [vfio_pci_core] Jun 01 01:30:28 pve kernel: vfio_pci_set_irqs_ioctl+0x38/0xe0 [vfio_pci_core] Jun 01 01:30:28 pve kernel: vfio_pci_core_ioctl+0x359/0x1150 [vfio_pci_core] Jun 01 01:30:28 pve kernel: ? vfio_pci_intx_unmask_handler+0xca/0x110 [vfio_pci_core] Jun 01 01:30:28 pve kernel: ? vfio_pci_set_intx_unmask+0x78/0x130 [vfio_pci_core] Jun 01 01:30:28 pve kernel: vfio_device_fops_unl_ioctl+0x1f/0x40 [vfio] Jun 01 01:30:28 pve kernel: __x64_sys_ioctl+0x92/0xd0 Jun 01 01:30:28 pve kernel: do_syscall_64+0x59/0xc0 Jun 01 01:31:56 pve kernel: watchdog: BUG: soft lockup - CPU#3 stuck for 108s! [kvm:3409] -- Reboot --