VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - got timeout

gavinxiaozu

New Member
May 31, 2023
1
0
1
VM 101 is DSM synology, which on high loading casue pve panic.
I got this issue on pve 7.2.3 then I update it to 7.4, the issue still be here.


Jun 01 01:30:16 pve pvestatd[1977]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - got timeout Jun 01 01:30:16 pve kernel: ------------[ cut here ]------------ Jun 01 01:30:16 pve kernel: NETDEV WATCHDOG: enp3s0 (igc): transmit queue 0 timed out Jun 01 01:30:16 pve kernel: WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:477 dev_watchdog+0x277/0x280 Jun 01 01:30:16 pve kernel: Modules linked in: vfio_pci vfio_pci_core vfio_virqfd vfio_iommu_type1 vfio veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel nf_tables softdog bonding tls nfnetlink_log nfnetlink intel_rapl_msr mei_hdcp intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp snd_sof_pci_intel_icl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda kvm_intel kvm snd_sof_pci snd_sof_xtensa_dsp irqbypass snd_sof snd_soc_hdac_hda snd_hda_ext_core crct10dif_pclmul ghash_clmulni_intel aesni_intel snd_soc_acpi_intel_match crypto_simd cryptd snd_soc_acpi snd_hda_codec_hdmi soundwire_bus ledtrig_audio snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine intel_cstate snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec efi_pstore Jun 01 01:30:16 pve kernel: wmi_bmof pcspkr snd_hda_core ee1004 snd_hwdep snd_pcm snd_timer snd soundcore mei_me mei acpi_pad acpi_tad mac_hid zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost vhost_iotlb tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi msr parport_pc ppdev lp parport sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic xor zstd_compress raid6_pq simplefb dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c nvme i2c_i801 crc32_pclmul i2c_smbus i915 nvme_core i2c_algo_bit ttm igc drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci cec rc_core libahci xhci_pci xhci_pci_renesas sdhci_pci intel_lpss_pci cqhci intel_lpss drm xhci_hcd sdhci idma64 wmi video pinctrl_jasperlake Jun 01 01:30:16 pve kernel: CPU: 2 PID: 0 Comm: swapper/2 Tainted: P O 5.15.107-2-pve #1 Jun 01 01:30:16 pve kernel: Hardware name: maiyunda www.maiyunda.com/www.maiyunda.com, BIOS JK4LV102 07/03/2022 Jun 01 01:30:16 pve kernel: RIP: 0010:dev_watchdog+0x277/0x280 Jun 01 01:30:16 pve kernel: Code: eb 97 48 8b 5d d0 c6 05 e8 b6 4c 01 01 48 89 df e8 ae 50 f9 ff 44 89 e1 48 89 de 48 c7 c7 f0 ed 2a ab 48 89 c2 e8 1e 91 1c 00 <0f> 0b eb 80 e9 09 ea 25 00 0f 1f 44 00 00 55 49 89 ca 48 89 e5 41 Jun 01 01:30:16 pve kernel: RSP: 0018:ffffb4870019ce70 EFLAGS: 00010282 Jun 01 01:30:16 pve kernel: RAX: 0000000000000000 RBX: ffff8fa316100000 RCX: ffff8fa670120588 Jun 01 01:30:16 pve kernel: RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff8fa670120580 Jun 01 01:30:16 pve kernel: RBP: ffffb4870019cea8 R08: 0000000000000003 R09: 0000000000000001 Jun 01 01:30:16 pve kernel: R10: 0000000000ffff0a R11: 0000000000000001 R12: 0000000000000000 Jun 01 01:30:16 pve kernel: R13: ffff8fa31610a440 R14: 0000000000000004 R15: ffff8fa3161004c0 Jun 01 01:30:16 pve kernel: FS: 0000000000000000(0000) GS:ffff8fa670100000(0000) knlGS:0000000000000000 Jun 01 01:30:16 pve kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jun 01 01:30:16 pve kernel: CR2: 00007f0fe03706c8 CR3: 0000000111e1a000 CR4: 0000000000352ee0 Jun 01 01:30:16 pve kernel: Call Trace: Jun 01 01:30:16 pve kernel: <IRQ> Jun 01 01:30:16 pve kernel: ? pfifo_fast_enqueue+0x160/0x160 Jun 01 01:30:16 pve kernel: call_timer_fn+0x29/0x120 Jun 01 01:30:16 pve kernel: __run_timers.part.0+0x1e1/0x270 Jun 01 01:30:16 pve kernel: ? ktime_get+0x43/0xc0 Jun 01 01:30:16 pve kernel: ? lapic_next_deadline+0x2c/0x40 Jun 01 01:30:16 pve kernel: ? clockevents_program_event+0xa8/0x130 Jun 01 01:30:16 pve kernel: run_timer_softirq+0x2a/0x60 Jun 01 01:30:16 pve kernel: __do_softirq+0xd6/0x2ea Jun 01 01:30:16 pve kernel: irq_exit_rcu+0x94/0xc0 Jun 01 01:30:16 pve kernel: sysvec_apic_timer_interrupt+0x80/0x90 Jun 01 01:30:16 pve kernel: </IRQ> Jun 01 01:30:16 pve kernel: <TASK> Jun 01 01:30:16 pve kernel: asm_sysvec_apic_timer_interrupt+0x1b/0x20 Jun 01 01:30:16 pve kernel: RIP: 0010:cpuidle_enter_state+0xd9/0x620 Jun 01 01:30:16 pve kernel: Code: 3d 64 6c 9e 55 e8 f7 2a 6d ff 49 89 c7 0f 1f 44 00 00 31 ff e8 38 38 6d ff 80 7d d0 00 0f 85 5e 01 00 00 fb 66 0f 1f 44 00 00 <45> 85 f6 0f 88 6a 01 00 00 4d 63 ee 49 83 fd 09 0f 87 e5 03 00 00 Jun 01 01:30:16 pve kernel: RSP: 0018:ffffb48700123e38 EFLAGS: 00000246 Jun 01 01:30:16 pve kernel: RAX: ffff8fa670130bc0 RBX: ffffd486ffd27100 RCX: 0000000000000000 Jun 01 01:30:16 pve kernel: RDX: 000000000000008a RSI: 00000000401a41a4 RDI: 0000000000000000 Jun 01 01:30:16 pve kernel: RBP: ffffb48700123e88 R08: 000001120e681e14 R09: 00000000002ff940 Jun 01 01:30:16 pve kernel: R10: 0000000000000004 R11: 071c71c71c71c71c R12: ffffffffabad4d60 Jun 01 01:30:16 pve kernel: R13: 0000000000000002 R14: 0000000000000002 R15: 000001120e681e14 Jun 01 01:30:16 pve kernel: ? cpuidle_enter_state+0xc8/0x620 Jun 01 01:30:16 pve kernel: cpuidle_enter+0x2e/0x50 Jun 01 01:30:16 pve kernel: do_idle+0x20d/0x2b0 Jun 01 01:30:16 pve kernel: cpu_startup_entry+0x20/0x30 Jun 01 01:30:16 pve kernel: start_secondary+0x12a/0x180 Jun 01 01:30:16 pve kernel: secondary_startup_64_no_verify+0xc2/0xcb Jun 01 01:30:16 pve kernel: </TASK> Jun 01 01:30:16 pve kernel: ---[ end trace c75db5bb8e7f4f09 ]--- Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: Register Dump Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: Register Name Value Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: CTRL 181c0641 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: STATUS 40780683 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: CTRL_EXT 10000040 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: MDIC 1805dde1 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: ICR 000000c1 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: RCTL 0440803a Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: RDLEN[0-3] 00001000 00001000 00001000 00001000 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: RDH[0-3] 000000a8 00000084 0000006a 000000b8 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: RDT[0-3] 000000a8 00000083 00000069 000000b7 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: RXDCTL[0-3] 02040808 02040808 02040808 02040808 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: RDBAL[0-3] ffffb000 ffffa000 ffff9000 ffff8000 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: RDBAH[0-3] 00000000 00000000 00000000 00000000 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: TCTL a503f0fa Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: TDBAL[0-3] fffff000 ffffe000 ffffd000 ffffc000 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: TDBAH[0-3] 00000000 00000000 00000000 00000000 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: TDLEN[0-3] 00001000 00001000 00001000 00001000 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: TDH[0-3] 00000066 000000ac 0000002d 0000006b Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: TDT[0-3] 00000066 000000ac 0000002d 0000006b Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: TXDCTL[0-3] 02100108 02100108 02100108 02100108 Jun 01 01:30:16 pve kernel: igc 0000:03:00.0 enp3s0: Reset adapter Jun 01 01:30:17 pve kernel: igc 0000:03:00.0 enp3s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX Jun 01 01:30:28 pve kernel: watchdog: BUG: soft lockup - CPU#3 stuck for 26s! [kvm:3409] Jun 01 01:30:28 pve kernel: Modules linked in: vfio_pci vfio_pci_core vfio_virqfd vfio_iommu_type1 vfio veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel nf_tables softdog bonding tls nfnetlink_log nfnetlink intel_rapl_msr mei_hdcp intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp snd_sof_pci_intel_icl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda kvm_intel kvm snd_sof_pci snd_sof_xtensa_dsp irqbypass snd_sof snd_soc_hdac_hda snd_hda_ext_core crct10dif_pclmul ghash_clmulni_intel aesni_intel snd_soc_acpi_intel_match crypto_simd cryptd snd_soc_acpi snd_hda_codec_hdmi soundwire_bus ledtrig_audio snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine intel_cstate snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec efi_pstore Jun 01 01:30:28 pve kernel: wmi_bmof pcspkr snd_hda_core ee1004 snd_hwdep snd_pcm snd_timer snd soundcore mei_me mei acpi_pad acpi_tad mac_hid zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost vhost_iotlb tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi msr parport_pc ppdev lp parport sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic xor zstd_compress raid6_pq simplefb dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c nvme i2c_i801 crc32_pclmul i2c_smbus i915 nvme_core i2c_algo_bit ttm igc drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci cec rc_core libahci xhci_pci xhci_pci_renesas sdhci_pci intel_lpss_pci cqhci intel_lpss drm xhci_hcd sdhci idma64 wmi video pinctrl_jasperlake Jun 01 01:30:28 pve kernel: CPU: 3 PID: 3409 Comm: kvm Tainted: P W O 5.15.107-2-pve #1 Jun 01 01:30:28 pve kernel: Hardware name: maiyunda www.maiyunda.com/www.maiyunda.com, BIOS JK4LV102 07/03/2022 Jun 01 01:30:28 pve kernel: RIP: 0010:__synchronize_hardirq+0xbb/0xd0 Jun 01 01:30:28 pve kernel: Code: 41 83 e6 01 75 9f 48 8b 45 d0 65 48 2b 04 25 28 00 00 00 75 1a 48 83 c4 10 5b 41 5c 41 5d 41 5e 41 5f 5d c3 cc cc cc cc f3 90 <e9> 76 ff ff ff e8 5b 53 c4 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f Jun 01 01:30:28 pve kernel: RSP: 0018:ffffb48701743cb0 EFLAGS: 00000206 Jun 01 01:30:28 pve kernel: RAX: 000000003744a200 RBX: ffff8fa3001f82a4 RCX: ffffffffab1f6c35 Jun 01 01:30:28 pve kernel: RDX: 0000000000000000 RSI: 0000000000000246 RDI: ffff8fa3001f82a4 Jun 01 01:30:28 pve kernel: RBP: ffffb48701743ce8 R08: 0000000000000001 R09: 0000000000000001 Jun 01 01:30:28 pve kernel: R10: ffff8fa316051300 R11: ffff8fa309ab4cc0 R12: 0000000000000001 Jun 01 01:30:28 pve kernel: R13: ffff8fa3001f8228 R14: 0000000000000001 R15: ffff8fa3001f8200 Jun 01 01:30:28 pve kernel: FS: 00007f2e247631c0(0000) GS:ffff8fa670180000(0000) knlGS:0000000000000000 Jun 01 01:30:28 pve kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jun 01 01:30:28 pve kernel: CR2: 00007fa3b3d3fafc CR3: 000000010a59e000 CR4: 0000000000352ee0 Jun 01 01:30:28 pve kernel: Call Trace: Jun 01 01:30:28 pve kernel: <TASK> Jun 01 01:30:28 pve kernel: free_irq+0x127/0x370 Jun 01 01:30:28 pve kernel: vfio_pci_set_intx_trigger+0x1e7/0x330 [vfio_pci_core] Jun 01 01:30:28 pve kernel: ? vfio_pci_get_irq_count+0xab/0x120 [vfio_pci_core] Jun 01 01:30:28 pve kernel: vfio_pci_set_irqs_ioctl+0x38/0xe0 [vfio_pci_core] Jun 01 01:30:28 pve kernel: vfio_pci_core_ioctl+0x359/0x1150 [vfio_pci_core] Jun 01 01:30:28 pve kernel: ? vfio_pci_intx_unmask_handler+0xca/0x110 [vfio_pci_core] Jun 01 01:30:28 pve kernel: ? vfio_pci_set_intx_unmask+0x78/0x130 [vfio_pci_core] Jun 01 01:30:28 pve kernel: vfio_device_fops_unl_ioctl+0x1f/0x40 [vfio] Jun 01 01:30:28 pve kernel: __x64_sys_ioctl+0x92/0xd0 Jun 01 01:30:28 pve kernel: do_syscall_64+0x59/0xc0 Jun 01 01:31:56 pve kernel: watchdog: BUG: soft lockup - CPU#3 stuck for 108s! [kvm:3409] -- Reboot --
 

Attachments

  • pve-7.4_soft_lockup_watchdog.log.txt
    27.4 KB · Views: 0

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!