Hi Guys
Every time we try to use a debian container in proxmox, the console of container is blackscreen.
We try to stop it or anything else, the host crash and reboot.
We used ovs switch and migrate to linux bond but same issue.
Don't crash with centos9 CT for exemple.
Environnement :
Proxmox 7.3.3
Kernel version 5.15.74-1-pve #1 SMP PVE 5.15.74-1
Below the syslog :
Jan 6 19:47:31 proxmox2 kernel: [ 5753.661066] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [swapper/0:0]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.661773] Modules linked in: veth rbd libceph ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter input_leds joyd
ev hid_generic usbmouse usbkbd usbhid hid sctp ip6_udp_tunnel udp_tunnel nf_tables nfnetlink_cttimeout bonding tls openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 softdog nfne
tlink_log nfnetlink intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul ghash_clmulni_intel aesni_intel mgag200 drm_kms_helper ce
c ipmi_ssif rc_core crypto_simd cryptd i2c_algo_bit cdc_ether fb_sys_fops syscopyarea rapl usbnet sysfillrect mei_me sysimgblt intel_cstate mii efi_pstore mei pcspkr ipmi_si mxm_wmi ipmi_devintf ipmi_msghan
dler acpi_power_meter acpi_pad mac_hid vhost_net vhost vhost_iotlb tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi drm sunrpc ip_tables x_tables autofs4 zfs(PO)
Jan 6 19:47:31 proxmox2 kernel: [ 5753.661824] zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs blake2b_generic xor zstd_compress raid6_pq libcrc32c simplefb crc32_pclmu
l xhci_pci xhci_pci_renesas i2c_i801 i2c_smbus lpc_ich ehci_pci tg3 xhci_hcd ehci_hcd ixgbe megaraid_sas xfrm_algo dca mdio wmi
Jan 6 19:47:31 proxmox2 kernel: [ 5753.669917] CPU: 0 PID: 0 Comm: swapper/0 Tainted: P O 5.15.74-1-pve #1
Jan 6 19:47:31 proxmox2 kernel: [ 5753.670694] Hardware name: LENOVO System x3650 M5: -[5462AC1]-/00YJ423, BIOS -[TCE158A-3.80]- 09/14/2022
Jan 6 19:47:31 proxmox2 kernel: [ 5753.671475] RIP: 0010:netdev_pick_tx+0xdb/0x310
Jan 6 19:47:31 proxmox2 kernel: [ 5753.672263] Code: 01 00 00 41 0f b7 46 7c 66 85 c0 0f 84 53 01 00 00 8d 48 ff 0f b7 c1 66 39 ca 0f 86 dd 01 00 00 39 c3 0f 87 55 01 00 00 29 d8 <39> c3 0f 87 4b 01 00 00
eb f4 0f 1f 44 00 00 49 8b 94 24 28 04 00
Jan 6 19:47:31 proxmox2 kernel: [ 5753.674009] RSP: 0018:ffffa42900003608 EFLAGS: 00000246
Jan 6 19:47:31 proxmox2 kernel: [ 5753.674873] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000057
Jan 6 19:47:31 proxmox2 kernel: [ 5753.675690] RDX: 0000000000000000 RSI: ffff96efc136e100 RDI: ffff96fa9e607000
Jan 6 19:47:31 proxmox2 kernel: [ 5753.676495] RBP: ffffa42900003648 R08: 0000000000000000 R09: ffff96efc136ef00
Jan 6 19:47:31 proxmox2 kernel: [ 5753.677360] R10: ffffa42900003958 R11: ffffa42900003988 R12: ffff96fa9e607000
Jan 6 19:47:31 proxmox2 kernel: [ 5753.678163] R13: 00000000ffffffff R14: ffff96efc136e100 R15: ffff96fa9e607000
Jan 6 19:47:31 proxmox2 kernel: [ 5753.678948] FS: 0000000000000000(0000) GS:ffff973230000000(0000) knlGS:0000000000000000
Jan 6 19:47:31 proxmox2 kernel: [ 5753.679736] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 6 19:47:31 proxmox2 kernel: [ 5753.680516] CR2: 000055da41e3bc80 CR3: 0000001d03c10006 CR4: 00000000001706f0
Jan 6 19:47:31 proxmox2 kernel: [ 5753.681376] Call Trace:
Jan 6 19:47:31 proxmox2 kernel: [ 5753.682182] <IRQ>
Jan 6 19:47:31 proxmox2 kernel: [ 5753.682962] ? common_interrupt+0x55/0xa0
Jan 6 19:47:31 proxmox2 kernel: [ 5753.683734] netdev_core_pick_tx+0xa4/0xb0
Jan 6 19:47:31 proxmox2 kernel: [ 5753.684488] __dev_queue_xmit+0x1b8/0xb30
Jan 6 19:47:31 proxmox2 kernel: [ 5753.685282] ? netif_rx_internal+0x3a/0x100
Jan 6 19:47:31 proxmox2 kernel: [ 5753.686052] dev_queue_xmit+0x10/0x20
Jan 6 19:47:31 proxmox2 kernel: [ 5753.686801] ovs_vport_send+0xae/0x170 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.687561] do_output+0x59/0x180 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.688277] do_execute_actions+0x1841/0x1b40 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.688993] ? __skb_flow_dissect+0x2f2/0x1920
Jan 6 19:47:31 proxmox2 kernel: [ 5753.689790] ? flow_lookup.constprop.0+0xa1/0x110 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.690496] ovs_execute_actions+0x48/0x110 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.691188] ? ovs_execute_actions+0x48/0x110 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.691866] ovs_dp_process_packet+0xa1/0x1f0 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.692579] ? ovs_ct_update_key.isra.0+0xa8/0x120 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.693303] ? ovs_ct_fill_key+0x1d/0x30 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.694001] ? ovs_flow_key_extract+0x2da/0x350 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.694669] ovs_vport_receive+0x77/0xd0 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.695380] ? tcp_ack_update_rtt+0x12f/0x450
Jan 6 19:47:31 proxmox2 kernel: [ 5753.696049] ? kfree_skbmem+0x6f/0xa0
Jan 6 19:47:31 proxmox2 kernel: [ 5753.696723] ? cpumask_next_and+0x24/0x30
Jan 6 19:47:31 proxmox2 kernel: [ 5753.697470] netdev_frame_hook+0xdf/0x1b0 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.698121] ? netdev_create+0x40/0x40 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.698757] __netif_receive_skb_core+0x238/0xef0
Jan 6 19:47:31 proxmox2 kernel: [ 5753.699373] __netif_receive_skb_list_core+0x107/0x260
Jan 6 19:47:31 proxmox2 kernel: [ 5753.699972] netif_receive_skb_list_internal+0x1a1/0x2c0
Jan 6 19:47:31 proxmox2 kernel: [ 5753.700557] ? dev_gro_receive+0x2d6/0x7b0
Jan 6 19:47:31 proxmox2 kernel: [ 5753.701148] ? kmem_cache_alloc+0x1ab/0x2f0
Jan 6 19:47:31 proxmox2 kernel: [ 5753.701746] ? __build_skb+0x26/0x60
Jan 6 19:47:31 proxmox2 kernel: [ 5753.702275] napi_complete_done+0x7a/0x1c0
Jan 6 19:47:31 proxmox2 kernel: [ 5753.702790] ixgbe_poll+0xe19/0x1240 [ixgbe]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.703299] ? tick_nohz_irq_exit+0x3b/0x50
Jan 6 19:47:31 proxmox2 kernel: [ 5753.703784] __napi_poll+0x33/0x180
Jan 6 19:47:31 proxmox2 kernel: [ 5753.704257] net_rx_action+0x126/0x280
Jan 6 19:47:31 proxmox2 kernel: [ 5753.704719] __do_softirq+0xd9/0x2ea
Jan 6 19:47:31 proxmox2 kernel: [ 5753.705201] irq_exit_rcu+0x94/0xc0
Jan 6 19:47:31 proxmox2 kernel: [ 5753.705686] common_interrupt+0x8e/0xa0
Jan 6 19:47:31 proxmox2 kernel: [ 5753.706102] </IRQ>
Jan 6 19:47:31 proxmox2 kernel: [ 5753.706502] <TASK>
Jan 6 19:47:31 proxmox2 kernel: [ 5753.706891] asm_common_interrupt+0x27/0x40
Jan 6 19:47:31 proxmox2 kernel: [ 5753.707286] RIP: 0010:cpuidle_enter_state+0xd9/0x620
Jan 6 19:47:31 proxmox2 kernel: [ 5753.707683] Code: 3d 64 64 9f 6c e8 27 24 6e ff 49 89 c7 0f 1f 44 00 00 31 ff e8 68 31 6e ff 80 7d d0 00 0f 85 5e 01 00 00 fb 66 0f 1f 44 00 00 <45> 85 f6 0f 88 6a 01 00
00 4d 63 ee 49 83 fd 09 0f 87 e5 03 00 00
Jan 6 19:47:31 proxmox2 kernel: [ 5753.708534] RSP: 0018:ffffffff94803da0 EFLAGS: 00000246
Jan 6 19:47:31 proxmox2 kernel: [ 5753.708962] RAX: ffff973230030bc0 RBX: ffffc428ff802900 RCX: 0000000000000000
Jan 6 19:47:31 proxmox2 kernel: [ 5753.709465] RDX: 00000000000161bf RSI: 0000000037a6f9a0 RDI: 0000000000000000
Jan 6 19:47:31 proxmox2 kernel: [ 5753.709892] RBP: ffffffff94803df0 R08: 000005351fb3c3f9 R09: 0000000000061a80
Every time we try to use a debian container in proxmox, the console of container is blackscreen.
We try to stop it or anything else, the host crash and reboot.
We used ovs switch and migrate to linux bond but same issue.
Don't crash with centos9 CT for exemple.
Environnement :
Proxmox 7.3.3
Kernel version 5.15.74-1-pve #1 SMP PVE 5.15.74-1
Below the syslog :
Jan 6 19:47:31 proxmox2 kernel: [ 5753.661066] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [swapper/0:0]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.661773] Modules linked in: veth rbd libceph ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter input_leds joyd
ev hid_generic usbmouse usbkbd usbhid hid sctp ip6_udp_tunnel udp_tunnel nf_tables nfnetlink_cttimeout bonding tls openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 softdog nfne
tlink_log nfnetlink intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul ghash_clmulni_intel aesni_intel mgag200 drm_kms_helper ce
c ipmi_ssif rc_core crypto_simd cryptd i2c_algo_bit cdc_ether fb_sys_fops syscopyarea rapl usbnet sysfillrect mei_me sysimgblt intel_cstate mii efi_pstore mei pcspkr ipmi_si mxm_wmi ipmi_devintf ipmi_msghan
dler acpi_power_meter acpi_pad mac_hid vhost_net vhost vhost_iotlb tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi drm sunrpc ip_tables x_tables autofs4 zfs(PO)
Jan 6 19:47:31 proxmox2 kernel: [ 5753.661824] zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs blake2b_generic xor zstd_compress raid6_pq libcrc32c simplefb crc32_pclmu
l xhci_pci xhci_pci_renesas i2c_i801 i2c_smbus lpc_ich ehci_pci tg3 xhci_hcd ehci_hcd ixgbe megaraid_sas xfrm_algo dca mdio wmi
Jan 6 19:47:31 proxmox2 kernel: [ 5753.669917] CPU: 0 PID: 0 Comm: swapper/0 Tainted: P O 5.15.74-1-pve #1
Jan 6 19:47:31 proxmox2 kernel: [ 5753.670694] Hardware name: LENOVO System x3650 M5: -[5462AC1]-/00YJ423, BIOS -[TCE158A-3.80]- 09/14/2022
Jan 6 19:47:31 proxmox2 kernel: [ 5753.671475] RIP: 0010:netdev_pick_tx+0xdb/0x310
Jan 6 19:47:31 proxmox2 kernel: [ 5753.672263] Code: 01 00 00 41 0f b7 46 7c 66 85 c0 0f 84 53 01 00 00 8d 48 ff 0f b7 c1 66 39 ca 0f 86 dd 01 00 00 39 c3 0f 87 55 01 00 00 29 d8 <39> c3 0f 87 4b 01 00 00
eb f4 0f 1f 44 00 00 49 8b 94 24 28 04 00
Jan 6 19:47:31 proxmox2 kernel: [ 5753.674009] RSP: 0018:ffffa42900003608 EFLAGS: 00000246
Jan 6 19:47:31 proxmox2 kernel: [ 5753.674873] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000057
Jan 6 19:47:31 proxmox2 kernel: [ 5753.675690] RDX: 0000000000000000 RSI: ffff96efc136e100 RDI: ffff96fa9e607000
Jan 6 19:47:31 proxmox2 kernel: [ 5753.676495] RBP: ffffa42900003648 R08: 0000000000000000 R09: ffff96efc136ef00
Jan 6 19:47:31 proxmox2 kernel: [ 5753.677360] R10: ffffa42900003958 R11: ffffa42900003988 R12: ffff96fa9e607000
Jan 6 19:47:31 proxmox2 kernel: [ 5753.678163] R13: 00000000ffffffff R14: ffff96efc136e100 R15: ffff96fa9e607000
Jan 6 19:47:31 proxmox2 kernel: [ 5753.678948] FS: 0000000000000000(0000) GS:ffff973230000000(0000) knlGS:0000000000000000
Jan 6 19:47:31 proxmox2 kernel: [ 5753.679736] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 6 19:47:31 proxmox2 kernel: [ 5753.680516] CR2: 000055da41e3bc80 CR3: 0000001d03c10006 CR4: 00000000001706f0
Jan 6 19:47:31 proxmox2 kernel: [ 5753.681376] Call Trace:
Jan 6 19:47:31 proxmox2 kernel: [ 5753.682182] <IRQ>
Jan 6 19:47:31 proxmox2 kernel: [ 5753.682962] ? common_interrupt+0x55/0xa0
Jan 6 19:47:31 proxmox2 kernel: [ 5753.683734] netdev_core_pick_tx+0xa4/0xb0
Jan 6 19:47:31 proxmox2 kernel: [ 5753.684488] __dev_queue_xmit+0x1b8/0xb30
Jan 6 19:47:31 proxmox2 kernel: [ 5753.685282] ? netif_rx_internal+0x3a/0x100
Jan 6 19:47:31 proxmox2 kernel: [ 5753.686052] dev_queue_xmit+0x10/0x20
Jan 6 19:47:31 proxmox2 kernel: [ 5753.686801] ovs_vport_send+0xae/0x170 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.687561] do_output+0x59/0x180 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.688277] do_execute_actions+0x1841/0x1b40 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.688993] ? __skb_flow_dissect+0x2f2/0x1920
Jan 6 19:47:31 proxmox2 kernel: [ 5753.689790] ? flow_lookup.constprop.0+0xa1/0x110 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.690496] ovs_execute_actions+0x48/0x110 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.691188] ? ovs_execute_actions+0x48/0x110 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.691866] ovs_dp_process_packet+0xa1/0x1f0 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.692579] ? ovs_ct_update_key.isra.0+0xa8/0x120 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.693303] ? ovs_ct_fill_key+0x1d/0x30 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.694001] ? ovs_flow_key_extract+0x2da/0x350 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.694669] ovs_vport_receive+0x77/0xd0 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.695380] ? tcp_ack_update_rtt+0x12f/0x450
Jan 6 19:47:31 proxmox2 kernel: [ 5753.696049] ? kfree_skbmem+0x6f/0xa0
Jan 6 19:47:31 proxmox2 kernel: [ 5753.696723] ? cpumask_next_and+0x24/0x30
Jan 6 19:47:31 proxmox2 kernel: [ 5753.697470] netdev_frame_hook+0xdf/0x1b0 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.698121] ? netdev_create+0x40/0x40 [openvswitch]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.698757] __netif_receive_skb_core+0x238/0xef0
Jan 6 19:47:31 proxmox2 kernel: [ 5753.699373] __netif_receive_skb_list_core+0x107/0x260
Jan 6 19:47:31 proxmox2 kernel: [ 5753.699972] netif_receive_skb_list_internal+0x1a1/0x2c0
Jan 6 19:47:31 proxmox2 kernel: [ 5753.700557] ? dev_gro_receive+0x2d6/0x7b0
Jan 6 19:47:31 proxmox2 kernel: [ 5753.701148] ? kmem_cache_alloc+0x1ab/0x2f0
Jan 6 19:47:31 proxmox2 kernel: [ 5753.701746] ? __build_skb+0x26/0x60
Jan 6 19:47:31 proxmox2 kernel: [ 5753.702275] napi_complete_done+0x7a/0x1c0
Jan 6 19:47:31 proxmox2 kernel: [ 5753.702790] ixgbe_poll+0xe19/0x1240 [ixgbe]
Jan 6 19:47:31 proxmox2 kernel: [ 5753.703299] ? tick_nohz_irq_exit+0x3b/0x50
Jan 6 19:47:31 proxmox2 kernel: [ 5753.703784] __napi_poll+0x33/0x180
Jan 6 19:47:31 proxmox2 kernel: [ 5753.704257] net_rx_action+0x126/0x280
Jan 6 19:47:31 proxmox2 kernel: [ 5753.704719] __do_softirq+0xd9/0x2ea
Jan 6 19:47:31 proxmox2 kernel: [ 5753.705201] irq_exit_rcu+0x94/0xc0
Jan 6 19:47:31 proxmox2 kernel: [ 5753.705686] common_interrupt+0x8e/0xa0
Jan 6 19:47:31 proxmox2 kernel: [ 5753.706102] </IRQ>
Jan 6 19:47:31 proxmox2 kernel: [ 5753.706502] <TASK>
Jan 6 19:47:31 proxmox2 kernel: [ 5753.706891] asm_common_interrupt+0x27/0x40
Jan 6 19:47:31 proxmox2 kernel: [ 5753.707286] RIP: 0010:cpuidle_enter_state+0xd9/0x620
Jan 6 19:47:31 proxmox2 kernel: [ 5753.707683] Code: 3d 64 64 9f 6c e8 27 24 6e ff 49 89 c7 0f 1f 44 00 00 31 ff e8 68 31 6e ff 80 7d d0 00 0f 85 5e 01 00 00 fb 66 0f 1f 44 00 00 <45> 85 f6 0f 88 6a 01 00
00 4d 63 ee 49 83 fd 09 0f 87 e5 03 00 00
Jan 6 19:47:31 proxmox2 kernel: [ 5753.708534] RSP: 0018:ffffffff94803da0 EFLAGS: 00000246
Jan 6 19:47:31 proxmox2 kernel: [ 5753.708962] RAX: ffff973230030bc0 RBX: ffffc428ff802900 RCX: 0000000000000000
Jan 6 19:47:31 proxmox2 kernel: [ 5753.709465] RDX: 00000000000161bf RSI: 0000000037a6f9a0 RDI: 0000000000000000
Jan 6 19:47:31 proxmox2 kernel: [ 5753.709892] RBP: ffffffff94803df0 R08: 000005351fb3c3f9 R09: 0000000000061a80