Hey guys,
since today I've got a weird issue with my Proxmox host, which I can't resolve on my own.
I noticed, that my Wi-Fi wasn't working, so I decided to take a look on my host, which wasn't available at all. It didn't respond to pings or https. I tried to reboot, but still, no connection was possible.
After looking on the console output, I saw a message saying something network related:
This issue shows up immediately after rebooting the host:
The only way I can fix this temporarily is restarting my switch. It will work for about 30 minutes, then this issue reoccurs. It will print the error message above into the console output and refuses to answer to all incoming network traffic.
I already tried using another switch, but this error persists. So my thoughts are that this issue is host-related.
Some information about the nic:
My Proxmox host is up to date. I'm using Virtual Environment 6.4-8 and my kernel is 5.4.119-1-pve
Any guesses what is going wrong here?
since today I've got a weird issue with my Proxmox host, which I can't resolve on my own.
I noticed, that my Wi-Fi wasn't working, so I decided to take a look on my host, which wasn't available at all. It didn't respond to pings or https. I tried to reboot, but still, no connection was possible.
After looking on the console output, I saw a message saying something network related:
r8169 0000:2d:00.0 enp45s0: rtl_txcfg_empty_cond == 0 (loop: 666, delay: 100).
This issue shows up immediately after rebooting the host:
Code:
[ 50.336167] ------------[ cut here ]------------
[ 50.336175] NETDEV WATCHDOG: enp45s0 (r8169): transmit queue 0 timed out
[ 50.336186] WARNING: CPU: 7 PID: 0 at net/sched/sch_generic.c:448 dev_watchdog+0x264/0x270
[ 50.336186] Modules linked in: nft_limit nft_compat nft_counter cfg80211 8021q garp mrp nf_tables nf_log_ipv6 ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt nf_log_ipv4 nf_log_common ipt_REJECT nf_reject_ipv4 xt_LOG xt_limit xt_addrtype xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter softdog nfnetlink_log nfnetlink snd_hda_codec_hdmi edac_mce_amd snd_hda_codec_realtek kvm_amd snd_hda_codec_generic ledtrig_audio kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel snd_hda_intel crypto_simd snd_intel_dspcfg cryptd snd_hda_codec glue_helper snd_hda_core snd_hwdep joydev snd_pcm pcspkr wmi_bmof snd_timer k10temp snd soundcore ccp mac_hid zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp nfsd libiscsi scsi_transport_iscsi auth_rpcgss
[ 50.336213] nfs_acl lockd grace sunrpc ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c hid_generic usbhid hid i2c_piix4 r8169 xhci_pci realtek ahci xhci_hcd libahci wmi gpio_amdpt gpio_generic
[ 50.336222] CPU: 7 PID: 0 Comm: swapper/7 Tainted: P O 5.4.119-1-pve #1
[ 50.336223] Hardware name: Micro-Star International Co., Ltd. MS-7C56/B550-A PRO (MS-7C56), BIOS A.30 08/31/2020
[ 50.336224] RIP: 0010:dev_watchdog+0x264/0x270
[ 50.336225] Code: 48 85 c0 75 e6 eb a0 4c 89 ef c6 05 60 b8 ef 00 01 e8 20 b8 fa ff 89 d9 4c 89 ee 48 c7 c7 70 5d c3 9f 48 89 c2 e8 95 55 15 00 <0f> 0b eb 82 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41
[ 50.336226] RSP: 0018:ffffbbef003a4e58 EFLAGS: 00010282
[ 50.336227] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[ 50.336227] RDX: ffff9cd63e9e7740 RSI: 00000000000000f6 RDI: 0000000000000300
[ 50.336227] RBP: ffffbbef003a4e88 R08: 0000000000000522 R09: 0000000000000004
[ 50.336228] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000001
[ 50.336228] R13: ffff9cd63ae7e000 R14: ffff9cd63ae7e480 R15: ffff9cd62bb55a80
[ 50.336229] FS: 0000000000000000(0000) GS:ffff9cd63e9c0000(0000) knlGS:0000000000000000
[ 50.336230] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 50.336230] CR2: 00007fa883f961c0 CR3: 0000000799a10000 CR4: 0000000000340ee0
[ 50.336231] Call Trace:
[ 50.336232] <IRQ>
[ 50.336234] ? pfifo_fast_enqueue+0x160/0x160
[ 50.336236] call_timer_fn+0x32/0x130
[ 50.336237] run_timer_softirq+0x1a5/0x430
[ 50.336238] ? ktime_get+0x3c/0xa0
[ 50.336240] ? lapic_next_event+0x20/0x30
[ 50.336242] ? clockevents_program_event+0x93/0xf0
[ 50.336243] __do_softirq+0xdc/0x2d4
[ 50.336246] irq_exit+0xa9/0xb0
[ 50.336246] smp_apic_timer_interrupt+0x79/0x130
[ 50.336247] apic_timer_interrupt+0xf/0x20
[ 50.336248] </IRQ>
[ 50.336249] RIP: 0010:poll_idle+0x98/0xbd
[ 50.336250] Code: 44 89 f0 41 5c 41 5d 41 5e 5d c3 4c 89 ef 48 89 de e8 1c a1 d8 ff 49 89 c5 b8 c9 00 00 00 65 48 8b 14 25 c0 6b 01 00 48 8b 12 <83> e2 08 75 aa f3 90 83 e8 01 75 e8 65 8b 3d 15 cc d4 60 e8 40 03
[ 50.336250] RSP: 0018:ffffbbef0016fe18 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13
[ 50.336251] RAX: 0000000000000096 RBX: ffff9cd62d74d800 RCX: 000000000000001f
[ 50.336251] RDX: 0000000080204000 RSI: ffff9cd62d74d800 RDI: ffffffff9ff669a0
[ 50.336252] RBP: ffffbbef0016fe38 R08: 0000000000000002 R09: 000000000002a680
[ 50.336252] R10: 000000343c685274 R11: ffff9cd63e9e9aa0 R12: 0000000bb844e460
[ 50.336252] R13: 00000000000007d0 R14: 0000000000000000 R15: ffffffff9ff669a0
[ 50.336254] ? poll_idle+0x84/0xbd
[ 50.336256] cpuidle_enter_state+0x75/0x450
[ 50.336257] cpuidle_enter+0x2e/0x40
[ 50.336259] call_cpuidle+0x23/0x40
[ 50.336259] do_idle+0x22c/0x270
[ 50.336260] cpu_startup_entry+0x1d/0x20
[ 50.336261] start_secondary+0x166/0x1c0
[ 50.336263] secondary_startup_64+0xa4/0xb0
[ 50.336264] ---[ end trace d1f5951d4dcc536f ]---
[ 50.426806] r8169 0000:2d:00.0 enp45s0: rtl_txcfg_empty_cond == 0 (loop: 666, delay: 100).
[ 55.286764] r8169 0000:2a:00.0 enp42s0: rtl_txcfg_empty_cond == 0 (loop: 666, delay: 100).
[ 60.406683] r8169 0000:2d:00.0 enp45s0: rtl_txcfg_empty_cond == 0 (loop: 666, delay: 100).
The only way I can fix this temporarily is restarting my switch. It will work for about 30 minutes, then this issue reoccurs. It will print the error message above into the console output and refuses to answer to all incoming network traffic.
I already tried using another switch, but this error persists. So my thoughts are that this issue is host-related.
Some information about the nic:
Code:
root@pve01:~# ethtool -i enp45s0
driver: r8169
version:
firmware-version: rtl8168h-2_0.0.2 02/26/15
expansion-rom-version:
bus-info: 0000:2d:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
My Proxmox host is up to date. I'm using Virtual Environment 6.4-8 and my kernel is 5.4.119-1-pve
Any guesses what is going wrong here?