PVE network kernel tg3 issue, intermittent lost of network

quadcube · May 19, 2021

I just made a scheduled shutdown during the weekend and after switching it back on, I periodically lose network access to the server for ~ 1 min.
While going through the syslog, I found it's mainly complaining about tg3, server was pretty idle most of the time, only running a couple of LXCs with individual services (DNS, HomeKit, Nextcloud).

Anyone had the same problem?

Below are some info about the problematic node..

Hardware: Mac mini 2014

Installed package

Code:

proxmox-ve: 6.4-1 (running kernel: 5.4.114-1-pve)
pve-manager: 6.4-6 (running version: 6.4-6/be2fa32c)
pve-kernel-5.4: 6.4-2
pve-kernel-helper: 6.4-2
pve-kernel-5.4.114-1-pve: 5.4.114-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.4-1
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-3
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-2
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.6-2
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-4
pve-cluster: 6.4-1
pve-container: 3.3-5
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-3
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1

/etc/network/interfaces

Code:

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

iface enp3s0f0 inet dhcp

auto vmbr0
iface vmbr0 inet static
        address 192.168.88.21/24
        gateway 192.168.88.1
        bridge-ports enp3s0f0
        bridge-stp off
        bridge-fd 0

Syslog

Code:

May 19 18:32:35 PVE-QuadCubeMacMini kernel: ------------[ cut here ]------------
May 19 18:32:35 PVE-QuadCubeMacMini kernel: NETDEV WATCHDOG: enp3s0f0 (tg3): transmit queue 0 timed out
May 19 18:32:35 PVE-QuadCubeMacMini kernel: WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:448 dev_watchdog+0x264/0x270
May 19 18:32:35 PVE-QuadCubeMacMini kernel: Modules linked in: tcp_diag inet_diag nf_log_ipv4 nf_log_common ipt_REJECT nf_reject_ipv4 xt_LOG xt_limit xt_addrtype xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables sctp iptable_filter bpfilter cmac bnep softdog nfnetlink_log nfnetlink dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c input_leds hid_appleir mei_hdcp intel_rapl_msr hid_generic btusb btrtl btbcm btintel bluetooth ecdh_generic ecc b43 intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp snd_hda_codec_hdmi kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel cordic mac80211 aesni_intel applesmc crypto_simd cfg80211 cryptd glue_helper i915 snd_hda_codec_cirrus drm_kms_helper input_polldev snd_hda_codec_generic ssb rapl ledtrig_audio intel_cstate snd_hda_intel snd_intel_dspcfg libarc4 snd_hda_codec snd_hda_core usbmouse drm snd_hwdep usbkbd snd_pcm pcspkr i2c_algo_bit
May 19 18:32:35 PVE-QuadCubeMacMini kernel:  snd_timer fb_sys_fops mei_me syscopyarea usbhid uas snd sysfillrect sysimgblt soundcore usb_storage mei hid mac_hid zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi wireguard ip6_udp_tunnel udp_tunnel vfio_pci vfio_virqfd irqbypass vfio_iommu_type1 vfio sunrpc ip_tables x_tables autofs4 lpc_ich ahci i2c_i801 libahci xhci_pci bcma sdhci_pci cqhci sdhci thunderbolt tg3 xhci_hcd video
May 19 18:32:35 PVE-QuadCubeMacMini kernel: CPU: 2 PID: 0 Comm: swapper/2 Tainted: P           O      5.4.114-1-pve #1
May 19 18:32:35 PVE-QuadCubeMacMini kernel: Hardware name: Apple Inc. Macmini7,1/Mac-35C5E08120C7EEAF, BIOS 428.0.0.0.0 11/04/2020
May 19 18:32:35 PVE-QuadCubeMacMini kernel: RIP: 0010:dev_watchdog+0x264/0x270
May 19 18:32:35 PVE-QuadCubeMacMini kernel: Code: 48 85 c0 75 e6 eb a0 4c 89 ef c6 05 80 c8 ef 00 01 e8 20 b8 fa ff 89 d9 4c 89 ee 48 c7 c7 98 5c 23 8d 48 89 c2 e8 c5 56 15 00 <0f> 0b eb 82 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41
May 19 18:32:35 PVE-QuadCubeMacMini kernel: RSP: 0018:ffffacae8012ce58 EFLAGS: 00010282
May 19 18:32:35 PVE-QuadCubeMacMini kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000000083f
May 19 18:32:35 PVE-QuadCubeMacMini kernel: RDX: 0000000000000000 RSI: 00000000000000f6 RDI: 000000000000083f
May 19 18:32:35 PVE-QuadCubeMacMini kernel: RBP: ffffacae8012ce88 R08: 0000000000000443 R09: 0000000000000004
May 19 18:32:35 PVE-QuadCubeMacMini kernel: R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000005
May 19 18:32:35 PVE-QuadCubeMacMini kernel: R13: ffff907d9cfca000 R14: ffff907d9cfca480 R15: ffff907d8f741d80
May 19 18:32:35 PVE-QuadCubeMacMini kernel: FS:  0000000000000000(0000) GS:ffff907d9f100000(0000) knlGS:0000000000000000
May 19 18:32:35 PVE-QuadCubeMacMini kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 19 18:32:35 PVE-QuadCubeMacMini kernel: CR2: 00007f247a4cd0a8 CR3: 00000002ef00a002 CR4: 00000000001606e0
May 19 18:32:35 PVE-QuadCubeMacMini kernel: Call Trace:
May 19 18:32:35 PVE-QuadCubeMacMini kernel:  <IRQ>
May 19 18:32:35 PVE-QuadCubeMacMini kernel:  ? pfifo_fast_enqueue+0x160/0x160
May 19 18:32:35 PVE-QuadCubeMacMini kernel:  call_timer_fn+0x32/0x130
May 19 18:32:35 PVE-QuadCubeMacMini kernel:  run_timer_softirq+0x1a5/0x430
May 19 18:32:35 PVE-QuadCubeMacMini kernel:  ? ktime_get+0x3c/0xa0
May 19 18:32:35 PVE-QuadCubeMacMini kernel:  ? lapic_next_deadline+0x2c/0x40
May 19 18:32:35 PVE-QuadCubeMacMini kernel:  ? clockevents_program_event+0x93/0xf0
May 19 18:32:35 PVE-QuadCubeMacMini kernel:  __do_softirq+0xdc/0x2d4
May 19 18:32:35 PVE-QuadCubeMacMini kernel:  irq_exit+0xa9/0xb0
May 19 18:32:35 PVE-QuadCubeMacMini kernel:  smp_apic_timer_interrupt+0x79/0x130
May 19 18:32:35 PVE-QuadCubeMacMini kernel:  apic_timer_interrupt+0xf/0x20
May 19 18:32:35 PVE-QuadCubeMacMini kernel:  </IRQ>
May 19 18:32:35 PVE-QuadCubeMacMini kernel: RIP: 0010:cpuidle_enter_state+0xbd/0x450
May 19 18:32:35 PVE-QuadCubeMacMini kernel: Code: ff e8 b7 79 88 ff 80 7d c7 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 63 03 00 00 31 ff e8 ba 81 8e ff fb 66 0f 1f 44 00 00 <45> 85 ed 0f 88 8d 02 00 00 49 63 cd 48 8b 75 d0 48 2b 75 c8 48 8d
May 19 18:32:35 PVE-QuadCubeMacMini kernel: RSP: 0018:ffffacae800b3e48 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
May 19 18:32:35 PVE-QuadCubeMacMini kernel: RAX: ffff907d9f12ae00 RBX: ffffffff8d557c40 RCX: 000000000000001f
May 19 18:32:35 PVE-QuadCubeMacMini kernel: RDX: 00000080bfb72bca RSI: 000000002aaaab99 RDI: 0000000000000000
May 19 18:32:35 PVE-QuadCubeMacMini kernel: RBP: ffffacae800b3e88 R08: 0000000000000002 R09: 000000000002a680
May 19 18:32:35 PVE-QuadCubeMacMini kernel: R10: 0000018ac8ffcfe4 R11: ffff907d9f129aa0 R12: ffffccae7fd2b300
May 19 18:32:35 PVE-QuadCubeMacMini kernel: R13: 0000000000000008 R14: ffffffff8d557f58 R15: ffffffff8d557f40
May 19 18:32:35 PVE-QuadCubeMacMini kernel:  ? cpuidle_enter_state+0x99/0x450
May 19 18:32:35 PVE-QuadCubeMacMini kernel:  cpuidle_enter+0x2e/0x40
May 19 18:32:35 PVE-QuadCubeMacMini kernel:  call_cpuidle+0x23/0x40
May 19 18:32:35 PVE-QuadCubeMacMini kernel:  do_idle+0x22c/0x270
May 19 18:32:35 PVE-QuadCubeMacMini kernel:  cpu_startup_entry+0x1d/0x20
May 19 18:32:35 PVE-QuadCubeMacMini kernel:  start_secondary+0x166/0x1c0
May 19 18:32:35 PVE-QuadCubeMacMini kernel:  secondary_startup_64+0xa4/0xb0
May 19 18:32:35 PVE-QuadCubeMacMini kernel: ---[ end trace 5b7b03655e1911b2 ]---

Code:

May 19 18:32:35 PVE-QuadCubeMacMini kernel: tg3 0000:03:00.0 enp3s0f0: transmit timed out, resetting
May 19 18:32:38 PVE-QuadCubeMacMini kernel: tg3 0000:03:00.0 enp3s0f0: 0x00000000: 0x168614e4, 0x00100406, 0x02000001, 0x00800040
May 19 18:32:38 PVE-QuadCubeMacMini kernel: tg3 0000:03:00.0 enp3s0f0: 0x00000010: 0xa070000c, 0x00000000, 0xa071000c, 0x00000000
...
...
May 19 18:32:38 PVE-QuadCubeMacMini kernel: tg3 0000:03:00.0 enp3s0f0: 0x00007000: 0x08000188, 0x00000000, 0x00000000, 0x000000c4
May 19 18:32:38 PVE-QuadCubeMacMini kernel: tg3 0000:03:00.0 enp3s0f0: 0x00007010: 0xd1c00001, 0x50408272, 0x00d70081, 0x03008200
May 19 18:32:38 PVE-QuadCubeMacMini kernel: tg3 0000:03:00.0 enp3s0f0: 0x00007020: 0x00000000, 0x00000000, 0x00000406, 0x10004000
May 19 18:32:38 PVE-QuadCubeMacMini kernel: tg3 0000:03:00.0 enp3s0f0: 0x00007030: 0x0001f200, 0x000000c8, 0x00030030, 0x00000000
May 19 18:32:38 PVE-QuadCubeMacMini kernel: tg3 0000:03:00.0 enp3s0f0: 0x00007500: 0x00000000, 0x00000000, 0x00000080, 0x00000000
May 19 18:32:38 PVE-QuadCubeMacMini kernel: tg3 0000:03:00.0 enp3s0f0: 0: Host status block [00000001:0000005d:(0000:0076:0000):(0000:0041)]
May 19 18:32:38 PVE-QuadCubeMacMini kernel: tg3 0000:03:00.0 enp3s0f0: 0: NAPI info [0000005d:0000005d:(002e:0041:01ff):0000:(013e:0000:0000:0000)]
May 19 18:32:38 PVE-QuadCubeMacMini kernel: tg3 0000:03:00.0 enp3s0f0: 1: Host status block [00000001:0000003f:(0000:0000:0000):(0304:0000)]
May 19 18:32:38 PVE-QuadCubeMacMini kernel: tg3 0000:03:00.0 enp3s0f0: 1: NAPI info [0000003f:0000003f:(0000:0000:01ff):0304:(0104:0104:0000:0000)]
May 19 18:32:38 PVE-QuadCubeMacMini kernel: tg3 0000:03:00.0 enp3s0f0: 2: Host status block [00000001:00000031:(0320:0000:0000):(0000:0000)]
May 19 18:32:38 PVE-QuadCubeMacMini kernel: tg3 0000:03:00.0 enp3s0f0: 2: NAPI info [00000031:00000031:(0000:0000:01ff):0320:(0120:0120:0000:0000)]
May 19 18:32:38 PVE-QuadCubeMacMini kernel: tg3 0000:03:00.0 enp3s0f0: 3: Host status block [00000001:00000035:(0000:0000:0000):(0000:0000)]
May 19 18:32:38 PVE-QuadCubeMacMini kernel: tg3 0000:03:00.0 enp3s0f0: 3: NAPI info [00000035:00000035:(0000:0000:01ff):03ff:(01ff:01ff:0000:0000)]
May 19 18:32:38 PVE-QuadCubeMacMini kernel: tg3 0000:03:00.0 enp3s0f0: 4: Host status block [00000001:00000006:(0000:0000:0053):(0000:0000)]
May 19 18:32:38 PVE-QuadCubeMacMini kernel: tg3 0000:03:00.0 enp3s0f0: 4: NAPI info [00000006:00000006:(0000:0000:01ff):0053:(0053:0053:0000:0000)]
May 19 18:32:38 PVE-QuadCubeMacMini kernel: tg3 0000:03:00.0 enp3s0f0: Link is down
May 19 18:32:38 PVE-QuadCubeMacMini kernel: vmbr0: port 1(enp3s0f0) entered disabled state
May 19 18:32:42 PVE-QuadCubeMacMini kernel: tg3 0000:03:00.0 enp3s0f0: Link is up at 1000 Mbps, full duplex
May 19 18:32:42 PVE-QuadCubeMacMini kernel: tg3 0000:03:00.0 enp3s0f0: Flow control is on for TX and on for RX
May 19 18:32:42 PVE-QuadCubeMacMini kernel: tg3 0000:03:00.0 enp3s0f0: EEE is disabled
May 19 18:32:42 PVE-QuadCubeMacMini kernel: vmbr0: port 1(enp3s0f0) entered blocking state
May 19 18:32:42 PVE-QuadCubeMacMini kernel: vmbr0: port 1(enp3s0f0) entered forwarding state

aaron · May 20, 2021

Did you install updates recently? Maybe the Broadcom drivers in that kernel version have some issues with the NIC in that Mac Mini. You could try the newer (opt in) kernel 5.11. apt install pve-kernel-5.11 and reboot the machine.

Did you check if there are any firmware/BIOS updates for the hardware?

quadcube · May 22, 2021

aaron said:
Did you install updates recently? Maybe the Broadcom drivers in that kernel version have some issues with the NIC in that Mac Mini. You could try the newer (opt in) kernel 5.11. apt install pve-kernel-5.11 and reboot the machine.

Did you check if there are any firmware/BIOS updates for the hardware?

Hi,

Thank you for the suggestion! I tried the latest pve-kernel-5.11. However the tg3 still crashed as usual.
A little more digging around the files reminded me of my previous attempt to passthrough the Mac mini's WiFi module to Kali Linux.
Removed all the passthrough modification I made in /etc/default/grub and /etc/modules seems to solve the issue so far, 12 hours without any ethernet issues.

kristian.kirilov · Sep 2, 2023

quadcube sorry, can you describe what you've removed? I have almost the same issue (no connection at all) I'm doing passthrough from TrueNAS Scale system to a VM running Proxmox.

pbee · 2024-11-08T13:52:02+0100

I have a Mac mini 2014 as well, but the TG3 with Broadcom remains. There are 3 NIC's, two are Apple Thunderbolt NIC's and one the internal NIC.
The network configuration was fully VirtIO at first (vSwitch), but now I have the two Thunderbolt NIC's as a pass-thru to my router VM. Those two are stable now, but the internal still that is in use by ProxMox keeps on crashing, with or without load.

Have changed the grub line:

Code:

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=off iommu=pt"

Turned it off for a while and on for a while. In both situations TG3 crashed within hours up to 4 weeks.
At this moment it is set to OFF, even though I have two NIC's In pass-thru.

Current running version: proxmox-kernel-6.8 (6.8.12-2).

Have done on other setting now for the internal NIC:

Code:

ethtool -K enp3s0f0 gro off gso off tso off

Search

Search

PVE network kernel tg3 issue, intermittent lost of network

quadcube

New Member

aaron

Proxmox Staff Member

quadcube

New Member

kristian.kirilov

Well-Known Member

pbee

New Member