System hanging after upgrade...NIC driver?

RoyalPro · Jun 23, 2023

After upgrading to 8 accessing my pi-hole web interface would cause my entire system to crash requiring a hard reset. The pi-hole would still work for DNS. When looking at the syslog I see:
Jun 22 16:15:59 prox kernel: NETDEV WATCHDOG: enp6s0 (r8169): transmit queue 0 timed out
Jun 22 16:15:59 prox kernel: WARNING: CPU: 26 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x23a/0x250....'a bunch of information'
....Jun 22 16:16:21 prox kernel: r8169 0000:06:00.0 enp6s0: rtl_chipcmd_cond == 1 (loop: 100, delay: 100).

The pi-hole was using a vlan that was plugged into the 2.5G port on my motherboard. After changing to different NIC the problem stopped. Is this a bug with the driver for my 2.5G (Dragon RTL8125AG)?

Edited to add NIC name. I could also supply more of my syslog that I omitted if that would be helpful.

nottledim · Jun 23, 2023

Similar here. My r8168 ether interface worked fine on 7 with r8169 driver but doesn't work on 8.


r8169 0000:01:00.0: Unable to change power state from D3cold to D0, device inaccessible
[10104.212219] r8169 0000:01:00.0: Mem-Wr-Inval unavailable
[10104.212244] r8169 0000:01:00.0: unknown chip XID fcf, contact r8169 maintainers (see MAINTAINERS file)

My device is a


Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)

riv76 · Jun 23, 2023

Similar here too.
Upgraded today to 8.0 from 7.4.
Never had this issue before.

my device is

02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

andrema2 · Jun 23, 2023

I'm seeing the same thing

riv76 · Jun 23, 2023

Could be a good idea install this Realtek binary firmare for Debian Bookworm (Proxmox 8.0)?

https://packages.debian.org/bookworm/firmware-realtek

andrema2 · Jun 23, 2023

riv76 said:
Could be a good idea install this Realtek binary firmare for Debian Bookworm (Proxmox 8.0)?

https://packages.debian.org/bookworm/firmware-realtek

Have you done it ? Did it work ?

RoyalPro · Jun 23, 2023

riv76 said:
Could be a good idea install this Realtek binary firmare for Debian Bookworm (Proxmox 8.0)?

https://packages.debian.org/bookworm/firmware-realtek

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package firmware-realtek is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
pve-firmware

E: Package 'firmware-realtek' has no installation candidate

Zucchini · Jun 23, 2023

Same issue after upgrade today:
(r8169): transmit queue 0 timed out proxmox

riv76 · Jun 24, 2023

RoyalPro said:
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package firmware-realtek is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
pve-firmware

E: Package 'firmware-realtek' has no installation candidate

Would you test this?

https://packages.debian.org/bookworm/r8168-dkms


dkms source for the r8168 network driver
r8168 is the Linux device driver released by RealTek for their network controllers with PCI-Express interface:

 * 10/100/1000M Gigabit Ethernet: RTL8111B, RTL8111C, RTL8111D, RTL8111E,
   RTL8111F, RTL8111G(S), RTL8111H(S), RTL8118(A)(S), RTL8119i, RTL8111L,
   RTL8111K, RTL8168B, RTL8168E, RTL8168H, RTL8111DP, RTL8111EP, RTL8111FP,
   RTL8411, RTL8411B

This driver should only be used for devices not yet supported by the in-kernel driver r8169. Please see the README.Debian for instructions how to report bugs against r8169 that made it necessary to use r8168-dkms.

Installation of the r8168-dkms package will disable the in-kernel r8169 module. To re-enable r8169, the r8168-dkms package must be purged.

This package provides the dkms source code for the r8168 kernel modules. Kernel source or headers are required to compile these modules.

(I'd run some tests myself, but I'm not experienced enough with these commands on linux and fear to broke the system definitely)

riv76 · Jun 24, 2023

This person seems to have the same problem as me with the 6.x kernel, ie after about 2 hours from boot the system loses the network (he doesn't specify the error messages though).In the end he writes a workaround, which with my low knowledge of Linux, I can't understand.
Could it be a good workaround?

https://groups.google.com/g/linux.debian.bugs.dist/c/qGqyFcHyE8w

riv76 · Jun 24, 2023

riv76 said:
Would you test this?

https://packages.debian.org/bookworm/r8168-dkms

dkms source for the r8168 network driver r8168 is the Linux device driver released by RealTek for their network controllers with PCI-Express interface: * 10/100/1000M Gigabit Ethernet: RTL8111B, RTL8111C, RTL8111D, RTL8111E, RTL8111F, RTL8111G(S), RTL8111H(S), RTL8118(A)(S), RTL8119i, RTL8111L, RTL8111K, RTL8168B, RTL8168E, RTL8168H, RTL8111DP, RTL8111EP, RTL8111FP, RTL8411, RTL8411B This driver should only be used for devices not yet supported by the in-kernel driver r8169. Please see the README.Debian for instructions how to report bugs against r8169 that made it necessary to use r8168-dkms. Installation of the r8168-dkms package will disable the in-kernel r8169 module. To re-enable r8169, the r8168-dkms package must be purged. This package provides the dkms source code for the r8168 kernel modules. Kernel source or headers are required to compile these modules.

(I'd run some tests myself, but I'm not experienced enough with these commands on linux and fear to broke the system definitely)

wget r8168-dkms_8.051.02-2_all.deb
apt install ./r8168-dkms_8.051.02-2_all.deb

I think has been installed without error, only a warning about I was in /root.

Now testing stabilty

How can I check if my Proxmox is using the just installed module (r8168-dkms_8.051.02-2_all.deb) ?

Zucchini · Jun 24, 2023

riv76 said:
wget r8168-dkms_8.051.02-2_all.deb
apt install ./r8168-dkms_8.051.02-2_all.deb

I think has been installed without error, only a warning about I was in /root.

Now testing stabilty

How can I check if my Proxmox is using the just installed module (r8168-dkms_8.051.02-2_all.deb) ?

# lsmod | grep 816
Running the in-kernel (8169) driver returns:
r8169 114688 0
You'd expect to see r8168

riv76 · Jun 24, 2023


root@pve-hp:~# lsmod | grep 816
r8168                 655360  0
root@pve-hp:~#

~~it's okay?~~
(my question was about the numbers on second column, but i understood that is the module size)

mzac23 · Jun 24, 2023

Also having a similar issue. Running proxmox on an old Thinkpad laptop with a Realtek 8168 NIC and just upgraded to v8. The system keeps locking up and have to power off and power back on. Real bummer to have this happen every few hours. I can't find anything in any of the system logs and nothing shows up on the console which is also locked up.

Edit 1: started getting errors on the console relating to the nic...

Edit 2: found a post with a similar issue suggesting to install r8168-dkms but seems this package is not available on pve 8

https://askubuntu.com/questions/140...340-15irh-gaming-wired-networking-not-working

Edit 3: Was able to install r8168-dkms after adding the non-free and non-free-firmware to my apt sources.list. I now see this kernel module. Will update if my system crashes again.

Code:

root@pve3:~# cat /etc/apt/sources.list
deb http://ftp.ca.debian.org/debian bookworm main contrib non-free non-free-firmware

deb http://ftp.ca.debian.org/debian bookworm-updates main contrib non-free non-free-firmware

# security updates
deb http://security.debian.org bookworm-security main contrib non-free non-free-firmware

deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription

Code:

root@pve3:~# apt install r8168-dkms
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  build-essential cpp cpp-12 dkms dpkg-dev fakeroot g++ g++-12 gcc gcc-12 libabsl20220623 libalgorithm-diff-perl libalgorithm-diff-xs-perl
  libalgorithm-merge-perl libaom3 libasan8 libatomic1 libavif15 libc-dev-bin libc-devtools libc6-dev libcc1-0 libcrypt-dev libdav1d6 libde265-0 libdpkg-perl
  libfakeroot libfile-fcntllock-perl libgav1-1 libgcc-12-dev libgd3 libgomp1 libheif1 libisl23 libitm1 liblsan0 libmpc3 libmpfr6 libnsl-dev libquadmath0
  librav1e0 libstdc++-12-dev libsvtav1enc1 libtirpc-dev libtsan2 libubsan1 libx265-199 libxpm4 libyuv0 linux-libc-dev make manpages-dev pve-headers-6.2
  pve-headers-6.2.16-3-pve rpcsvc-proto sudo
Suggested packages:
  cpp-doc gcc-12-locales cpp-12-doc menu debian-keyring g++-multilib g++-12-multilib gcc-12-doc gcc-multilib autoconf automake libtool flex bison gdb gcc-doc
  gcc-12-multilib glibc-doc bzr libgd-tools libstdc++-12-doc make-doc
The following NEW packages will be installed:
  build-essential cpp cpp-12 dkms dpkg-dev fakeroot g++ g++-12 gcc gcc-12 libabsl20220623 libalgorithm-diff-perl libalgorithm-diff-xs-perl
  libalgorithm-merge-perl libaom3 libasan8 libatomic1 libavif15 libc-dev-bin libc-devtools libc6-dev libcc1-0 libcrypt-dev libdav1d6 libde265-0 libdpkg-perl
  libfakeroot libfile-fcntllock-perl libgav1-1 libgcc-12-dev libgd3 libgomp1 libheif1 libisl23 libitm1 liblsan0 libmpc3 libmpfr6 libnsl-dev libquadmath0
  librav1e0 libstdc++-12-dev libsvtav1enc1 libtirpc-dev libtsan2 libubsan1 libx265-199 libxpm4 libyuv0 linux-libc-dev make manpages-dev pve-headers-6.2
  pve-headers-6.2.16-3-pve r8168-dkms rpcsvc-proto sudo
0 upgraded, 57 newly installed, 0 to remove and 0 not upgraded.
Need to get 84.6 MB of archives.
After this operation, 372 MB of additional disk space will be used.
Do you want to continue? [Y/n]

Code:

root@pve3:~# lsmod | grep 816
r8168                 655360  0

LiloBzH · Jun 25, 2023

hi

same problem here since PROXMOX 8.
test in progress with another driver

r8168-dkms installed via APT

Before

Code:

root@proxmox:~# lsmod | grep 816
r8169                 114688  0

After

Code:

root@proxmox:~# lsmod | grep 816
r8168                 655360  0

riv76 · Jun 25, 2023

all stable so far

LiloBzH · Jun 25, 2023

laz · Jun 25, 2023

After reducing the PVE8 kernel to 5.15.108-1, the network adapter(r8169) works fine.

Code:

root@PVE:~# uname -a
Linux PVE 5.15.108-1-pve #1 SMP PVE 5.15.108-1 (2023-06-17T09:41Z) x86_64 GNU/Linux

root@PVE:~# uptime
04:37:15 up 18:56,  1 user,  load average: 0.22, 0.27, 0.29

root@PVE:~# lsmod | grep 816
r8169                 102400  0

spirit · Jun 25, 2023

please do a bugreport on bugzilla.proxmox.com with all the infos

zeha · Jun 26, 2023

Chiming in here because I have the same problem on 6.2.16-3-pve:

Linux las-vh01 6.2.16-3-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-3 (2023-06-17T05:58Z) x86_64 GNU/Linux

02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)

Edit: worked just fine on 5.15.104-1-pve.

Code:

kernel: [36008.072686] ------------[ cut here ]------------
kernel: [36008.072695] NETDEV WATCHDOG: enp2s0 (r8169): transmit queue 0 timed out
kernel: [36008.072712] WARNING: CPU: 4 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x23a/0x250
kernel: [36008.072724] Modules linked in: tcp_diag inet_diag veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter scsi_transport_iscsi nf_tables softdog bonding tls sunrpc nfnetlink_log nfnetlink binfmt_misc intel_rapl_msr intel_rapl_common amdgpu edac_mce_amd snd_sof_amd_rembrandt snd_sof_amd_renoir snd_sof_amd_acp kvm_amd snd_sof_pci snd_sof_xtensa_dsp iommu_v2 drm_buddy snd_sof snd_hda_codec_realtek gpu_sched mt7921e drm_ttm_helper snd_sof_utils kvm snd_hda_codec_generic ttm mt7921_common snd_hda_codec_hdmi snd_soc_core mt76_connac_lib irqbypass crct10dif_pclmul drm_display_helper snd_compress polyval_clmulni snd_hda_intel polyval_generic ac97_bus mt76 btusb cec snd_pcm_dmaengine ghash_clmulni_intel snd_intel_dspcfg snd_intel_sdw_acpi sha512_ssse3 btrtl snd_hda_codec snd_pci_ps aesni_intel rc_core btbcm snd_rpl_pci_acp6x crypto_simd btintel snd_hda_core asus_nb_wmi eeepc_wmi snd_acp_pci btmtk cryptd mac80211 drm_kms_helper asus_wmi
kernel: [36008.072787]  snd_pci_acp6x snd_hwdep bluetooth snd_pcm ledtrig_audio i2c_algo_bit rapl sparse_keymap ecdh_generic cp210x syscopyarea cfg80211 platform_profile ecc snd_timer sysfillrect pcspkr snd_pci_acp5x wmi_bmof usbserial sysimgblt snd snd_rn_pci_acp3x k10temp snd_acp_config ucsi_acpi soundcore snd_soc_acpi libarc4 ccp snd_pci_acp3x typec_ucsi typec serial_multi_instantiate mac_hid vhost_net vhost vhost_iotlb tap drm efi_pstore dmi_sysfs ip_tables x_tables autofs4 zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs blake2b_generic xor raid6_pq libcrc32c simplefb xhci_pci nvme xhci_pci_renesas crc32_pclmul ahci i2c_piix4 libahci nvme_core r8169 nvme_common xhci_hcd realtek video wmi
kernel: [36008.072869] CPU: 4 PID: 0 Comm: swapper/4 Tainted: P           O       6.2.16-3-pve #1
kernel: [36008.072875] Hardware name: ASUSTeK COMPUTER INC. MINIPC PN52/PN52, BIOS 11300 06/15/2022
kernel: [36008.072878] RIP: 0010:dev_watchdog+0x23a/0x250
kernel: [36008.072884] Code: 00 e9 2b ff ff ff 48 89 df c6 05 8a 6f 7d 01 01 e8 6b 08 f8 ff 44 89 f1 48 89 de 48 c7 c7 58 64 40 9d 48 89 c2 e8 06 ab 30 ff <0f> 0b e9 1c ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00
kernel: [36008.072890] RSP: 0018:ffff9a11c02cce38 EFLAGS: 00010246
kernel: [36008.072894] RAX: 0000000000000000 RBX: ffff8d45543bc000 RCX: 0000000000000000
kernel: [36008.072898] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
kernel: [36008.072901] RBP: ffff9a11c02cce68 R08: 0000000000000000 R09: 0000000000000000
kernel: [36008.072904] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8d45543bc4c8
kernel: [36008.072906] R13: ffff8d45543bc41c R14: 0000000000000000 R15: 0000000000000000
kernel: [36008.072909] FS:  0000000000000000(0000) GS:ffff8d540e700000(0000) knlGS:0000000000000000
kernel: [36008.072913] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: [36008.072916] CR2: 00007f01c75ba000 CR3: 0000000934810000 CR4: 0000000000750ee0
kernel: [36008.072920] PKRU: 55555554
kernel: [36008.072922] Call Trace:
kernel: [36008.072925]  <IRQ>
kernel: [36008.072929]  ? __pfx_dev_watchdog+0x10/0x10
kernel: [36008.072935]  call_timer_fn+0x2c/0x160
kernel: [36008.072940]  ? __pfx_dev_watchdog+0x10/0x10
kernel: [36008.072945]  __run_timers+0x259/0x310
kernel: [36008.072950]  run_timer_softirq+0x1d/0x40
kernel: [36008.072953]  __do_softirq+0xd9/0x346
kernel: [36008.072958]  ? hrtimer_interrupt+0x11f/0x250
kernel: [36008.072963]  __irq_exit_rcu+0xa2/0xd0
kernel: [36008.072968]  irq_exit_rcu+0xe/0x20
kernel: [36008.072972]  sysvec_apic_timer_interrupt+0x92/0xd0
kernel: [36008.072977]  </IRQ>
kernel: [36008.072978]  <TASK>
kernel: [36008.072980]  asm_sysvec_apic_timer_interrupt+0x1b/0x20
kernel: [36008.072984] RIP: 0010:cpuidle_enter_state+0xde/0x6f0
kernel: [36008.072990] Code: 2a 97 63 e8 54 7e 4a ff 8b 53 04 49 89 c7 0f 1f 44 00 00 31 ff e8 82 86 49 ff 80 7d d0 00 0f 85 eb 00 00 00 fb 0f 1f 44 00 00 <45> 85 f6 0f 88 12 02 00 00 4d 63 ee 49 83 fd 09 0f 87 c7 04 00 00
kernel: [36008.072995] RSP: 0018:ffff9a11c019fe38 EFLAGS: 00000246
kernel: [36008.072998] RAX: 0000000000000000 RBX: ffff8d45473d5c00 RCX: 0000000000000000
kernel: [36008.073001] RDX: 0000000000000004 RSI: 0000000000000000 RDI: 0000000000000000
kernel: [36008.073004] RBP: ffff9a11c019fe88 R08: 0000000000000000 R09: 0000000000000000
kernel: [36008.073007] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff9ded6600
kernel: [36008.073009] R13: 0000000000000003 R14: 0000000000000003 R15: 000020bfc8617ca1
kernel: [36008.073015]  ? cpuidle_enter_state+0xce/0x6f0
kernel: [36008.073018]  ? sched_clock_local+0x13/0x90
kernel: [36008.073023]  cpuidle_enter+0x2e/0x50
kernel: [36008.073027]  do_idle+0x216/0x2a0
kernel: [36008.073033]  cpu_startup_entry+0x1d/0x20
kernel: [36008.073036]  start_secondary+0x122/0x160
kernel: [36008.073042]  secondary_startup_64_no_verify+0xe5/0xeb
kernel: [36008.073052]  </TASK>
kernel: [36008.073054] ---[ end trace 0000000000000000 ]---
kernel: [36008.101687] r8169 0000:02:00.0 enp2s0: rtl_chipcmd_cond == 1 (loop: 100, delay: 100).
kernel: [36008.102878] r8169 0000:02:00.0 enp2s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
kernel: [36008.103979] r8169 0000:02:00.0 enp2s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
kernel: [36008.105116] r8169 0000:02:00.0 enp2s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
kernel: [36008.106232] r8169 0000:02:00.0 enp2s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
kernel: [36008.107331] r8169 0000:02:00.0 enp2s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
kernel: [36008.108476] r8169 0000:02:00.0 enp2s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
kernel: [36008.128718] r8169 0000:02:00.0 enp2s0: rtl_mac_ocp_e00e_cond == 1 (loop: 10, delay: 1000).
kernel: [36008.129391] r8169 0000:02:00.0 enp2s0: rtl_ocp_gphy_cond == 1 (loop: 10, delay: 25).
kernel: [36008.129995] r8169 0000:02:00.0 enp2s0: rtl_ocp_gphy_cond == 1 (loop: 10, delay: 25).
kernel: [36008.130585] r8169 0000:02:00.0 enp2s0: Link is Down
kernel: [36008.130827] vmbr0: port 1(enp2s0) entered disabled state

System hanging after upgrade...NIC driver?

New Member

New Member

New Member

Attachments

Member

New Member

Member

New Member

New Member

New Member

New Member

New Member

New Member

New Member

New Member

Member

New Member

Member

Member

Distinguished Member

Well-Known Member