Networking issues PVE8

[c.weilguny]

Just another "upvote" for this solution.

I already had issues in Proxmox 7, no disconnects but the interface was only on 100 MBits most of the time. Did a complete reinstall with PVE 8 some weeks ago, all OK until today, when the interface suddenly went offline out of nothing. Reconnecting the cable sometimes brought it up, but only for seconds, then it was down again. Somehow feels like 20 years ago when WiFi was kinda new, drivers were trash and you solved network issues on your laptop by stopping and starting the interface ...

As the device is an Intel NUC, I can only add USB devices, so looking for some USB NIC with an Intel chip for fallback purposes now - or even as main interface if the onboard Realtek goes nuts again. (EDIT: and found out that no Intel USB NICs exist because USB ...)

 

[scooter]

I started having this same issue. Loaded Proxmox 2 weeks ago along with Home Assistant. Worked fine for 2 weeks. Then, I loaded the Dahua integration, but didn't do much testing. Then added RTSPtoWeb and also added it to HA. When I then opened RTSPtoWeb in the included web browser and then opened sever camera views my Proxmox locked up and begin returning the following error (abbreviated):

r8169 0000:01:00.0 enp1s0 rtl_chipcmd_cond == 1 (loop: 100, delay: 100),
r8169 0000:01:00.0 enp1s0 rtl_ephyar_cond == 1 (loop: 100, delay: 10),
r8169 0000:01:00.0 enp1s0 erier _cond == 1 (loop: 100, delay: 100),

Running a Dell thinclient Optiplex 5070 with 16gb of memory.

The loop error happened again when I again opened RTSPtoWeb in it's web browser. Not all that tech savvy, so I'm only posting this in case it helps with diagnosis. I plan to first remove RTSPtoWeb integration and see if the Dahua cameras alone cause the same error. May also try some of the above-mentioned possible solutions. But I do think it is related to the load crated by camera video.

 

[promok]

Unfortunately the same here, after several days (I think like 10). Looking into syslog there was a problem after some update.
I found this thread that says it was fixed by adding the following into kernel:
r8168.aspm=0 r8168.eee_enable=0 pcie_aspm=off loglevel=3

Rock solid since ..

Hello,
How did you "add into kernel"? This says NOTHING to all the non-system programmers.
What command(s)? Do you have a detailed instruction, please?
Thanks

 

[scooter]

Hello,
How did you "add into kernel"? This says NOTHING to all the non-system programmers.
What command(s)? Do you have a detailed instruction, please?
Thanks

I'm a bit of a newbie at this but I followed the posts in this thread to downgrade the network "driver" on my Dell micro.

https://forum.proxmox.com/threads/system-hanging-after-upgrade-nic-driver.129366/

Hope that helps.

 

[himi]

Hello,
How did you "add into kernel"? This says NOTHING to all the non-system programmers.
What command(s)? Do you have a detailed instruction, please?
Thanks

To add kernel parameters:
Ubuntu Wiki - Permanently Add a Kernel Boot Parameter

So in my case it looks like this:

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet r8168.aspm=0 r8168.eee_enable=0 pcie_aspm=off loglevel=3"
GRUB_CMDLINE_LINUX=""

# If your computer has multiple operating systems installed ..

 

[wolpi]

For those with problems with Proxmox 8.1 - see my post HERE

 

[bruno69]

Hello
A month ago, I had exactly the same issue that is my ethernet connection broke after some days ...
I Installed the R8168-dkms package as rRobbie and Chris said, and that solved the problem for kernel
6.2.16-19

But now dist-upgrade to kernel 6.5.11.4 failed. The message I get is :

r8168.ko:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/6.2.16-19-pve/updates/dkms/
depmod...
Building initial module for 6.5.11-4-pve
Error! Bad return status for module build on kernel: 6.5.11-4-pve (x86_64)
Consult /var/lib/dkms/r8168/8.051.02/build/make.log for more information.
dpkg: error processing package r8168-dkms (--configure):
installed r8168-dkms package post-installation script subprocess returned error exit status 10

Here is the content of /var/lib/dkms/r8168/8.051.02/build/make.log:
Sun Nov 26 11:56:11 AM CET 2023
make: Entering directory '/usr/src/linux-headers-6.5.11-4-pve'
CC [M] /var/lib/dkms/r8168/8.051.02/build/r8168_n.o
CC [M] /var/lib/dkms/r8168/8.051.02/build/r8168_asf.o
CC [M] /var/lib/dkms/r8168/8.051.02/build/rtl_eeprom.o
CC [M] /var/lib/dkms/r8168/8.051.02/build/rtltool.o
/var/lib/dkms/r8168/8.051.02/build/r8168_n.c: In function ‘r8168_csum_workaround’:
/var/lib/dkms/r8168/8.051.02/build/r8168_n.c:29208:24: error: implicit declaration of function ‘skb_gso_segment’; did you mean ‘skb_gso_reset’? [-Werror=implicit-function-
declaration]
29208 | segs = skb_gso_segment(skb, features);
| ^~~~~~~~~~~~~~~
| skb_gso_reset
/var/lib/dkms/r8168/8.051.02/build/r8168_n.c:29208:22: warning: assignment to ‘struct sk_buff *’ from ‘int’ makes pointer from integer without a cast [-Wint-conversion]
29208 | segs = skb_gso_segment(skb, features);
| ^
cc1: some warnings being treated as errors
make[2]: *** [scripts/Makefile.build:251: /var/lib/dkms/r8168/8.051.02/build/r8168_n.o] Error 1
make[1]: *** [/usr/src/linux-headers-6.5.11-4-pve/Makefile:2039: /var/lib/dkms/r8168/8.051.02/build] Error 2
make: *** [Makefile:234: __sub-make] Error 2
make: Leaving directory '/usr/src/linux-headers-6.5.11-4-pve'


Can anyone help ?

Thanks,

 

[promok]

Hello
A month ago, I had exactly the same issue that is my ethernet connection broke after some days ...
I Installed the R8168-dkms package as rRobbie and Chris said, and that solved the problem for kernel
6.2.16-19

But now dist-upgrade to kernel 6.5.11.4 failed. The message I get is :

r8168.ko:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/6.2.16-19-pve/updates/dkms/
depmod...
Building initial module for 6.5.11-4-pve
Error! Bad return status for module build on kernel: 6.5.11-4-pve (x86_64)
Consult /var/lib/dkms/r8168/8.051.02/build/make.log for more information.
dpkg: error processing package r8168-dkms (--configure):
installed r8168-dkms package post-installation script subprocess returned error exit status 10

Here is the content of /var/lib/dkms/r8168/8.051.02/build/make.log:
Sun Nov 26 11:56:11 AM CET 2023
make: Entering directory '/usr/src/linux-headers-6.5.11-4-pve'
CC [M] /var/lib/dkms/r8168/8.051.02/build/r8168_n.o
CC [M] /var/lib/dkms/r8168/8.051.02/build/r8168_asf.o
CC [M] /var/lib/dkms/r8168/8.051.02/build/rtl_eeprom.o
CC [M] /var/lib/dkms/r8168/8.051.02/build/rtltool.o
/var/lib/dkms/r8168/8.051.02/build/r8168_n.c: In function ‘r8168_csum_workaround’:
/var/lib/dkms/r8168/8.051.02/build/r8168_n.c:29208:24: error: implicit declaration of function ‘skb_gso_segment’; did you mean ‘skb_gso_reset’? [-Werror=implicit-function-
declaration]
29208 | segs = skb_gso_segment(skb, features);
| ^~~~~~~~~~~~~~~
| skb_gso_reset
/var/lib/dkms/r8168/8.051.02/build/r8168_n.c:29208:22: warning: assignment to ‘struct sk_buff *’ from ‘int’ makes pointer from integer without a cast [-Wint-conversion]
29208 | segs = skb_gso_segment(skb, features);
| ^
cc1: some warnings being treated as errors
make[2]: *** [scripts/Makefile.build:251: /var/lib/dkms/r8168/8.051.02/build/r8168_n.o] Error 1
make[1]: *** [/usr/src/linux-headers-6.5.11-4-pve/Makefile:2039: /var/lib/dkms/r8168/8.051.02/build] Error 2
make: *** [Makefile:234: __sub-make] Error 2
make: Leaving directory '/usr/src/linux-headers-6.5.11-4-pve'


Can anyone help ?

Thanks,

The same thing happened to me when i upgraded.
So, i had to boot to previous kernel in GRUB, then remove all the DKMS stuff like this:
sudo dkms remove r8168/8.051.02 --all sudo apt-get purge r8168-dkms sudo apt autoremove reboot
And since I have not blacklisted the r8169 driver, it took over automatically and it works well already more than 2 weeks.
You should try without DKMS first, maybe it will work OK, before you try to reinstall that DKMS heavy workaround.

 

[promok]

The same thing happened to me when i upgraded.
So, i had to boot to previous kernel in GRUB, then remove all the DKMS stuff like this:
sudo dkms remove r8168/8.051.02 --all sudo apt-get purge r8168-dkms sudo apt autoremove reboot
And since I have not blacklisted the r8169 driver, it took over automatically and it works well already more than 2 weeks.
You should try without DKMS first, maybe it will work OK, before you try to reinstall that DKMS heavy workaround.

And it continue to work fine on stock driver without problems. I hope it will stay solid after the future updates.

 

[Dunuin]

Hi,

So is the problem with the r8111/8411/8169 fixed meanwhile? Got a node here I upgraded from 7.4 to 8.0 or 8.1 and wasn't able to ssh into it. I connected it to a display and it is stuck at the Initramfs where I need to unlock the encrypted root filesystem via dropbear which isn't working as the NIC is dead. Display shows the NIC is down, managed switch reports there is no link and the display shows the r8169 is in use.
So looks like I'm hit with that driver problem too.
The question now would be if I just need to upgrade to the lastest PVE8 (which would require doing it via some USB NIC or USB Wifi as there i no other onbord interface or accessible PCIe slot) to get it working again or if I need to install the r8168-dkms.
And if I would need to install that r8168-dkms in PVE, what would I need to change so the initramfs will also use the r8168-dkms instead of the default r8169?

 

[Dunuin]

Ok, so I got a keyboard, unlocked the root filesystem and logged in as root. NIC is indeed a "8111/8411/8169"...


...and PVE was already on PVE 8.1.3 booted with 6.5 kernel. ip addr complained "NO-CARRIER". unplugged and plugged the cable again and suddenly it was working. I quickly ran "apt update && apt full-upgrade" and upgraded PVE to the latest version. After rebooting it the NICs wasn't working anymore...
So here its still not working out of the box...

Then I tried to install the r8168-dkms via "apt update && apt install r8168-dkms". This worked but it only built the driver for the 6.1 kernel because it couldn't find the 6.2 and 6.5 kernel headers. But the installation finished.
Then i removed it again via "apt remove r8168-dkms && apt purge r8168-dkms", installed the 6.5 headers via "apt install proxmox-default-headers" and tried to install the r8168-dkms again. But there is a problem and it can't be build for the 6.5 kernel.


So I removed and purged r8168-dkms again.

Looks like it could work with the 6.2 Kernel (at least it tell the link is up) but my ZFS pool isn't compatible with that:


As with Kernel 6.5 it was always complaining that the device (my r8169 NIC was missing):


Now after rebooting I'm stuck at the initramfs and the initramfs-dropbear script is stuck in a error loop and I can't even locally unlock the root filesystem anymore...looks like I will have to boot some live linux und chroot into PVE to build a new initramfs...