[SOLVED] PVE on Mini-PC with Realtek NIC got not connection (not the typical solutions)

[fsedarkalex]

Hi!

I have been reading and searching the web for about a week now and got no solution.
I got a Mini-PC utilizing an AMD Ryzen 5 3550h mobile processor as well as aRealtek GBit Ethernet Family network chip planned to be used as a home server.

Now I got the following issue(s):
Neighter the onboard Realtek LAN is working, nor any USB-Adapter I tried (Surface USB3-LAN, an RTL8152B 100mbit USB 2.0 adapter from MOBI LOCK and finally another USB3 gbit adapter with intagrated USB-Hub from "Techole")

"Not working" means for all of there asapters:

• They are detected, they get a name (eno1 for the integrated, enxXXXXXXXXXXXX - where the capital Xes are the mac address - for the USB-Adapters)
• They come "UP" in "ip link show"
• The interface does not get assigned an IP address (in the installer)
• BUT: There is no communication, neighter when added to vmbr0, nor when I set an IP/Netmask manually using ip addr add XXX.XXX.XXX.XXX/24 dev enxXXXXXXXXXXXX

The box came preinstalled with Windows 10, thus I reinstalled a clean Windows10 just downloaded and the integrated LAN works instantly without any setup or any issues.
The hardware can't be defective. The cable and switchport are okay, no VLAN tags applied (untagged network) and I double-checked the VLAN the port is member of (got a DHCP address assigned while running under windows)

I also tried a clean Debian install - Same issues reproducible.
I tried to compile and install the r8168 driver (the version I found in one thread here on the forum as well as the current from realtek website). It has been loaded, the nic has been recognized and alle the issues persisted - including usb lan adapters.
I tried resetting the uefi/bios settings, I tried legacy boot and I removed the wireless card and did all the tests from above again.

A side-note: The BIOS has hundred of options I have never heared of - I guess those are low level settings meant to be set and locked by manufacturers. Maybe some of those are causing trouble...

Hopefully someone got an idea where to look next... Thanks for any suggestions!

 

[obsolete]

Do you know which Realtek chipset you're working with?

I installed PVE on two Asrock 4x4 Box-4800U which come with two onboard NICs... one of them being the Realtek 8125.

Here's what I see when I enumerate my on-board devices:

root@pve-node2:~# lspci | grep Ethernet
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. Device 8125 (rev 05)
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0e)

The first device is the Realtek 8125 2.5GB ethernet adapter (no drivers present in the current version of the PVE kernel), and the second is a Realtek GBE ethernet adapter (has drivers in the current version of the PVE kernel).

Once you determine what device chipset you have, you can search for the appropriate driver and install it.

 

[fsedarkalex]

Oh forgot to tell this

lspci only shows
Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev15)

in dmesg I see that it loads RTL8168h/8111h (Edit: Remember, I already tried the r8168 module in two different versions without success.)

 

[fsedarkalex]

Hope bumping is okay after two days

Any more ideas?

 

[obsolete]

Honestly, I'm out of ideas...

When I installed the module for my 8125 adapter, I did something like this:

1. Installed kernel headers apt install pve-headers-$(uname -r)
2. Cloned a dkms version of the driver from github
3. Installed dkms apt install dkms
4. Ran the dkms install script in the dkms install project I cloned from github
5. Done

 

[mijanek]

@obsolete I would like to buy the same MiniPC for buliding 3 Proxmox clusters. Sofar I read, you need at least Kernel 5.9 to get them work properly but I don’t think Proxmox will switch the kernel in the neae future. I don’t need iommu which is one of the issues but I dislike the PWM issue I read about. Does your Asus 4x4 Box 4800U work properly in that way? I would like to use CephFS on them but I‘m affraid this could also have some issue on it.

 

[fsedarkalex]

I was able to solve the issue. IT was a combination of personal dumbness and a "feature" under windows...

My Switchport was configured to server tagged VLANs - I did not check that as it WAS WORKING under windows.
Looks liek the realtek driver automatically switches to tagged VLAN 1 if it detectes VLAN1 tagged pakcets on the link under windows...

This is so stupid, I hope this might help someone else having this issue.

I am now successfully running the device with the universal 8169 driver (the 8168 would also work though but I preferr to stay as near to the original system as possible so as long as I do not have issues, I'll stick with 8169)

 

[obsolete]

@obsolete I would like to buy the same MiniPC for buliding 3 Proxmox clusters. Sofar I read, you need at least Kernel 5.9 to get them work properly but I don’t think Proxmox will switch the kernel in the neae future. I don’t need iommu which is one of the issues but I dislike the PWM issue I read about. Does your Asus 4x4 Box 4800U work properly in that way? I would like to use CephFS on them but I‘m affraid this could also have some issue on it.

@mijanek You’re correct in that the 8125 driver isn’t included in the Linux kernel until 5.9. As mentioned above, I had to install the driver via dkms. After doing so, the 2.5G NIC functions perfectly. The only thing you’ll need to be aware of is that you’ll need to reinstall the driver if/when you upgrade the kernel.

Also, I have my 4x4 Box-4800U’s set up to run ceph using the NVMe drive for the RBDs (i.e. storage for VMs) and have dedicated the 2.5G NIC for a dedicated ceph network.

So, you can definitely do what you’re wanting with the 4x4 Box.

 

[mijanek]

Tanks for your quick response. So am I right, you do not experience any issue with the power management of the CPU (high fan load)? Do you know the approx power consumption of you system?

 

[obsolete]

Tanks for your quick response. So am I right, you do not experience any issue with the power management of the CPU (high fan load)? Do you know the approx power consumption of you system?

I haven't encountered any power management issues; I've had one of the nodes under a fairly high load (80% CPU utilization) and didn't notice any performance issues. I also didn't notice much fan noise.

Unfortunately, I haven't taken the time to determine power consumption of the nodes... but AnandTech did a comparison of the 4x4 Box-4800U (and other mini PCs) back in November: https://www.anandtech.com/show/16236/asrock-4x4-box4800u-renoir-nuc-review/11

 

[mijanek]

@obsolete Thanks you for your advises!
Mine 3x AsRock 4x4 4800U boxes have arrived and with the new Proxmox 6.4 and the pve-kernel-5.11 from testing repo these tiny computers are really awsome! I don't have to deal with the nic driver any more it works out of the box, even the WIFI and buetooth are now supported (however I don't need them)
The only think with the setup was the switch from the >>amdgpu<< to >>fbdev<< to get the setup (still on kernel 5.4) in X11 running.

 

[obsolete]

@mijanek Awesome! Glad to hear that it's working as you wanted!

I haven't applied the recent update to Proxmox 6.4 - I'll probably wait until next week because I need to use some of the VMs this week.

That's good to hear that the driver issues have been worked out with the test pvt-kernel-5.11 - I've been wanting to see if I could create a bridge using the WiFi adapters for use as a private network for the VMs across all three nodes (I want to set up a Windows domain for testing).

 

[Rattlehead.ie]

Hey guys.
I didnt want to create a new thread over something very similar. Have myself a new Asus PN50 and installing Proxmox on it. Like the OP if I do a
"ip addr show" I see my RealTek 2.5G 8125 but it's in a down state despite it being connected to a working switch port.
Further if I do a "lspci | grep Ethernet" I get

02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. Device 8125 (rev 05)

So it does look like it knows the device is there and what it is. I can do a force up, it brings the I/F up but no packets at all.
Am I still missing the drivers here? What can I do to get this work? The other thing to note is after every reboot this I/F stays down. Any help would be appreciated.

 

[fsedarkalex]

Can you post your /etc/network/interfaces? Does it contain an "auto xyz" line where xyz is the name of your interface?
Also does it work when you do "ip link up dev xyz"

 

[Rattlehead.ie]

Hi @fsedarkalex thanks for the quick response

auto lo
iface lo inet loopback

iface enp2s0 inet manual      <this is the Realtek 2.5G 8125

auto vmbr0
iface vmbr0 inet static
      address 172.16.252.5/28
      gateway 172.16.252.1
      bridge-ports enp2s0
      bridge-stp off
      bridge-fd 0

I have only tried "ip link set enp2s0 up"
This did bring the I/F into an up state, lights on the NIC showing, but still no input or output packets.
I did note your earlier comment about VLAN1. I have the PN50 attached to an untagged VLAN1 port that I have tested, works with my windows and any Ubuntu machine I plug in naturally. So this should work? or so I need to change tagging on the Asus NIC?

 

[fsedarkalex]

Oh you are right about the syntax... Got it wrong out of my mind... I am still one of those "ifup" users

is vmbr0 up?

Maybe you should open your own topic as this one here is marked as solved and also I noticed that my issue was a completely different thing. I was using a VLAN tagged port by accident and my proxmox was set to untagged (which is the default operating mode of a normal desktop switch)

 

[Rattlehead.ie]

Funny enough, if I do an "ip addr show" vmbr0 doesnt show in the list. Surely it should, shouldnt it?

 

[fsedarkalex]

I am wondering why my box shows its nic as eno1 while yours says enp... IMHO it should be named eno as it's an onboard nic...
But okay the system detected it I guess so it should be the correct name.

I dont think thats an issue but could you try to remove the extra whitespace after "gateway"?

 

[Rattlehead.ie]

Thats my bad, just a typo, there is no extra space between gateway and the address.

 

[Rattlehead.ie]

Ok least I know Im on the right track, but what dependency.
What I have been able to do is run
1. >systemctl stop networking
2. >systemctl start networking.

The vmbr0 and physical interface come up and work perfect, full connectivity.
However, if I reboot then the issue comes back and as the system is not expected to be sitting on my desk and in a rack, this wont do.

I have read about running
>systemctl mask ifupdown-pre.service

However this doesn't fix the issue when the system is rebooted.
Is there a way to identify what the dependency is and stop it from being checked so I can reboot this system without a monitor and keyobard attached.