1GBps NIC stuck at 100MBps (and it's not the cable)

[nazgul]

Hi.

First time poster. I recently moved from bare metal Debian to a Proxmox setup on my HP G3 800 mini as I would like to run a couple of VMs in addition to Debian. To start with, I got a few LXCs up and running for now (Pihole, Homebridge, WireGuard) and loving the experience so far, except for one thing:

I can't figure this one out though - I seem to be stuck at 100MBps speeds (confirmed through iperf3 and ethtool installed on the host). This is a single NIC machine (Intel E1000), which supports 1GBps speeds otherwise.

It is not the cable nor the the router I think - it's the same cable/port on the router (using Mikrotik) that I used just prior when the machine ran Debian on bare metal.

I have already looked through a number of threads on here where people have faced similar issue, however, none of them seemed to work for me. What I tried already:
- forcing 1000mbps on the host machine via ethtool -s eth0 speed 1000 duplex full autoneg on made me lose LAN access to the host/LXCs
- forcing 1000mbps and turning off autoneg on the host machine via ethtool -s eth0 speed 1000 duplex full autoneg off also made me also lose LAN access to the host/LXCs
- forcing 1000mbps from the router side (ticking the 1000M options per the screenshot below) also made me lose LAN access to the host/LXCs
- confirmed the ethernet cable is still fine - plugging a different machine to it gets me full gigabit speed (confirmed through iperf3) and obviously it worked just fine for 1y+ when the machine was running Debian only. Switch back to Proxmox and the link is downgraded to 100MBps...

Any ideas please?

Thank you!


Couple of diagnostics:

dmesg grep -i "e1000"

[    1.336939] e1000e: Intel(R) PRO/1000 Network Driver
[    1.336940] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    1.337389] e1000e 0000:00:1f.6: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[    1.420381] e1000e 0000:00:1f.6 0000:00:1f.6 (uninitialized): registered PHC clock
[    1.490349] e1000e 0000:00:1f.6 eth0: (PCI Express:2.5GT/s:Width x1) [redacted]
[    1.490352] e1000e 0000:00:1f.6 eth0: Intel(R) PRO/1000 Network Connection
[    1.490480] e1000e 0000:00:1f.6 eth0: MAC: 12, PHY: 12, PBA No: [redacted]
[    1.693601] e1000e 0000:00:1f.6 eno1: renamed from eth0

ethtool eno0

        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 100Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        MDI-X: on (auto)
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

ethtool vmbr0

Settings for vmbr0:
        Supported ports: [  ]
        Supported link modes:   Not reported
        Supported pause frame use: No
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  Not reported
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Unknown! (255)
        Auto-negotiation: off
        Port: Other
        PHYAD: 0
        Transceiver: internal
        Link detected: yes

/etc/network/interfaces

auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.0.36/24
        gateway 192.168.0.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

iface wlp1s0 inet manual

Router settings for the port

 

[ness1602]

What is the output of:
dmesg |grep -i "eno1"

 

[nazgul]

What is the output of:
dmesg |grep -i "eno1"

Thanks. This is posted in my original post.

FWIW, here's the full output as of now (you can tell the couple of times the link went down was after I forced the 1000mbps and/or autoneg off):

[    1.693601] e1000e 0000:00:1f.6 eno1: renamed from eth0
[   94.758103] vmbr0: port 1(eno1) entered blocking state
[   94.758106] vmbr0: port 1(eno1) entered disabled state
[  100.882811] device eno1 entered promiscuous mode
[  273.545204] e1000e 0000:00:1f.6 eno1: NIC Link is Up 100 Mbps Half Duplex, Flow Control: None
[  273.545208] e1000e 0000:00:1f.6 eno1: 10/100 speed: disabling TSO
[  273.546432] e1000e 0000:00:1f.6 eno1: NIC Link is Down
[  275.163393] e1000e 0000:00:1f.6 eno1: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[  275.163397] e1000e 0000:00:1f.6 eno1: 10/100 speed: disabling TSO
[  275.163485] vmbr0: port 1(eno1) entered blocking state
[  275.163489] vmbr0: port 1(eno1) entered forwarding state
[ 7889.133112] e1000e 0000:00:1f.6 eno1: NIC Link is Down
[ 7889.133141] vmbr0: port 1(eno1) entered disabled state
[ 8961.531499] e1000e 0000:00:1f.6 eno1: NIC Link is Up 100 Mbps Half Duplex, Flow Control: None
[ 8961.531503] e1000e 0000:00:1f.6 eno1: 10/100 speed: disabling TSO
[ 8961.532744] e1000e 0000:00:1f.6 eno1: NIC Link is Down
[ 8963.150207] e1000e 0000:00:1f.6 eno1: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[ 8963.150211] e1000e 0000:00:1f.6 eno1: 10/100 speed: disabling TSO
[ 8963.150295] vmbr0: port 1(eno1) entered blocking state
[ 8963.150299] vmbr0: port 1(eno1) entered forwarding state
[308818.290918] e1000e 0000:00:1f.6 eno1: NIC Link is Down
[308818.290951] vmbr0: port 1(eno1) entered disabled state
[308837.884551] e1000e 0000:00:1f.6 eno1: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[308837.884554] e1000e 0000:00:1f.6 eno1: 10/100 speed: disabling TSO
[308837.884636] vmbr0: port 1(eno1) entered blocking state
[308837.884640] vmbr0: port 1(eno1) entered forwarding state

 

[ness1602]

[308837.884551] e1000e 0000:00:1f.6 eno1: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
As you can see here, try changing the cabling.

 

[nazgul]

[308837.884551] e1000e 0000:00:1f.6 eno1: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
As you can see here, try changing the cabling.

I will try changing the cable on Tuesday as at the moment I'd rather not disrupt a running process that requires internet connection.

That said, which bit in the message says it could be the cable? What makes me confident it might not be the cable (hence the title) is that (1) same cable ran at 1gbps prior to reformat from Debian bare metal to Proxmox and (2) when I realized it's connecting only at 100mbps with Proxmox, I tried the cable using my MacBook, which connected at 1gbps, thereby ruling out the cable immediately.

Is there something specific with Proxmox requiring a special cable to negotiate speeds at 1gpbs perhaps? The cable is 5m, CAT6, full copper.

 

[gbschenkel]

I got some problems with CAT6 Shielded with Ubiquiti UDM Pro, using old CAT5E U/UTP cable got it working. From what some people told me, if the CAT6 Shielded cable is not 100% well crimped, it will have issues. Try a CAT6 not shielded if you have or just a CAT5E. In my specific case the internet port(9) on UDM Pro was not running at 1GB, if I use this cable on the others lan ports(1-8), it worked. I had put a TP-Link switch between both internet and the modem until I swapped the cable.

 

[nazgul]

[308837.884551] e1000e 0000:00:1f.6 eno1: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
As you can see here, try changing the cabling.

Tried a different cable (CAT5E, also connects at gigabit using different computer), still 100mbps (have rebooted as well):

ethtool eno1 output:

Settings for eno1:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 100Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        MDI-X: on (auto)
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

Any other ideas? :/

 

[bobmc]

It might be useful to know what make/model of nic is installed on the host and what host driver is installed... is it actually an Intel E1000e?
lspci | grep 'Ethernet'


Would be interesting to know if a link between the host and a client laptop/desktop (i.e delete vmbr0 and put an IP directly on eno1 then run iperf against that IP) works as expected - and if not, do you have another switch to try?

 

[nazgul]

It's 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (5) I219-LM.

How would I go about assigning an IP directly to eno1? Could I just make eno1 'active' and assign it an IP without deleting vmbr0? Afraid of deleting vmbr0 and losing access points to all the LXCs I have set up...


EDIT:
A couple of more outputs from the console:

lspci -knn:

00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (5) I219-LM [8086:15e3]
        DeviceName: Onboard Lan
        Subsystem: Hewlett-Packard Company Ethernet Connection (5) I219-LM [103c:829a]
        Kernel driver in use: e1000e
        Kernel modules: e1000e

ethtool vmbr0:

Settings for vmbr0:
        Supported ports: [  ]
        Supported link modes:   Not reported
        Supported pause frame use: No
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  Not reported
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Unknown! (255)
        Auto-negotiation: off
        Port: Other
        PHYAD: 0
        Transceiver: internal
        Link detected: yes

 

[bobmc]

Looks like this is a kernel bug
https://bugzilla.kernel.org/show_bug.cgi?id=213377

there might also be some relevant info here
https://serverfault.com/questions/6...expectedly-detected-hardware-unit-hang#616623

 

[nazgul]

Thanks bobmc (and all).

I tried booting into Ubuntu from a USB drive just to rule out any potential funniness with proxmox kernel/drivers given the discussions on those links and I think there may be something physically wrong with the port now - under Ubuntu GUI I saw everything from 'no link' to 10mbps to 1gpbs after playing with different cables and ports on the router. I also went and updated the BIOS (was 2-3 revisions/c.1 year behind on updates) through the updater built into the BIOS and that seemed to struggle getting the connection as well - got quite a few network time outs inside Bios (!) which required a couple of re-plugings/reboots till it established the link and proceeded with the upgrade.

I actually have another one of the same machines that does not exhibit any of these symptoms, which is why I think this is a physical issue with the NIC now rather than a software one.

In the end, after a bunch of (forced) reboots from there being no link at all under proxmox host, it booted up now with a 1000mbps link established . No idea what went differently this time (I'm using the same cable/setup as in the original post). I'm sure this will revert back to 100mbps or no link in further reboots. Let's see how long I can go without a reboot of the host.