Older HP DL380 G7 NIC speed now limited to 100 Mb/s, wasn't before

[Jack_9846]

I have an HP server here that was recently built as a PVE 5.3 machine. About 8 months ago this machine was built as a PVE 5.2 machine (sorry, I can't be more specific than that, there's no record of which exact version). It was then wiped and sat unused until now.

The box has a Broadcom 1 Gb/s controller (see below for full model). Previously there was no issue with speed, it could easily hit the advertised 1 Gb/s advertised speed (and get over 100 MB/s with local file transfers). With the new install though the adapters are limited to 100 Mb/s (and ~11 MB/s in transfers). Also ethtool shows that the speed of the NIC is 100 Mb/s.

These speeds were tested with iperf between boxes on the LAN, and observed with rsync (from a mounted nfs drive, from the same box that used to get the 100 MB/s pre-wipe). There have been no changes to the network over that time.

The NIC drivers only come with rpm package of course, I've tried building them from source, but without success (the scripts provided run into an error which apparently is caused by a bug in gcc, of which I am not knowledgeable enough to fix). But since it was working before out of the box, I'm wondering if it is possible that the drivers for the NIC were messed up between updates? Or maybe there is some other reason.

Thank you for any help/insight you can provide!

Spoiler: lspci -v | grep net

03:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
03:00.1 Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
04:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
04:00.1 Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

Spoiler: ethtool enp3s0f0

Settings for enp3s0f0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 100Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: off
Supports Wake-on: g
Wake-on: g
Link detected: yes

Spoiler: pveversion -v

proxmox-ve: 5.3-1 (running kernel: 4.15.18-9-pve)
pve-manager: 5.3-8 (running version: 5.3-8/2929af8e)
pve-kernel-4.15: 5.3-1
pve-kernel-4.15.18-10-pve: 4.15.18-32
pve-kernel-4.15.18-9-pve: 4.15.18-30
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-43
libpve-guest-common-perl: 2.0-19
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-36
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-2
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-22
pve-cluster: 5.0-33
pve-container: 2.0-33
pve-docs: 5.3-1
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-17
pve-firmware: 2.0-6
pve-ha-manager: 2.0-6
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-1
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-44
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1

 

[sumsum]

We are Running a few Proliant G7 Servers with the same NIC and PVE 5.3. so far w/o any issues. Have you tried to Check/replace the network cables? Or Even change the Switch Port.

 

[Jack_9846]

We are Running a few Proliant G7 Servers with the same NIC and PVE 5.3. so far w/o any issues. Have you tried to Check/replace the network cables? Or Even change the Switch Port.

Yep, I've verified with a laptop plugged into the same cable that 1 Gb/s is working for the cable/switch/etc. This one has definitely stumped me (it's not urgent or anything though, it's just a test box).

 

[czechsys]

Try different kernel.

 

[Jack_9846]

Update: Turns out swapping out the cable the cable fixed it, but I'm even more confused now. The cable seemed to be selectively biased against linux (???). Two different windows machines could get 1 GB/s when connected, but no linux machine (RHEL and ubuntu) could when connected to the same cable. This includes a laptop set up with a dual boot. When booted into windows 7 it worked, when booted into ubuntu it didn't.

This was all fixed by swapping out the cable (for a different cat 5e, using the same port). I think that cable might have been haunted.