10G NIC but No 10G Speed

V

virtualtechie

Member
Dec 12, 2022
19
3
8
Hey all. So I have PVE running on an HP Proliant DL360 G6 with Dual Xeon X5670 @ 2.93Ghz CPUs, 144GB of RAM, NC522SFP 10Gb Fiber NIC. I recently upgraded to 10Gb. I got a fiber SFP+ NIC installed and connected to a MicroTik CRS305-1G-4S+IN 4 Port 10G Switch. I have two other servers running TrueNAS, both with 10Gb NICs. When doing iPerf speed tests between either of the TrueNAS servers to the PVE host, I'm only getting around 6.5Gbps speeds.

I ran another iPerf test on the PVE host while running from a Live Linux disc and got 9.36Gbps. Something in Proxmox is preventing the NIC from getting the faster speeds. Looking for some advice/suggestions/opinions as to what within Proxmox is causing this.
 
Hey,

just so I understood this correctly,
Code: 
PVE <-> TrueNAS: 6.5Gbps
PVE <-> something else on the same network: 9.36Gbps
right? PVE, especially on the host itself, is just the linux networking stack. Do you have anything setup that would interact with the traffic?

What is ethtool <NIC> saying?
 
Hannes Laimer said:
Hey,

just so I understood this correctly,
Code: 
PVE <-> TrueNAS: 6.5Gbps
PVE <-> something else on the same network: 9.36Gbps
right? PVE, especially on the host itself, is just the linux networking stack. Do you have anything setup that would interact with the traffic?

What is ethtool <NIC> saying?
Click to expand...

Not quite...
PVE < - > TrueNAS: 6.5Gbps
TrueNAS < - > TrueNAS: 9.36Gbps

When running live disc on PVE host, I also get 9.36Gbps. So issue is only when running PVE on host, Nothing is setup that would interact with traffic. Everything is pretty vanilla as of right now.

root@pve:~# ethtool ens1f0
Settings for ens1f0:
Supported ports: [ TP FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: off
Port: FIBRE
PHYAD: 0
Transceiver: internal
Supports Wake-on: g
Wake-on: g
Link detected: yes
 
Last edited:
I see, thanks for the clarification! Hmm, my best guess is that during the test something else was happening on the network or the TrueNAS server. Did you repeat the iperf3 PVE<>TrueNAS with some time in between? If yes, did the BW change, or was it ~6.5 pretty constantly?
 
Hannes Laimer said:
I see, thanks for the clarification! Hmm, my best guess is that during the test something else was happening on the network or the TrueNAS server. Did you repeat the iperf3 PVE<>TrueNAS with some time in between? If yes, did the BW change, or was it ~6.5 pretty constantly?
Click to expand...
I’ve repeated the test numerous times and it’s the same results. Again, when I do the test with the live disc, I get the higher speeds. So it’s something within ProxMox. TrueNAS < - > TrueNAS I get the higher speeds every time I run the test.
 
hmmm, so there is nothing else going on on the network. Could you post the output of ip a and cat /etc/network/interfaces? Does ethtool <NIC> look different with the live disc? What kernel are you running on PVE and with the live disc?
 
Hannes Laimer said:
hmmm, so there is nothing else going on on the network. Could you post the output of ip an and cat /etc/network/interfaces? Does ethtool <NIC> look different with the live disc? What kernel are you running on PVE and with the live disc?
Click to expand...
Aside from regular network activity, can’t think of anything “specific” that’s impeding the speeds.

I’ll post the requested info once I’m back home and can check.
 
Do you have VLANs? Are all the devices you are testing all on the same VLAN? If, by chance, PVE and TrueNAS are on different VLANs, that could explain it.
 
louie1961 said:
Do you have VLANs? Are all the devices you are testing all on the same VLAN? If, by chance, PVE and TrueNAS are on different VLANs, that could explain it.
Click to expand...

No VLANs. LAN is pretty basic as of right now. Since upgrading to 10G I want to start configuring VLANs and such, but for right now everything is pretty vanilla.
 
You must log in or register to reply here.
Top Bottom