CentOS 7 VM 10000baseT/Full Issue

[Joe Sudhoff]

I have installed the latest version of ProxMox onto a Dell R620 with a 2 port 10Gig NIC card and ProxMox recognizes it and shows it is connected at 10000Mb/ s, shown below:

CPU(s) 40 x Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz (2 Sockets)

Kernel Version Linux 5.0.15-1-pve #1 SMP PVE 5.0.15-1 (Wed, 03 Jul 2019 10:51:57 +0200)

PVE Manager Version pve-manager/6.0-4/2a719255

Settings for enp5s0f0:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes


The problem I am having is that installed CentOS 7 as a VM and ran a speed test via iperf and speedtest.net via python script and I can only achieve 1000Mb/ s speeds as shown below:

VMware vmxnet3

Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Sonic.net, Inc. (San Jose, CA) [33.83 km]: 2.179 ms
Testing download speed................................................................................
Download: 503.83 Mbit/s
Testing upload speed................................................................................................
Upload: 418.27 Mbit/s

e1000

Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Sonic.net, Inc. (San Jose, CA) [33.83 km]: 2.832 ms
Testing download speed................................................................................
Download: 610.06 Mbit/s
Testing upload speed................................................................................................
Upload: 532.32 Mbit/s

VirtIO (paravirtualized)

Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Sonic.net, Inc. (San Jose, CA) [33.83 km]: 2.696 ms
Testing download speed................................................................................
Download: 591.29 Mbit/s
Testing upload speed................................................................................................
Upload: 563.22 Mbit/s

Realtek RTL8139

Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Sonic.net, Inc. (San Jose, CA) [33.83 km]: 1.977 ms
Testing download speed................................................................................
Download: 277.71 Mbit/s
Testing upload speed................................................................................................
Upload: 138.68 Mbit/s

I have tried all 4 adapters and they all seem to have the same speeds. No matter what I do it will not connect at 10Gig. Below is the CenttOS 7 Interface:

Settings for ens18:
Supported ports: [ TP ]
Supported link modes: 1000baseT/Full
10000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
MDI-X: Unknown
Supports Wake-on: uag
Wake-on: d
Link detected: yes

The interface shows Supported link modes: 10000baseT/Full but the speed shows 1000Mb/s.

Thanks in advance for assistance.

Joe

 

[t.lamprecht]

I have tried all 4 adapters and they all seem to have the same speeds. No matter what I do it will not connect at 10Gig.

I mean, for speedtest to really show 10 GBit/s (which would be 1.25 GByte/s) your connection to the internet and the speedtest server would need to both have also 10 GBit/s available. While that can be possible I really do not believe it is, even if the Speedtest server had 10GBit/s, anybody else doing a speedtest would limit that too.

So, from your posted results I'd rather guess that the servers connection to the WAN is about 500 MBit/s to 1 GBit/s..
I mean you easily could compare the results to a test run executed on the PVE server itself, if those do not significantly differ you know they can both achieve the same rate (to the internet).

The interface shows Supported link modes: 10000baseT/Full but the speed shows 1000Mb/s.

The speed is rather virtual, for it's used as it's common, but the kernel won't stop sending packets once 1000MBit/s are being send, it sends as much as possible (if no traffic control/shaping is active, which by default isn't).
I for example, I can do >60GBit/s in a local test (between two VMs with NIC on the same Linux Bridge), effectively limited by the speed of my RAM.

ran a speed test via iperf

How did you ran the iperf test? That one needs also be done between to hosts which are connected both over 10GBit/s links.
So the PVE server and VM are already connected over such a NIC, that should be OK. Then the other side of the test in the LAN needs also be connected with a 10 GBit/s NIC and finally, if the two are not connected directly, the switch between needs also to have 10 GBit/s ports where both are connected too. Else, the test does not makes sense...

 

[Joe Sudhoff]

t.lamprecht

Thanks for the quick reply. Below is how it is physically hooked up. It is on the same Nexus switch but 2 different 10Gig ports.

PVE(A)-10Gig-------NEXUS 10Gig Port 10---NEXUS 10Gig Port 20-------10Gig-PVE(B)

I did run iperf from 1 pve to the other and below is what I got:
**All IP's have been replaced with "x" for privacy and using "A" and "B" as the names.**

Client Side:
root@proxmoxB:~# iperf -c xxx.xxx.xxx.xxx
------------------------------------------------------------
Client connecting to xxx.xxx.xxx.xxx, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local xxx.xxx.xxx.xxx port 48520 connected with xxx.xxx.xxx port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 2.42 GBytes 2.08 Gbits/sec

Server Side
root@proxmoxA:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 128 KByte (default)
------------------------------------------------------------
[ 4] local xxx.xxx.xxx.xxx port 5001 connected with xxx.xxx.xxx.xxx port 48520
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 2.42 GBytes 2.08 Gbits/s

So with the above it looks like it is doing 10 gig from 1 proxmox to the other. This switch also has 2 10Gig paths out to the internet.

For the CentOS I ran a python script I found on a website that allow you to speed test using Speedtest.net so if it is like the GUI OOKLA server test then it should do 10gig, right?

As far as the RAM, the Dell Poweredge R620 has 128Gig of RAM in which each CentOS VM is allocated 60Gig. There are 2 VM's on this and will only be 2 VM's on this, both being CentOS 7.

So even tho CentOS is reporting it is at 1Gig, it could be doing more?

Thanks again

Joe

 

[t.lamprecht]

For the CentOS I ran a python script I found on a website that allow you to speed test using Speedtest.net so if it is like the GUI OOKLA server test then it should do 10gig, right?

In the best case yes, but I often had such scripts and test report less or more than available, so I would not put too much trust in them.

So even tho CentOS is reporting it is at 1Gig, it could be doing more?

Yes. You could repeat the iperf test but move one site of the test into the CentOS VM, the other side can stay on the proxmox (but should not be on the same as the CentOS VM you use for testing).