10G XG-C100F slower upload problem.

[zerocdv]

Hi, first off, sorry if this is in the wrong category.
I am kinda new to Proxmox and Linux in general but find it really useful, so I started tinkering a bit.

I have a Windows 10 (main workstation) and a "home server", an E5-2678V3 on a dual socket Chinese board.
My server is running a TrueNAS VM that has my Media library but since I bought x 2 Asus XG-C100F cards with 2 DAC cables I want to store also my work files to work over the network.

My issue is that I don't get 10G upload from the server to the Windows pc, it only reaches 6Gbits/sec after changing the MTU to 9000, in both Proxmox and Windows.
If I reverse the transfer, I get 9.76Gbits/Sec, so uploading from the Windows PC to Proxmox.

Proxmox -------> Windows

Connecting to host 192.168.1.152, port 5201
Reverse mode, remote host 192.168.1.152 is sending
[ 5] local 192.168.1.111 port 61757 connected to 192.168.1.152 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 704 MBytes 5.91 Gbits/sec
[ 5] 1.00-2.00 sec 705 MBytes 5.91 Gbits/sec
[ 5] 2.00-3.00 sec 706 MBytes 5.92 Gbits/sec
[ 5] 3.00-4.00 sec 700 MBytes 5.87 Gbits/sec
[ 5] 4.00-5.00 sec 702 MBytes 5.89 Gbits/sec
[ 5] 5.00-6.00 sec 698 MBytes 5.85 Gbits/sec
[ 5] 6.00-7.00 sec 708 MBytes 5.94 Gbits/sec
[ 5] 7.00-8.00 sec 705 MBytes 5.91 Gbits/sec
[ 5] 8.00-9.00 sec 705 MBytes 5.91 Gbits/sec
[ 5] 9.00-10.00 sec 707 MBytes 5.93 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.05 sec 6.88 GBytes 5.88 Gbits/sec 194 sender
[ 5] 0.00-10.00 sec 6.87 GBytes 5.90 Gbits/sec receiver


Windows -------> Proxmox

Connecting to host 192.168.1.152, port 5201
[ 5] local 192.168.1.111 port 61716 connected to 192.168.1.152 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.15 GBytes 9.88 Gbits/sec
[ 5] 1.00-2.00 sec 1.13 GBytes 9.70 Gbits/sec
[ 5] 2.00-3.00 sec 1.14 GBytes 9.80 Gbits/sec
[ 5] 3.00-4.00 sec 1.15 GBytes 9.86 Gbits/sec
[ 5] 4.00-5.00 sec 1.14 GBytes 9.81 Gbits/sec
[ 5] 5.00-6.00 sec 1.14 GBytes 9.78 Gbits/sec
[ 5] 6.00-7.00 sec 1.14 GBytes 9.82 Gbits/sec
[ 5] 7.00-8.00 sec 1.14 GBytes 9.83 Gbits/sec
[ 5] 8.00-9.00 sec 1.14 GBytes 9.82 Gbits/sec
[ 5] 9.00-10.00 sec 1.14 GBytes 9.83 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 11.4 GBytes 9.81 Gbits/sec sender
[ 5] 0.00-10.05 sec 11.4 GBytes 9.76 Gbits/sec receiver


Proxmox Network config

auto lo
iface lo inet loopback

iface enp5s0 inet manual

iface enp6s0 inet manual

auto ens14
iface ens14 inet manual
mtu 9000

auto vmbr0
iface vmbr0 inet static
address 192.168.1.152/24
gateway 192.168.1.1
bridge-ports ens14
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
mtu 9000


ens14 ethtool

root@pve:~# ethtool ens14
Settings for ens14:
Supported ports: [ FIBRE ]
Supported link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
2500baseT/Full
5000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
2500baseT/Full
5000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: on
Port: FIBRE
PHYAD: 0
Transceiver: internal
Supports Wake-on: pg
Wake-on: g
Current message level: 0x00000005 (5)
drv link
Link detected: yes



ens14 is the 10G NIC, I set it as the only active NIC since before I was having 10G to the server but just 1G from the server.
So I thought it was maybe sending the connection using the 1G NIC.

Sorry if there are some big issues I am not quite familiar with networking and much less under Linux, but I am trying to learn.
Any help would be appreciated.

 

[androme13]

Hi, try to add "-P 3" to your iperf command.

 

[zerocdv]

Thank you.

Yes using the -P 3 I get the whole 9.8, 9.7Gbits.
I think this is maybe related just to windows. I actually did a test with the the 2 cards between a Proxmox and a unRAID server and was getting the 10gb symmetrical.

I think I might leave it as is if no one else has something to try. I do not have experience tinkering with networks and don't want to make the issue worst.