Proxmox 8.1.4 10gbe not working properly...

[Dariusz]

Hey

Pretty much this is my issue > (right now my cable is only 5gbe... another strange issue where cat7 cable does not push 10gbe speeds, anyway...)
Sending is as expected 5gb (Current cable limit)

iperf3.exe -c 192.168.3.30 -n 2G
Connecting to host 192.168.3.30, port 5201
[  5] local 192.168.3.11 port 58006 connected to 192.168.3.30 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   581 MBytes  4.87 Gbits/sec
[  5]   1.00-2.01   sec   582 MBytes  4.83 Gbits/sec
[  5]   2.01-3.01   sec   588 MBytes  4.93 Gbits/sec
[  5]   3.01-4.00   sec   297 MBytes  2.51 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-4.00   sec  2.00 GBytes  4.29 Gbits/sec                  sender
[  5]   0.00-4.01   sec  2.00 GBytes  4.29 Gbits/sec                  receiver

But receiving is bad :- (

iperf3.exe -c 192.168.3.30 -n 2G -R
Connecting to host 192.168.3.30, port 5201
Reverse mode, remote host 192.168.3.30 is sending
[  5] local 192.168.3.11 port 58549 connected to 192.168.3.30 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec   138 MBytes  1.15 Gbits/sec
[  5]   1.01-2.00   sec   177 MBytes  1.49 Gbits/sec
[  5]   2.00-3.00   sec   175 MBytes  1.47 Gbits/sec
[  5]   3.00-4.01   sec   185 MBytes  1.53 Gbits/sec
[  5]   4.01-5.01   sec   177 MBytes  1.49 Gbits/sec
[  5]   5.01-6.01   sec   178 MBytes  1.50 Gbits/sec
[  5]   6.01-7.01   sec   180 MBytes  1.51 Gbits/sec
[  5]   7.01-8.01   sec   180 MBytes  1.51 Gbits/sec
[  5]   8.01-9.01   sec   178 MBytes  1.50 Gbits/sec
[  5]   9.01-10.00  sec   179 MBytes  1.50 Gbits/sec
[  5]  10.00-11.00  sec   177 MBytes  1.49 Gbits/sec
[  5]  11.00-12.00  sec   124 MBytes  1.04 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-12.00  sec  2.00 GBytes  1.43 Gbits/sec  33729             sender
[  5]   0.00-12.00  sec  2.00 GBytes  1.43 Gbits/sec                  receiver

I tried jumbo frames going from 4000 to 16000, tried MTU on proxmox to 9000, enabled jumbo frames on switch > T1700G-28TQ

I'm lost,

When I do test on windows > VM > TrueNas, I get 120/110 MBytes download, its even worse!

Any ideas?

 

[_gabriel]

is iperf3 between two VMs ?
if yes, physical link isn't used, it's CPU tied link.

120/110 MBytes

= 1 Gbits/s link

 

[Dariusz]

No, it's direct proxmox to windows. U can see that from windows to proxmox speed is 581 MBytes but from proxmox to windows 180.

I installed iperf with apt get on proxmox directly

Proxmox > cable > windows system. 10gbe cards etc. Separate systems. Ots strangr that one transfer is 1.7 faster than other.

No vm test here

 

[_gabriel]

what are the NIC model ?

 

[Dariusz]

what are the NIC model ?

I have been trying to figure it out from start, is there any command that can tell me that?
Its add on card, 2x 10gbe sfp+ from ebay, I've had it for a while now...
I think its a flavor of > Intel X520-DA2 2x SFP+ 10GB May be HP or Dell tho... I have 2 of them, I was considering swapping them to see if its card problem related...

 

[Dariusz]

Ok I tried another card, same issue, upload to proxmox fast, download from it slow.

 

[_gabriel]

try with ubuntu live and a debian live.

 

[Dariusz]

try with ubuntu live and a debian live.

Why? If native proxmox is failing, when I tried in TrueNas & iperf I'm already loosing speed going from 180 to 110-120 :/
I'll try some proxmox tunables but its looking grim ;c

 

[_gabriel]

I mean try ubuntu live and a debian live on host directly not in VM, that's what I will do to dig out.

 

[Dariusz]

I mean try ubuntu live and a debian live on host directly not in VM, that's what I will do to dig out.

isnt that what proxmox run on ? I installed iperf on proxmox directly and do tests with that. I don't test vms. Just direct proxmox iperf to windows PC... going via VM is even slower.

 

[_gabriel]

yes but the kernel is customized.
testing debian less recent kernel and Ubuntu latest kernel can help to find root cause.
a pure Windows help too, to exclude cable.