10Gb connection between two proxmox only works at 1Gb

[xekhz]

Hello,
I have two proxmox's
One version 6 and one version 8

Proxmox version 6 as
interfaces
ens2f0 192.168.27.83 # ens2f0 10Gb
vmbr0 ens2f1 192.168.40.36/24 gateway 192.168.40.1 # ens2f1 10Gb

Proxmox version 8
ens3f1 192.168.27.84 # ens3f1 10Gb
vmbr0 eno1 192.168.39.85/24 gateway 192.168.39.1 # eno1 1Gb


Proxmox version 6 CONNECTS with interface ens2f0 to proxmox version 8 to interface ens3f1

| pve 6 | - ens2f0 ------ ens3f1 - | pve 8 |

The routings

PVE 6
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.40.1 0.0.0.0 UG 0 0 0 vmbr0
192.168.27.0 0.0.0.0 255.255.255.0 U 0 0 0 ens2f0
192.168.40.0 0.0.0.0 255.255.255.0 U 0 0 0 vmbr0

PVE8
root@node5:~# ip route
default via 192.168.39.1 dev vmbr0 proto kernel onlink
192.168.27.0/24 dev ens3f1 proto kernel scope link src 192.168.27.84
192.168.39.0/24 dev vmbr0 proto kernel scope link src 192.168.39.85


ETHTOOL INFO
PVE6
ethtool ens2f0
Settings for ens2f0:
Supported ports: [ TP ]
Supported link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: on
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
MDI-X: Unknown
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

ETHTOOL INFO
PVE8
ethtool ens3f1
Settings for ens3f1:
Supported ports: [ TP ]
Supported link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: on
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
MDI-X: Unknown
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes


The Question

Sending 5T from PVE6 to PVE8 at 10Gb

with File size: 5T at 10Gb/s Download/Upload time needed is 1 hrs, 6 mins, 40 secs

What i am sending... from pve6 ( 192.168.27.83 )

cat DISK | ssh root@192.168.27.84 ' cat - > /dev/DISK'



Im sending a 5T file and it is going for 7 Hours

What could be wrong... how can i achive the 10Gb/s connection...
Thanks


More INFO
Making iperf3 between pve 6 and pve 8

# iperf3 -c 192.168.27.83
Connecting to host 192.168.27.83, port 5201
[ 5] local 192.168.27.84 port 48568 connected to 192.168.27.83 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 809 MBytes 6.78 Gbits/sec 572 1.31 MBytes
[ 5] 1.00-2.00 sec 594 MBytes 4.98 Gbits/sec 264 1.39 MBytes
[ 5] 2.00-3.00 sec 771 MBytes 6.47 Gbits/sec 24 1.42 MBytes
[ 5] 3.00-4.00 sec 535 MBytes 4.49 Gbits/sec 575 1.45 MBytes
[ 5] 4.00-5.00 sec 928 MBytes 7.78 Gbits/sec 294 1.14 MBytes
[ 5] 5.00-6.00 sec 1.01 GBytes 8.68 Gbits/sec 193 1.40 MBytes
[ 5] 6.00-7.00 sec 512 MBytes 4.30 Gbits/sec 27 1.40 MBytes
[ 5] 7.00-8.00 sec 459 MBytes 3.85 Gbits/sec 23 1.41 MBytes
[ 5] 8.00-9.00 sec 309 MBytes 2.59 Gbits/sec 45 1.41 MBytes
[ 5] 9.00-10.00 sec 768 MBytes 6.44 Gbits/sec 19 1.45 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 6.56 GBytes 5.64 Gbits/sec 2036 sender
[ 5] 0.00-10.04 sec 6.56 GBytes 5.61 Gbits/sec receiver

iperf Done.

 

[spirit]

cat DISK | ssh root@192.168.27.84 ' cat - > /dev/DISK

don't use ssh, it's single threaded, and you can reach more than around 2gbit/s. (+the time for read the source disk && write to destination disk)

 

[xekhz]

How can i achive more transfer speed doing the same ?

 

[ness1602]

You can always use socat or netcat for faster tp

 

[AndyRH]

What does the performance of the disks look like?

 

[Dunuin]

One version 6 and one version 8

PVE6 is End-of-Life and not receiving security patches for over a year. You should think about upgrading it to version 7 or even 8.

 

[xekhz]

What does the performance of the disks look like?

They are 6Gbs sas, IO delay was always below 10%