PVE Networking stuck at 1Gb/s with 10Gb link

[Cocolin]

Hi,
I have a probleme with my proxmox cluster, I have a bond for CEPH derectly connected betwen the nodes in 10Gb and a bond for public network with 10Gb link to.
But when i do an iperf with public network I have my 10Gb/s but not with my CEPH network.

root@arobase-pve1:~# iperf3 -B 10.28.11.11 -c 10.28.11.12
Connecting to host 10.28.11.12, port 5201
[ 5] local 10.28.11.11 port 50955 connected to 10.28.11.12 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 114 MBytes 954 Mbits/sec 0 378 KBytes
[ 5] 1.00-2.00 sec 113 MBytes 947 Mbits/sec 0 378 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 941 Mbits/sec 0 378 KBytes
[ 5] 3.00-4.00 sec 112 MBytes 940 Mbits/sec 0 378 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 943 Mbits/sec 0 378 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 936 Mbits/sec 0 378 KBytes
[ 5] 6.00-7.00 sec 113 MBytes 947 Mbits/sec 0 378 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 940 Mbits/sec 0 378 KBytes
[ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 0 481 KBytes
[ 5] 9.00-10.00 sec 113 MBytes 947 Mbits/sec 0 481 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 944 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec receiver

root@arobase-pve1:~# iperf3 -B 172.28.111.11 -c 172.28.111.12
Connecting to host 172.28.111.12, port 5201
[ 5] local 172.28.111.11 port 36817 connected to 172.28.111.12 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.08 GBytes 9.31 Gbits/sec 1294 1.02 MBytes
[ 5] 1.00-2.00 sec 884 MBytes 7.41 Gbits/sec 311 1.02 MBytes
[ 5] 2.00-3.00 sec 1.07 GBytes 9.22 Gbits/sec 1754 786 KBytes
[ 5] 3.00-4.00 sec 876 MBytes 7.35 Gbits/sec 854 646 KBytes
[ 5] 4.00-5.00 sec 1.09 GBytes 9.35 Gbits/sec 403 675 KBytes
[ 5] 5.00-6.00 sec 1.09 GBytes 9.35 Gbits/sec 711 653 KBytes
[ 5] 6.00-7.00 sec 1.09 GBytes 9.33 Gbits/sec 227 701 KBytes
[ 5] 7.00-8.00 sec 876 MBytes 7.35 Gbits/sec 1001 345 KBytes
[ 5] 8.00-9.00 sec 1.09 GBytes 9.35 Gbits/sec 355 1.08 MBytes
[ 5] 9.00-10.00 sec 1.09 GBytes 9.33 Gbits/sec 873 731 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.2 GBytes 8.74 Gbits/sec 7783 sender
[ 5] 0.00-10.00 sec 10.2 GBytes 8.74 Gbits/sec receiver

root@arobase-pve1:~# ethtool ens1f0
Settings for ens1f0:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: off
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Supports Wake-on: g
Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

root@arobase-pve1:~# ethtool ens2f1
Settings for ens2f1:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: off
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

root@arobase-pve1:~# ethtool ens2f0
Settings for ens2f0:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: off
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Supports Wake-on: g
Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

Before this configuration I haved a LACP 4x1Gb LACP for my public network and CEPH worked at 10Gb/s but LACP whas stucked at 1Gb/s.


I fured out that all my clusters with LACP doesn't as it should be, but they are all stuck at 1Gb/s.

Thanks for your help.

 

[sb-jw]

I don't see anywhere that you have LACP configured. Either the interface is not included in the screenshot or you are not using LACP.

 

[Cocolin]

I'm sorry, I probably didn't explain it good, my English is bad.
The LACP was before, because my LACP was stuck a 1Gb/s . I changed it from LACP to 10Gb link direct to my switch. And since I did this, the CEPH become stuck at 1Gb/s.

 

[sb-jw]

Your CEPH network is 10.1.1.1/24 and not 10.28.11.1/24? The first iperf is at least not for your CEPH network, but for your VoIP.

 

[Cocolin]

This is a really good point. I'm just dumb.

I was really convinced that my CEPH was stuck a 1Gb/s because I can't have the same disk speed as another cluster with the same proxmox configuration, same VM configuration... And the disk speed on my VM in this cluster is stuck a 1Gb/s when in my other cluster disk speed can go up to 15Gb/s.

 

[sb-jw]

I was really convinced that my CEPH was stuck a 1Gb/s because I can't have the same disk speed as another cluster with the same proxmox configuration, same VM configuration...

You're talking about hard drive speed here, which means you use 1 GB/s x 8 to get the network bandwidth of 8 GbE. So it's pretty much due to the 10 GbE interface, theoretically no more than 1.25 GB/s is possible.

And the disk speed on my VM in this cluster is stuck a 1Gb/s when in my other cluster disk speed can go up to 15Gb/s.

But we are probably talking about 2x 100 GbE interface and NVMe. Or it's just local storage with a RAID controller and cache with BBU.