4 node cluster - very slow network/ceph speeds

[panman_gr]

Hello everybody,

I have a 4 node cluster:

• Identical HP DL380 G9 hosts
• dual Xeon E5-2680v3 CPUs
• 128GB of RAM per host
• HP 640FLR-SFP28 (i.e. Mellanox Connectx-4 LX) dual 25Gbit NICs, updated to the latest 14.32.10.10 firmware from HP
• A Ceph cluster with one NVME OSD per host.

All 4 hosts are connected to a Mikrotik CRS504 switch via Mikrotik's breakout DAC cables, and the switch reports successful connections at 25Gbits.

There is no bonding, one 25Gbit port per host is used for networking, the other is for Ceph's internal network.

This is the ethtool output for one of the networking NICs on a host (all 8 ethtool outputs are identical):

Settings for eno49np0:
        Supported ports: [ Backplane ]
        Supported link modes:   1000baseKX/Full
                                10000baseKR/Full
                                25000baseCR/Full
                                25000baseKR/Full
                                25000baseSR/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: None        RS      BASER
        Advertised link modes:  1000baseKX/Full
                                10000baseKR/Full
                                25000baseCR/Full
                                25000baseKR/Full
                                25000baseSR/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: None
        Speed: 25000Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: g
        Wake-on: g
        Link detected: yes

However, backup/restore/migrate/ceph traffic speeds never exceed 400Mbits, even host-to-host, as you can see in this iperf3 dump:

Connecting to host 10.0.0.3, port 5201
[  5] local 10.0.0.4 port 58302 connected to 10.0.0.3 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  48.8 MBytes  0.41 Gbits/sec   40    204 KBytes       
[  5]   1.00-2.00   sec  47.8 MBytes  0.40 Gbits/sec   32    154 KBytes       
[  5]   2.00-3.00   sec  48.0 MBytes  0.40 Gbits/sec    8    175 KBytes       
[  5]   3.00-4.00   sec  48.2 MBytes  0.40 Gbits/sec   14    136 KBytes       
[  5]   4.00-5.00   sec  47.7 MBytes  0.40 Gbits/sec    9    148 KBytes       
[  5]   5.00-6.00   sec  47.7 MBytes  0.40 Gbits/sec    9    180 KBytes       
[  5]   6.00-7.00   sec  47.8 MBytes  0.40 Gbits/sec   14    197 KBytes       
[  5]   7.00-8.00   sec  48.4 MBytes  0.41 Gbits/sec   10    199 KBytes       
[  5]   8.00-9.00   sec  47.7 MBytes  0.40 Gbits/sec   13    165 KBytes       
[  5]   9.00-10.00  sec  48.1 MBytes  0.40 Gbits/sec   11    189 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   480 MBytes  0.40 Gbits/sec  160             sender
[  5]   0.00-10.00  sec   479 MBytes  0.40 Gbits/sec                  receiver

Any ideas as to why this happens?
Thanks in advance.

 

[mfed]

I know in certain mikrotik misconfigurations the traffic needs to be processed by CPU resulting in very poor performance... You can check if this is the case by running 'system resource monitor' on mikrotik, then your iperf3 and watch the CPU. In properly configured mikrotik there should be no CPU spike...