Mellanox ConnectX®-4 EN driver and server configuration review

[Rares]

Hi,

We want to build a 5 node PX cluster with Ceph storage. For this we are considering the SuperServer 1029U-TN10RT from SuperMicro equipped with:
- 2 x Intel 16 core CPU: SKL-SP 6130 16C/32T 2.1G 22M 10.4GT UPI
- 128 Gb RAM
- 4 x 1TB NVMe Intel DC P4510 PCIe 3.0 3D TLC 2.5"
- 2 x Mellanox ConnectX®-4 EN dual port 100Gbps controller
- 2 x Intel® X540 Dual Port 10GBase-T

All of these will be connected to a 25 Gb ports switches using 100Gb to 4x25Gb splitters. The bandwith will be as:
10 Gb - Public Internet
10 Gb - Migration VLAN, intranet, Corosync ring 3
75 Gb - Ceph private VLAN
5 Gb - Corosync ring 1
75 Gb - Ceph public VLAN
5 Gb - Management, Corosync ring 2

Do you have good experience with the drivers for Mellanox ConnectX®-4 EN dual port 100Gbps controller and Intel® X540 Dual Port 10GBase-T ?
Do you think this setup is ok and reliable for a hyper-converged cluster?
Do you have any other suggestions?

Thank you,
Rares

 

[wolfgang]

Hi,

Do you have good experience with the drivers for Mellanox ConnectX®-4 EN dual port 100Gbps controller and Intel® X540 Dual Port 10GBase-T ?

this both cards are in our kernel and are actively maintained.

 

[Rares]

Thank you Wolfgang for your replay.

Regards,
Rares

 

[Rares]

this both cards are in our kernel and are actively maintained.

Dear Wolfgang,

Do you know if this cars is also supported?
Mellanox MCX516A-CDAT ConnectX-5 Ex EN Network Interface Card 100GbE Dual-Port QSFP28 PCIe4.0 x16

I am interested in the OVS offloading:

ConnectX-5 Accelerated Switching and Packet Processing (ASAP2™) technology enhances offloading of virtual switches and virtual routers, for example, Open V-Switch (OVS), which results in significantly higher data transfer performance without overloading the CPU. Together with native RoCE and DPDK (Data Plane Development Kit) support, ConnectX-5 dramatically improves Cloud and NFV platform efficiency.

Thank you,
Rares

 

[wolfgang]

Yes it should work.
It is not tested but the driver information says it is fine.

OVS offloading depends on DPDK and you have to compile it yourself.

 

[spirit]

I'm running a lot of connect-x4 && connect-x5 card. 2x25gb , works pretty fine. no driver bug.