Mellanox ConnectX®-4 EN driver and server configuration review

Rares

Well-Known Member
Feb 28, 2017
75
9
48
Hi,

We want to build a 5 node PX cluster with Ceph storage. For this we are considering the SuperServer 1029U-TN10RT from SuperMicro equipped with:
- 2 x Intel 16 core CPU: SKL-SP 6130 16C/32T 2.1G 22M 10.4GT UPI
- 128 Gb RAM
- 4 x 1TB NVMe Intel DC P4510 PCIe 3.0 3D TLC 2.5"
- 2 x Mellanox ConnectX®-4 EN dual port 100Gbps controller
- 2 x Intel® X540 Dual Port 10GBase-T

All of these will be connected to a 25 Gb ports switches using 100Gb to 4x25Gb splitters. The bandwith will be as:
10 Gb - Public Internet
10 Gb - Migration VLAN, intranet, Corosync ring 3
75 Gb - Ceph private VLAN
5 Gb - Corosync ring 1
75 Gb - Ceph public VLAN
5 Gb - Management, Corosync ring 2

Do you have good experience with the drivers for Mellanox ConnectX®-4 EN dual port 100Gbps controller and Intel® X540 Dual Port 10GBase-T ?
Do you think this setup is ok and reliable for a hyper-converged cluster?
Do you have any other suggestions?

Thank you,
Rares
 
Hi,

Do you have good experience with the drivers for Mellanox ConnectX®-4 EN dual port 100Gbps controller and Intel® X540 Dual Port 10GBase-T ?
this both cards are in our kernel and are actively maintained.
 
this both cards are in our kernel and are actively maintained.

Dear Wolfgang,

Do you know if this cars is also supported?
Mellanox MCX516A-CDAT ConnectX-5 Ex EN Network Interface Card 100GbE Dual-Port QSFP28 PCIe4.0 x16

I am interested in the OVS offloading:
ConnectX-5 Accelerated Switching and Packet Processing (ASAP2™) technology enhances offloading of virtual switches and virtual routers, for example, Open V-Switch (OVS), which results in significantly higher data transfer performance without overloading the CPU. Together with native RoCE and DPDK (Data Plane Development Kit) support, ConnectX-5 dramatically improves Cloud and NFV platform efficiency.

Thank you,
Rares
 
Yes it should work.
It is not tested but the driver information says it is fine.

OVS offloading depends on DPDK and you have to compile it yourself.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!