recommended 100 Gbit/s NICs for Ceph

D

d147

Active Member
May 13, 2019
2
0
41
60
Hi,

I'm planning a HCI cluster with MikroTik CRS520-4XS-16XQ switches for the Ceph network. Are there recommendations regarding the manufacturer of the server NICs? The dual port NICs Broadcom P2100G and NVIDIA ConnectX-6 Dx EN are offered by the distributors. The former are less expensive, but as reliability has precedence I would pay for the more expensive NVIDIA/Mellanox if they are better. I don't want to risk instability within the storage network.

It would be nice if you could share your experiences.

Thanks and regards
 
At work, the order of precedence for reliability is:

1) Mellanox (CX-6)
2) Intel (E810)
3) Broadcom (NetXtreme/Thor)
 
I read that in the past E810 caused a lot of problems with hci proxmox cluster with ceph, but according with your list it seems that today you say that is reliable. It means that with proxmox 9.2 all the problems are solved?
 
giorgino9 said:
I read that in the past E810 caused a lot of problems with hci proxmox cluster with ceph, but according with your list it seems that today you say that is reliable. It means that with proxmox 9.2 all the problems are solved?
Click to expand...

So far, E810 issues have been resolved at work. As with all things, one must do their own due diligence and test their own infrastructure. YMMV.
 
  • Like
Reactions: Johannes S
Maybe think about used intel/cornelis opa (48x 100/115Gbit) or mellanox edr infiniband (36x 100Gbit) switches (+cables+cards) which provide higher netto bandwidth (opa rw 12,5/9,5GB/s nfs, ib rw 11,5GB/s nfs, both without rdma setup access needed for) with even less latency and less budget used where you even can get 3 switches with 1 as spare for your yet planned budget ... just my 2cent as tip
:)
 
Last edited:
Another +1 for Mellanox/NVIDIA.

I think the E810s are serviceable with modern driver and firmware. We had some big issues with these to start but they are better behaved now.
 
You must log in or register to reply here.
Top Bottom
Back