network recommendations for ceph

[jeronimo.asensio]

We are designing a hyperconverged solution with Proxmox+CEPH and I would like you to guide me with the network part. We are going to use NVME and SSD disk storage, we will start with 3 nodes with the perspective of growing, I have experience in other HCI systems like NUTANIX, but not CEPH and from what I am reading the network needs are quite high, not only in latency but also in capacity. Based on your experiences, would 4x10GiB SFP+ per node be enough? Or should I go to 25G or 40G networks?

Thank you very much!

 

[spirit]

if you use lacp with layer3+4, it should loadbalance fine across multiple 4x10gbit links. (each vms is connecting to each osd, and each osd connect to each other osd. That's give you a lot of tcp connections. (1 connection can use only 1link).

If course, if you can have 25gb or 40gb network, it's better. (but you need to have some kind of bond for availability, as ceph don't have any multipath implemention).

I'm running ceph nodes with 2x10gb, it's working fine for my workload. (small iops, not too much bandwidth). Only rebuild of disk or rebalancing of datas can really push nic to the limit, but you can configure it to limit speed.