CEPH with mixed-rate (10g/25g) nodes - bad idea?


Active Member
Jan 24, 2019

So we've got a relatively small CEPH NVMe cluster consisting of 4x nodes, each with a Samsung datacenter 3.8TB M.2 SSD inside. Each node has 4x 10g connections, 2x in LACP for normal traffic, 2x in LACP for CEPH traffic. Connected using a pair of S4048-ON switches.

We're looking at moving towards 25G, as two of these nodes have Intel 25G NICs in them, and are running at only 10G with the current switches. However, we're not ready to move to full-25G on all NICs yet. This means we'd have two nodes on 10G, and the other two nodes on 25G.

My question is, how well does CEPH handle situations like this? If a 25G node starts sending traffic to a 10G node at >10Gbps, that obviously risks buffer issues. If we decide to go with a switch that does not have a very large packet buffer (models seem to either have ~16MB or like 8GB, but not in between), are we likely to have notable issues?



The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!