Proxmox VE cluster and ceph network

[tt01x]

Hello,

What would be the best practice for creating a Proxmox VE cluster (5 nodes) using Ceph?

All nodes have two 1G and two 25G ports.

Our idea is this:
ClusterLink0: 10.0.1.0 (this would be using 25G ports LACP)
ClusterLink1: 10.0.2.0 (this would be using 1G ports active-backup)
Ceph Public Network: 10.0.3.0 (this would be using 25G ports LACP)
Ceph Cluster Network: 10.0.4.0 (this would be using 25G ports LACP)
VM Migration Network: 10.0.5.0 (this would be using 25G ports LACP)

Is this configuration ok, or should Ceph Cluster Network be using 1G active-backup ports?
Or should ClusterLink1 (the backup cluster network) be using also the 25G LACP ports?


Br,
tt01x

 

[sb-jw]

I would configure ClusterLink1 as a single link. In principle, it only makes sense if it is a different switch. But I would avoid active backup.

I wouldn't separate CEPH either, I would just let everything run in one network. Ultimately, you only need this if you actually have several ports and you can and have to physically separate them because your cluster is so large and you have several clients connected.

See: https://docs.ceph.com/en/quincy/rad...k-config-ref/#network-configuration-reference

When it comes to switches, you should make sure that the LACP can work with L3+4 hashing and jumbo frames.

 

[tt01x]

sb-jw, thanks for your reply.

What about ClusterLink0? Should it be set to 25G ports or 1G ports?
If 1G, I assume your suggestion is to set it to single link as well?

Br,
tt01x

 

[sb-jw]

By "ClusterLink" I mean Corosync and Corosync should ideally have two separate paths. Therefore, ClusterLink0 should run on 25 G (LACP also works fundamentally differently than active-backup) and Link1 can run on 1 G if it is a separate switch. Otherwise, the added value may no longer exist.

 

[spirit]

corosync don't need to much bandwith (maybe 10-20mbits), but it's really sensitive to latency,

so if you mix it with another traffic (vms,...), you need to be 100% to not have any saturation on the link.

so you can use your 2 gigabits link (dedicated to corosync), without bonding, but 2 corosync links.

 

[tt01x]

Thank you both for your answers.