Cluster and Ceph network on 10G

trilljester

Member
Oct 9, 2018
10
0
6
49
I need to free up one of my 1G NICs on my cluster for a new network, however, I only have 1 free NIC and it's a 10G NIC. Currently I have my cluster network on a separate network. Would putting the cluster network on the Ceph network (10G) cause many issues? I can see it causing issues if it was on a 1G network, but with 10G, would that cause corosync issues and what not?
 
In general: corosync does not need a lot of bandwith but it really wants low latency.
If you have corosync on the same physical network with other services you always run the risk that the other services will congest the network which will increase the latency for corosync.

In the worst case the corosync communication will fail completely. If HA is enabled this will cause the nodes to fence themselves and the whole cluster might shut off.

So now to having corosync on the same physical 10G network with Ceph. Not the greatest idea because it is also not too hard to congest 10G with Ceph. If you really have no other way try to have multiple corosync links on different physical networks. This way corosync can switch to a different physical network in case of problems.

This is not ideal at all but if you cannot have a dedicated corosync network still your best bet to avoid single point of failues.

See our documentation on how to add more links to your corosync config [0].

[0]: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_redundancy
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!