I need to free up one of my 1G NICs on my cluster for a new network, however, I only have 1 free NIC and it's a 10G NIC. Currently I have my cluster network on a separate network. Would putting the cluster network on the Ceph network (10G) cause many issues? I can see it causing issues if it was on a 1G network, but with 10G, would that cause corosync issues and what not?
Cluster and Ceph network on 10G
- Thread starter trilljester
- Start date
In general: corosync does not need a lot of bandwith but it really wants low latency.
If you have corosync on the same physical network with other services you always run the risk that the other services will congest the network which will increase the latency for corosync.
In the worst case the corosync communication will fail completely. If HA is enabled this will cause the nodes to fence themselves and the whole cluster might shut off.
So now to having corosync on the same physical 10G network with Ceph. Not the greatest idea because it is also not too hard to congest 10G with Ceph. If you really have no other way try to have multiple corosync links on different physical networks. This way corosync can switch to a different physical network in case of problems.
This is not ideal at all but if you cannot have a dedicated corosync network still your best bet to avoid single point of failues.
See our documentation on how to add more links to your corosync config [0].
[0]: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_redundancy
If you have corosync on the same physical network with other services you always run the risk that the other services will congest the network which will increase the latency for corosync.
In the worst case the corosync communication will fail completely. If HA is enabled this will cause the nodes to fence themselves and the whole cluster might shut off.
So now to having corosync on the same physical 10G network with Ceph. Not the greatest idea because it is also not too hard to congest 10G with Ceph. If you really have no other way try to have multiple corosync links on different physical networks. This way corosync can switch to a different physical network in case of problems.
This is not ideal at all but if you cannot have a dedicated corosync network still your best bet to avoid single point of failues.
See our documentation on how to add more links to your corosync config [0].
[0]: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_redundancy