Which network should use for Corosync?

Dec 28, 2019
9
0
1
26
Hi,

Currently, our Proxmox corosync in on public IP. Is this a best practice? A few articles mentioning about
'RING' while discussing corosync.


# corosync-cfgtool -s
Printing link status.
Local node ID 5
LINK ID 0
addr = 117.xxx.x.x
status:
nodeid 1: link enabled:1 link connected:1
nodeid 2: link enabled:1 link connected:1
nodeid 3: link enabled:1 link connected:1
nodeid 4: link enabled:1 link connected:1
nodeid 5: link enabled:1 link connected:1
nodeid 6: link enabled:1 link connected:1

Below is a result I got from an online article (which specifying RING)


[root@pcmk-1 ~]# corosync-cfgtool -s
Printing ring status.
Local node ID 1
RING ID 0
id = 192.168.122.101
status = ring 0 active with no faults


Is our configuration seems risky? Why the command ran on our proxmox installation not showing RING ID? In fact what the term RING really mean here?

Thanks in advance
 

aaron

Proxmox Staff Member
Staff member
Jun 3, 2019
646
50
28
Best practice is to have Corosync on a dedicated physical network that is just used for Corosync.
A 1GBit network is more than fast enough. Corosync doesn't need a lot of bandwidth but it really needs low latency.

If you have Corosync running on a network with other traffic, especially anything storage-related like NFS, Ceph, Backup, iSCSI,... you can easily run into the situation that the other traffic is congesting the network. This, in turn, increases the latency for the corosync packets and in a worst-case scenario the cluster will "fall apart" until the corosync services on each node can reach the others in a timely manner again.
Should you have any HA guests active on your nodes you will have the problem that they will fence themselves after 2 minutes without being part of the quorate cluster. If the whole cluster fell apart, this means that each node with HA guests on it will fence itself.

Additional rings, or links (it's the same in the context of corosync) increase the redundancy. Corosync will switch to another link if the main one cannot be used anymore. Corosync 3 (used since PVE 6.x) supports up to 8 links.
 
  • Like
Reactions: Moayad

t.lamprecht

Proxmox Staff Member
Staff member
Jul 28, 2015
2,393
365
103
South Tyrol/Italy
shop.maurer-it.com
Hi,

all corosync traffic is encrypted with an authkey only known by the cluster members (it is exchanged on join), so from a security stand point it doesn't really matter where it runs. Albeit public networks could be DDOS'd, so from a reliability and availability stand point it can be better to run it on a private network/LAN.

"Ring" or nowadays often also called "link" are a way of corosync to use more than one network for communicating, this allows to fallback to another if one network fails.

In general the most important thing is that the network on which corosync runs isn't used by IO traffic, as this can disrupt corosync easily. While corosync isn't using much bandwidth it really is sensitive to latency (spikes)

See also: https://pve.proxmox.com/pve-docs/chapter-pvecm.html#_cluster_network (and the rest of that chapter)
 
  • Like
Reactions: Moayad

t.lamprecht

Proxmox Staff Member
Staff member
Jul 28, 2015
2,393
365
103
South Tyrol/Italy
shop.maurer-it.com
  • Like
Reactions: adrian.jiang

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!