You need two corosync links. For 12 nodes on gigabit I would use dedicated links for both, just in case, even if having it just for Link0 would be enough. Max I've got in production with gigabit corosync is 8 hosts, no problems at all.
Proxmox with 48 nodes
- Thread starter Nathan Stratton
- Start date
The number of cluster members is an inexact limit. that ACTUAL limit has to do with how much data the cluster members have to keep synchronized- if each of your cluster members had 400vms with continuous api traffic- your cluster would probably die due to tripping timeouts. if you have 5vms mostly sitting there and not being molested you'll be just fine.
Just for the record the VM config sizing and changes are related to the PXMCFS https://pve.proxmox.com/wiki/Proxmox_Cluster_File_System_(pmxcfs)
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
A status update on this:
Two corosync parameters that are especially relevant for larger clusters are the "token timeout" and the "consensus timeout". When a node goes offline, corosync (or rather the totem protocol it implements) will need to go through at least one full "token timeout" and one full "consensus timeout" to form a membership without that node.
The token timeout is, by default, calculated based on the number of nodes multiplied by the "token coefficient" (plus a base value), see [0] for the exact formula. The consensus timeout is, by default, defined as 1.2 * token timeout [1].
To avoid unwanted node fencing if HA is active on that node, these two timeouts together should not become much larger than 40-45 seconds. Both timeouts depend on the number of nodes. For up to ~25 nodes, the default timeouts are within that range and no adjustment is necessary. On larger (approximately more than 25 nodes) clusters it is necessary to lower these timeouts.
We are currently performing tests to come up with general recommendations how to tweak the corosync configuration to achieve this. But this will not be ready before the holidays. At the moment, decreasing the token coefficient (which is 650ms by default) seems like the most promising option. We are still performing tests to find a good default. Currently, our tests suggest it should be safe to lower token the coefficient to 125ms (if the network is stable with low latency), but going to 325ms should already be enough for clusters with 30-40 nodes as discussed here.
Until our tests have concluded, if you would like to help and if you have the resources to test, one option would be to test whether decreasing the token coefficient solves the issue of unwanted node fencing in case a node goes offline for you, and otherwise maintains cluster stability as well.
If you want to test that: First make sure your cluster network fulfills the requirements [2], most importantly, stable low latency (<5ms is required, but I'd recommend <1ms). Then, edit
/etc/pve/corosync.conf [3] and add inside the totem section the option token_coefficient, for example token_coefficient: 125 which will lower the token coefficient to 125ms. Of course you can also set other values (as noted above, token_coefficient: 325 should be enough for the clusters in the range of 30-40 nodes) or go back to the default. Don't forget to increase config_version [3]. You need to restart corosync on each node for the change to fully take effect.If you test a custom token coefficient, we would greatly appreciate if you report back with your observations.
In general, and as noted by others, going for several smaller clusters instead of one larger cluster can have other operational advantages, especially if you also use the Proxmox Datacenter Manager which recently saw the stable 1.0 release [4].
[0] https://manpages.debian.org/trixie/corosync/corosync.conf.5.en.html#token_coefficient
[1] https://manpages.debian.org/trixie/corosync/corosync.conf.5.en.html#consensus
[2] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_cluster_network_requirements
[3] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_edit_corosync_conf
[4] https://pdm.proxmox.com/