Very Large Proxmox cluster possibility

[wahmed]

I understand Proxmox is tested in a 16 node cluster lab. Although there are no hard coded limit, Redhat and Suse limits to 32 nodes for technical support perspective.

My question is it possible to scale say 80 nodes in a cluster without breaking something. Let's say I want to deploy a Ceph cluster of 120 nodes and want to use proxmox to ease management from a single GUI.
If I understand right Corosync/Pacemaker needs to send/receive cluster data over the network. So a high number of nodes causes extra load on the network. But, Pacemaker Remote is the currently recommended way to scale a cluster larger. With Pacemaker Remote, a small number of nodes run the full cluster stack including corosync and all pacemaker daemons, while the Pacemaker Remote nodes run only a single pacemaker daemon (the local resource manager). This allows the number of nodes to scale much higher.

Is this scenario possible in proxmox? Anybody using large node cluster?

 

[alexskysilk]

the limiting factor is corosync, not proxmox. corosync is known to become, umm, undependable above 32 nodes (I would say 16, but people on the mailing lists swear that they have larger deployments without trouble.)

The vector to stability at larger scale is to switch to pacemaker remote (https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html-single/Pacemaker_Remote/) but that would be a pretty major change to proxmox. unless and until someone really wants (and wants to invest the development time) to scale beyond ~30 nodes I doubt this will happen.

 

[dietmar]

I am quite sure we do not switch to pacemaker. Instead, we could try to implement some kind of proxmox satellite node (like pacemaker remote) - but I am not sure if it is worth the efforts.

Recent ceph changes makes it even easier to run large ceph clusters without the requirement that all nodes are part of the pve cluster (only minimal configuration is stored inside ceph.conf). So maybe one way to go is to manage the monitor nodes using PVE, but keep all the OSD only nodes outside the pve cluster...

 

[wahmed]

.......So maybe one way to go is to manage the monitor nodes using PVE, but keep all the OSD only nodes outside the pve cluster...

That's a great idea @dietmar ! By having the Ceph MON nodes in the cluster and everything else outside looks like the Ceph cluster can still be managed. Just tried it on a Proxmox+Ceph cluster. I removed some OSD nodes from the cluster and I can still see all the OSDs from PMX GUI.

Thanks dietmar!