Proxmox cluster beyond 16 Nodes

[wahmed]

Proxmox suggests upto 16 physical nodes or "probably more" per cluster.
https://pve.proxmox.com/wiki/Proxmox_VE_2.0_Cluster

I read somewhere sometimes ago KVM can be pushed up to 64 physical nodes without breaking anything. Anybody has experience beyond 16 nodes?

 

[udo]

Hi,
AFAIK the 16 node limit is due the cluster communication. A time ago (app. one year) there was an thread where someone get hints to use more nodes... but I'm remember not realy clear.

Udo

 

[udo]

Hi again,
found the post: http://forum.proxmox.com/threads/10388-How-many-nodes-can-be-in-one-cluster?p=58163#post58163

Udo

 

[spirit]

The limit come from corosync.

16 is not the hardlimit, I think it's around 64 , but it's depend of network latency I think, and it's need some tuning. (timeout and max message size).

Maybe with really fast network, big mtu, it could work.


I think we should ask to the corosync mailing list.

 

[wahmed]

16 is not the hardlimit, I think it's around 64 , but it's depend of network latency I think, and it's need some tuning. (timeout and max message size).
Maybe with really fast network, big mtu, it could work.

Thats what i was thinking. We are in the process of upgrading Gigabit network to 40gbps. So it should handle the increased traffic. I will try to get more details from Corosync guys.

 

[e100]

Our 20 node cluster works fine over 10G Infiniband with default settings.