How many nodes in the same PVE cluster with Unicast can i have?

cesarpk

Well-Known Member
Mar 31, 2012
770
3
58
Hi to all, in special to PVE developers team

I had read this link:
http://www.linkedin.com/groups/Proxmox-VE-21-Cluster-4-3739095.S.188944434

And in this link Mr. Alejandro P. tell us about of his professional experince about of PVE Cluster with more of 8 Nodes that are working in Unicast mode (I translate to English part of the text):
"Yes, but it seems there is no problem, the drawback is the bandwidth, but I already have several clusters without multicast with more than 8 nodes
And it works perfectly on unicast."

Then, my question Million is:
How many nodes maximum in the same PVE cluster in Unicast mode configured can i have on my LAN? (Mr. Google don't help me)

Why this question:
1- Because if i have a PVE Cluster with more of four Nodes and managed switches, I should not worry of configure the Switches, even before buying them (a concern less, that it is my case)
2- Unicast generates less network traffic

Other important Note:
If it say Mr. Alejandro P. is correct, please correct the wiki of proxmox that say us about of Unicast "do not use it with more that 4 cluster nodes":
http://pve.proxmox.com/wiki/Multicast_notes.

Best regards
Cesar
 
Cesar, this is a good question, wish I had an answer for you.

I am using 10g infiniband with multicast, my largest Proxmox cluster is 20 nodes. Everything runs just fine.
My understanding is that unicast requires much more bandwidth.
With multicast each node sends one packet that is received by all of the other nodes, so with 20 nodes that is 20 packets sent.
Using unicast each node needs to send one packet to each node. With 20 nodes each node sends 19 packets. 20 nodes x 19 packets per node is 380 packets, that is a huge difference. Packets needed when using multicast grows liner, with unicast it grows exponentially.

It is important to remember that latency is a concern with the cluster communications. Sure you might be able to run lots of nodes using unicast but when your cluster is under high load will it be able to process the cluster packets fast enough to prevent problems?
 
Cesar, this is a good question, wish I had an answer for you.

I am using 10g infiniband with multicast, my largest Proxmox cluster is 20 nodes. Everything runs just fine.
My understanding is that unicast requires much more bandwidth.
With multicast each node sends one packet that is received by all of the other nodes, so with 20 nodes that is 20 packets sent.
Using unicast each node needs to send one packet to each node. With 20 nodes each node sends 19 packets. 20 nodes x 19 packets per node is 380 packets, that is a huge difference. Packets needed when using multicast grows liner, with unicast it grows exponentially.

It is important to remember that latency is a concern with the cluster communications. Sure you might be able to run lots of nodes using unicast but when your cluster is under high load will it be able to process the cluster packets fast enough to prevent problems?

Good answer e100 !!!

Then, if I have a PVE Cluster with NICs of 1 Gb/s for use with:
1- The PVE Cluster communication
2- The LAN of his VMs
3- The Live Backup of VMs
Question:
Until how many Hosts as maximum is recommended to have in a PVE Cluster for get a good performance in the VMs?

And, if I have a PVE Cluster with NICs of 1 Gb/s (bonded in 2 NICs 802.3ad trunk) for use with:
1- The PVE Cluster communication
2- The LAN of his VMs
(The live Backups will use other NICs exclusively)
Question:
Until how many Hosts as maximum is recommended to have in a PVE Cluster for get a good performance in the VMs?

I think that two variables very important for these questions are:
1- How much bandwidth are necessary for his VMs?
2- How many packets are necessary transmit for each Node of PVE Cluster?, and How much each package weighs?

Then, It would be nice to have some reference parameters in a PVE wiki with the goal of avoid the structural errors at the time of building a PVE Cluster of many Nodes. This leads me to the following question:
Someone can make a map of all this maze? (Many people of this forum will be
grateful)

Best regards
Cesar
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!