Proxmox cluster scaling best practices

[Paspao]

Hello,

I have a 5 node PVE cluster with Ceph (2 OSD every node on Intel DC SSD on dedicated 10G network).

I am actually using it for LXC containers and my idea is to always keep one node empty to move containers in case of node failure.

I will need to scale up in the near future probably to a total of 21 servers.

What are the best practices and bottlenecks to consider while scaling?

Will my 10GB network become the bottleneck adding nodes ? ( I have dual 10GB adapter so I could use bonding to get 20GB)

I could create separate smaller clusters (3 cluster of 7 nodes) but I will need more hot standby servers.

Any suggestion is appreciated.

Thank you.
P.

 

[wolfgang]

Hi,

generall if you will scale up to 21 nodes I would split into two clusters.
About 7 nodes for storage(Ceph) and the rest for computation(LXC/KVM).

Resources management is better to handle.
Also, you can use cheaper CPUs for the Ceph cluster.

Assume 21 nodes with 2 OSD per node is the same as 7 nodes with 6 OSD.
This means you need a CPU with 8 threads.
Ceph will benefit if you use a CPU with high clock cycles.

The next thing is you should separate the cluster and the private network of ceph.
It would use 40 GBit nics on the Ceph cluster and 10GBits to link to the computation Cluster.

 

[Paspao]

Hi Wolfgang,

thank you for your reply.

My plan was to always use Hyper-converged clusters and not to split Ceph from Compute even for easier and progressive growth.

In my 5 nodes I have seen peaks of 20 MB/s and 2000 Iops on Ceph network and I think there room to grow.

One thing is not very clear to me is if adding nodes will increase traffic on Ceph network and with which progression rate, as adding nodes and OSDs allows to share peaks of load.

Thank you.
P.

 

[wolfgang]

One thing is not very clear to me is if adding nodes will increase traffic on Ceph network and with which progression rate, as adding nodes and OSDs allows to share peaks of load.

generally, traffic stays the same.
The most traffic comes from the data and not the management.
The traffic normally gets more distributed and a single node has less network traffic.
Monitors produce a lot of packages but on a huge cluster, the recommendation is max 5 mons.
The osd itself does produce minimal traffic, the most traffic at osd level come from the write/read procedure and this stays the same.

 

[Paspao]

Thanks a lot !

So I can think of scaling from 5 to 9 nodes (and from 10 to 18 OSDs) without any specific worries.

Best regards
P.

 

[wolfgang]

So I can think of scaling from 5 to 9 nodes (and from 10 to 18 OSDs) without any specific worries.

Adding the new OSD will produce extra network load, because of rebalancing.
So do this on a weekend or in nonproductive time.