Proxmox cluster scaling best practices

Paspao

Member
Aug 1, 2017
55
1
8
50
Hello,

I have a 5 node PVE cluster with Ceph (2 OSD every node on Intel DC SSD on dedicated 10G network).

I am actually using it for LXC containers and my idea is to always keep one node empty to move containers in case of node failure.

I will need to scale up in the near future probably to a total of 21 servers.

What are the best practices and bottlenecks to consider while scaling?

Will my 10GB network become the bottleneck adding nodes ? ( I have dual 10GB adapter so I could use bonding to get 20GB)

I could create separate smaller clusters (3 cluster of 7 nodes) but I will need more hot standby servers.

Any suggestion is appreciated.

Thank you.
P.
 

wolfgang

Proxmox Staff Member
Staff member
Oct 1, 2014
4,966
331
83
Hi,

generall if you will scale up to 21 nodes I would split into two clusters.
About 7 nodes for storage(Ceph) and the rest for computation(LXC/KVM).

Resources management is better to handle.
Also, you can use cheaper CPUs for the Ceph cluster.

Assume 21 nodes with 2 OSD per node is the same as 7 nodes with 6 OSD.
This means you need a CPU with 8 threads.
Ceph will benefit if you use a CPU with high clock cycles.

The next thing is you should separate the cluster and the private network of ceph.
It would use 40 GBit nics on the Ceph cluster and 10GBits to link to the computation Cluster.
 

Paspao

Member
Aug 1, 2017
55
1
8
50
Hi Wolfgang,

thank you for your reply.

My plan was to always use Hyper-converged clusters and not to split Ceph from Compute even for easier and progressive growth.

In my 5 nodes I have seen peaks of 20 MB/s and 2000 Iops on Ceph network and I think there room to grow.

One thing is not very clear to me is if adding nodes will increase traffic on Ceph network and with which progression rate, as adding nodes and OSDs allows to share peaks of load.

Thank you.
P.
 

wolfgang

Proxmox Staff Member
Staff member
Oct 1, 2014
4,966
331
83
One thing is not very clear to me is if adding nodes will increase traffic on Ceph network and with which progression rate, as adding nodes and OSDs allows to share peaks of load.
generally, traffic stays the same.
The most traffic comes from the data and not the management.
The traffic normally gets more distributed and a single node has less network traffic.
Monitors produce a lot of packages but on a huge cluster, the recommendation is max 5 mons.
The osd itself does produce minimal traffic, the most traffic at osd level come from the write/read procedure and this stays the same.
 

Paspao

Member
Aug 1, 2017
55
1
8
50
Thanks a lot !

So I can think of scaling from 5 to 9 nodes (and from 10 to 18 OSDs) without any specific worries.

Best regards
P.
 

wolfgang

Proxmox Staff Member
Staff member
Oct 1, 2014
4,966
331
83
So I can think of scaling from 5 to 9 nodes (and from 10 to 18 OSDs) without any specific worries.
Adding the new OSD will produce extra network load, because of rebalancing.
So do this on a weekend or in nonproductive time.
 
  • Like
Reactions: Paspao

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!