More then 3

[mattman110]

Hi all.

So I'm building a production environment proxmox, we have more then 3 servers and what not, so we will be building them as a cluster for ease of management, migration and HA.

We are planning on having 2 drives mirrored for the proxmox os and all other drives will be used in a ceph config.

We have multiple network connections on all machines and will have a dedicated network for ceph.

My question here refers to 2 things.

- What is the disadvantages of having more then 3 nodes in a proxmox cluster... And is there any document online to help support this (I can't find anything talking about more then 3 nodes )

- How dose ceph work in a cluster with more then 3 nodes, with my currently knolage, I know the osd are smart enough to know ware they are and eatch proxmox node will only habe one copy of the data, that and the rule you set in the ceph config, max copy 3 min 2. Ect..

But how dose ceph work with multiple nodes above 3..
Do all the HDDs in eatch node need to be the same size.
What's the limiting factor.

I'm thinking the second part is way more ceph related but if someone can signpost me to ware I can find the reverent documentation or videos to understand the use case here better ,as I need to increase my knolage so I can guarantee this is the best option for my production environment.

Thanks all for being the amazing supportive bunch you are.

 

[Falk R.]

- What is the disadvantages of having more then 3 nodes in a proxmox cluster... And is there any document online to help support this (I can't find anything talking about more then 3 nodes )

3 Nodes are the Minimum for a Cluster with Ceph.

- How dose ceph work in a cluster with more then 3 nodes, with my currently knolage, I know the osd are smart enough to know ware they are and eatch proxmox node will only habe one copy of the data, that and the rule you set in the ceph config, max copy 3 min 2. Ect..

But how dose ceph work with multiple nodes above 3..
Do all the HDDs in eatch node need to be the same size.
What's the limiting factor.

The limiting factor is probably the network and your know-how.

Ceph is a scale-out storage system and scales up to several thousand nodes if you have the right network and know-how.
With our small Proxmox setups with up to 32 nodes, this is all quite easy.

1. if you have an even number of nodes, you should use a Q-Device for the cluster.
2. if you want to distribute Ceph across several data centers, you should place a monitor node in a third data center.
Otherwise simply define 3 monitor and for clusters with more than 9 nodes I often use 5 Ceph-monitor.
Please always set a minimum of 3 monitor and a maximum of 5 monitor.
3. ceph distributes the data according to your crush rule and the default setting is to distribute the data per node. If you run a replica 3/2, Ceph always distributes the data to 3 different nodes, no matter how many OSDs there are.
4. the disks for the OSD do not have to be the same, but if you want a simple setup with consistent performance, it is recommended to use the same disks. Ceph also scales well with different disks, but then you have to put a lot more brainpower into performance troubleshooting.
If you should get the idea to use HDDs for Ceph, this works, but is not recommended for VMs. When using HDDs, be sure to use fast SSDs with good write endurance as bluestore. I always recommend SSD only setups.
5. The network is very important for Ceph.WWeWhoIf If you only want to use 10 GBit, for example, you should be aware that you can achieve a maximum throughput of 3.3 GBit in the worst case with a 3/2 replica setup. After all, you have to distribute the data across 3 OSDs.
I recommend not starting with Ceph below 25 GBit and for newly planned setups I do not start below 100 GBit. I achieve up to 60 GBit network load with small 3-node clusters with only 4 OSDs per node (NVMe). If your cluster gets a bit bigger, you should plan enough reserves.
P.S. 100 GBit is no longer expensive these days, unless you want to use Cisco or HPE.