Proxmox Ceph Converged (HCI) or external ceph

[sander93]

Hello,

On this moment we have:

6 x Proxmox Nodes
2 x 10 cores (2 nodes have 2 x 14 cores)
512 GB RAM
4 x 10 GB (2 x 10 GB LACP for network en corosync and 2 x 10 GB LACP for Storage)

3 x Ceph Monitor
Dual Core
4 GB RAM
2 x 10 GB LACP

4 x Ceph OSD
2 x 6 Core 2,6 Ghz
96 GB RAM
4 x 10 GB (2 x 10 GB LACP for replication en 2 x 10 GB for public network)
All OSD are Enterprise SSD

We are running around 120 VM's on this moment.

We need to expand our storage capacitiy, so we want to add 2 servers.
Now we have 2 options:
- Add the 2 new servers to the ceph cluster so we have 6 osd nodes
- Install and add them to Proxmox cluster, activate/install ceph inside the proxmox cluster en move data to the new ceph cluster en migrate the existing hosts to proxmox nodes so we go hyper converged.

What is the recommend way to go?
I read differents approaches, seperate can be usefull when making changes/updates/etc
But converged can maybe make more use of the cpu capacity and we have more ceph nodes, they say that is a good thing for latency and performance for the ceph cluster.

In Proxmox v5 the corosync limit the number of nodes in a single cluster, i think this problem has been "resolved" with the new corosync v3..

I have read a lot of topics about this but i want to know some recommendation for our specific envoirement.
Also i want to keep is scalable.

 

[Alwin]

Have you already seen our docs, the benchmark paper, and the corresponding thread?
https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_precondition
https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark-2018-02.41761/
https://proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark

 

[sander93]

Hello Thanks for you response.

Yes i have read them already, but they tell me (almost) nothing about the pre's or con's for going converged or to continu with a separate ceph cluster and separate proxmox cluster.

I am looking for any advice, recommendation or experience what option is going to work best for us.
Hyper Converged or External Ceph cluster

 

[jsimbr]

Hello Thanks for you response.

Yes i have read them already, but they tell me (almost) nothing about the pre's or con's for going converged or to continu with a separate ceph cluster and separate proxmox cluster.

I am looking for any advice, recommendation or experience what option is going to work best for us.
Hyper Converged or External Ceph cluster

Hi, sorry for extending a rather old thread, but I was curious to learn which way you went. Did you go for the HCI setup or decided to sick with the external Ceph cluster?

 

[sander93]

Hi,

We stick to te external Ceph cluster, to not make it more complex then it is.

Currently we are looking if we can setup a second cluster as HCI, where we can do a RBD-Mirror for say a disaster recovery.
So in worst case we can boot VM's on this DR HCI cluster

 

[jsimbr]

I see, it makes sense. So, is your cluster running with 6 OSD nodes now or it grew even larger? I’m trying to figure out what would be a reasonable (safe) size to build our own production Ceph cluster.

 

[sander93]

Our production Ceph cluster has on this moment:
3 x Mon/MGR Nodes
6 x SSD OSD Nodes
3 x HDD OSD Nodes

Currently we are running 9 Proxmox nodes in a cluster which are connected/using to this Ceph cluster.

If you have any questions please let me know, always good to share experience.

 

[jsimbr]

Thanks for sharing your valuable experience. I wonder how you connect your Proxmox cluster to Ceph storage. Do you use Ceph RBD directly or other protocol (e.g. iSCSSI or NFS)?

 

[sander93]

We Use RBD directly, i think the best option in most situations.

 

[jsimbr]

Great. Thanks again for sharing. We’ll soon build our 4-node cluster and I’ll post the results to the forum.

 

[oendaps]

Hi @sander93 may i ask for best practice that i have 4 nodes with 2 nodes for compute and 2 nodes for CEPH Storage and i think i'm wrong to mix all nodes on 1 Cluster, i'm planning to add 1 node for the ceph to get quorum and redudancy, Here is the existing topology