Proxmox VE Ceph with separate Ceph OSD servers

[hanisirfan]

Hi. I'm been playing around with Ceph using cephadm and would like to explore Proxmox for our "management" cluster for our upcoming Apache CloudStack project. I noticed that Proxmox did have integration with Ceph but not using cephadm as the ochestrator.

My question is, is it possible to have the Ceph MGR, MON and a few OSD in a Proxmox VE cluster (say 3 nodes) and then add a totally OSD dedicated nodes which is installed with Linux and not Proxmox VE. What I meant by this is the OSD dedicated nodes is not added in Proxmox VE cluster. Thanks

 

[alexskysilk]

My question is, is it possible to have the Ceph MGR, MON and a few OSD in a Proxmox VE cluster (say 3 nodes) and then add a totally OSD dedicated nodes which is installed with Linux and not Proxmox VE.

Possible- sure- but why? it would be a LOT more work.

 

[UdoB]

My question is, is it possible to have the Ceph MGR, MON and a few OSD in a Proxmox VE cluster (say 3 nodes) and then add a totally OSD dedicated nodes which is installed with Linux and not Proxmox VE. What I meant by this is the OSD dedicated nodes is not added in Proxmox VE cluster.

Probably yes. But I do not know how.

Could you elaborate why you would do that?

I ask this because: you can just install PVE on those "dedicated Ceph nodes", join the cluster to get the advantage of the known WebGui and NOT run any VM on those nodes. Mission accomplished. No?

 

[hanisirfan]

I've actually never properly play around with Proxmox VE so I've limited knowledge. But, after seeing peers used it for their cluster, looks like a really decent virtualization tool. If I did install Proxmox VE on those dedicated Ceph nodes, did you guys have any estimates on how it would scale? Is it possible to scale to hundreds or even thousands of nodes like some/most Ceph clusters?

 

[UdoB]

Is it possible to scale to hundreds or even thousands of nodes like some/most Ceph clusters?

Probably not. Not in the HC way I proposed.

I have no good answer, just one quick hint: a PVE cluster can not grow endless. For Version 4 (very, very old) the maximum number of member it was somewhere at 32 (https://pve.proxmox.com/wiki/Proxmox_VE_4.x_Cluster). For version 8 I did not find a newer statement. Probably this should be answered by a Staff member...

 

[VictorSTS]

There's no hardcoded limit on how many hosts you can have in a cluster. Estimations set a limit around 50 hosts, depending on network speed, CPU speed and packet latency needed for Corosync to keep quorum. Remember that PVE has it's quorum maintained by Corosync and Ceph uses it's MON to manage it's own quorum, even when used in HCI.

If you really plan to having hundreds or even thousands of Ceph hosts, you need a dedicated Ceph cluster built with native tools (i.e. cephadm) and just access that storage from your PVE cluster nodes.

 

[hanisirfan]

There's no hardcoded limit on how many hosts you can have in a cluster. Estimations set a limit around 50 hosts, depending on network speed, CPU speed and packet latency needed for Corosync to keep quorum. Remember that PVE has it's quorum maintained by Corosync and Ceph uses it's MON to manage it's own quorum, even when used in HCI.

If you really plan to having hundreds or even thousands of Ceph hosts, you need a dedicated Ceph cluster built with native tools (i.e. cephadm) and just access that storage from your PVE cluster nodes.

Thanks for the clue. I was thinking of using Proxmox with Ceph as HCI because we're building something and only have the budget for 3 nodes for start. These 3 node I was thinking to be the "management KVM cluster" and I'll add more Ceph nodes later on. Looks like maybe not an option to run Proxmox Ceph due to the 50 hosts limit.