Minimum requirements for full high availability (PVE+CEPH)

[Strigi]

Hi there,

I'm quite new to Ceph and Proxmox.
And I was wondering what the minimum requirements are for running a full high available cluster.
Right now to my understanding this would be:

- 3 PVE nodes.
- Ceph with 6(?) OSDs

And of course the redundant power/network etc. but this question is more focused on Ceph and Proxmox.
Am I near the minimum requirements? Is it possible with fewer OSDs?
Would it be possible to use (as a proof of concept) 3 partitioned HDDs with each virtual OSD having it's own partition.

I'm planning on using PVE 5.1 and Ceph luminous.

Any advice would be very welcome!
Helpfull criticism as well.

Greetings,

Strigi

 

[Klaus Steinberger]

yep for HA this is the minimum. But keep in mind that CEPH is gaining speed from the number of OSD's as well their latency. I would recommend backing spinning disk OSD's with a SSD for journal (with filestore) or block.db (for bluestore).

you can use a partioned SSD for multiple spinning OSD's

Don't pack multiple OSD's on partitions of spinning disks, they will fight for iOPS.
With SSD's for the OSD it can be a idea to use multiple OSD on partitions (as some sources say that one OSD can't beat out a fast SSD), but I did not yet figure out how to do that with bluestore.

 

[dietmar]

Note: Using minimum requirements is probably a bad approach if you want high availability ...

 

[PigLover]

Absolute minimum: 3 nodes, 3 OSDs. Will still run with a failed node - but Ceph will report "degraded". There are some cases where Ceph may not be able to support writes, at which point VMs with images on RBD may stall or fault. Good for labs and small deployments that need to be "sorta HA" because outages are inconvenient though not actually damaging.

"Better" minimum: 4 noes, 4 OSDs. With a single node failed Ceph can recover to a fully stable operating condition (disk space permitting on the remaining 3 nodes, of course). Can survive two-node failure (degraded as above) IFF two nodes with Ceph MON function remain operating, though will lose Proxmox quorum due to even number of nodes (2 nodes is not >half). Sorta OK for production if you really have to shave costs

"Production" minimum: 5 nodes, 5 OSDs. Assume runs 5 MON copies, one per node (don't add more MONs if you grow to bigger sets, but do try to keep to odd numbers of nodes). Can survive and achieve full stable CEPH state with any two nodes failed. Better because odd number of nodes permits expansion of MON set to 5 and retains Proxmox quorum on multiple node failure. This is the minimum that I'd consider "production HA". Allows for nodes to be taken out of service for maintenance operations safely, etc (i.e., you can take one out on purpose, have one fault while that one is being worked on, and still have Ceph converge to a fully stable operating state).

In all cases, more OSDs will give better performance.

 

[5mart3ch]

For a homelab situation where I want to test out Ceph, do I just add additional SSD on each node for ceph? When I am not testing ceph, is it okay to power down one of the three node minimum and would that effect the operation of the other two nodes in anyway?

 

[Strigi]

Thanks for the replies everyone.
I have been given the task to find out for a company what the minimum specs should be to run Ceph with Proxmox.
They already use Proxmox but not in combination with Ceph!
Their current setup has 5 PVE nodes.
I was thinking about running a ceph monitor on each PVE node with maybe 3 OSDs for each node as well.

So how does Ceph respond to poweroutages (for whole of the system or partial), weird question, I know, but there have been problems with electricity in the area due to stuff flying into powerlines...

I've been reading alot of information about Ceph on their documentation site and the articles found on there.
But it's hard to find a quick overview how to respond in Disaster Recovery Situations or something similar.

Any advice on that?