Install new Ceph Manager in the Production Cluster

[adriano_da_silva]

Hello everyone,

I've set up a highly available hyper-converged Proxmox 7.4-3 cluster with Ceph Quincy (17.2.5), featuring ten nodes, with the first three as monitors, and only the first node acting as a Ceph Manager. Each node has two OSDs. There are two pools in Ceph, each linked to one OSD on every node. Virtual machines run on these nodes utilizing the pools. The cluster has been running smoothly until today when the first node went down due to a boot disk issue. Consequently, the cluster is currently without a Ceph Manager, at least temporarily.

Is it possible to install the Ceph Manager on a new node? Should I do so? If yes, what would be the best way to proceed, considering the cluster is in production? Thank you!

 

[Falk R.]

You can install many manager, but only one is active. When the Node with the active Manager fails, another Manager goes active.

Your OSD Setup is not good/recommended.
It’s recommended that you have a minimum of three OSD per Node, better is 4.
And then use a single Pool. When you need a replicated Pool for Server and an erasure code Pool for slow Filedata, then is OK with many Pools. In default all OSD are used by all Pools, except you build own rules. Please do that only, when you have enough Disks for every Pool.

 

[adriano_da_silva]

Thank you for your help! My questions now are:

A) Can I then install a new Manager and make it active, even if the first node is inactive, without this causing cluster confusion when the first node comes back? Is there any sequence of mandatory commands to pay attention to?

B) I use 2 pools because I have two OSD classes: HDD and SSD. There is a crush rule to separate the writings. One pool writes only to SSDs while the other writes only to HDDs. One of the pools is used for database data, OS's disks, the other for performing database dumps for backup purposes. This is bad idea?

C) I didn't know this rule that there must be at least 3 OSD's per node. I believed that if I kept the OSD's more spread out across many different nodes, I would have greater data availability, even in the event of the loss of 1, 2 or even more nodes simultaneously. So I set it up like this for safety. The objective is to maintain the possibility of scaling in the future, increasing the number of SSD's and HDD's (OSD's) per node as needed. I always try to keep the size of OSDs per node balanced. Is there any risk or problem with keeping it like this now?

Thank you!!

 

[sb-jw]

You should always set up 3 Mons and 3 Mgr straight away. Works via the GUI and is done in 1 minute. Just fix it now and the topic is over.

 

[adriano_da_silva]

Guys, I created two new managers using the GUI and it worked. I'll see how it goes when the first node is back up and running. But for now, there have been no problems.
Thanks!