[SOLVED] Dedicated Ceph Monitor Nodes

[mihanson]

Hello all,
I'm running a 3 node Proxmox 6.2 cluster at home with Ceph Nautilus (14.2.10). Currently, each Proxmox node also acts as a Ceph Monitor node as well. I would like to separate the Ceph monitors from my 3 Proxmox nodes, running the 3 Ceph monitors on 3 dedicated computers in my Proxmox/Ceph public network. My main question is: Should I just install Proxmox on the dedicated Ceph Monitor hardware and simply add the monitors or install Debian stable on the monitor hardware and add the PVE repository to keep Ceph versions the same? IIRC, Debian stable is a few Ceph versions behind Proxmox. A third option would be to install Arch Linux (my distro of choice/most comfortable using) on the monitor hardware and install Arch's Ceph packages, which are currently 14.2.8 or create my own 14.2.10 packages. I'm open to suggestions, but I feel like installing Proxmox for dedicated Ceph Monitors would be overkill and I'm not sure if simply adding Proxmox repos to vanilla Debian stable would work, but feel that would be the easiest way to keep Ceph versions in sync. Any insights welcome. Thanks!

Mike

 

[Alwin]

I feel like installing Proxmox for dedicated Ceph Monitors would be overkill

Why would it be? It would be the same management as the other nodes. And using them in a separate cluster, could make it easier.

and I'm not sure if simply adding Proxmox repos to vanilla Debian stable would work

That's usually the way to get Proxmox VE installed, if the ISO is not used. Also you could only use the Proxmox Ceph repositories. But you will need to manage those nodes differently.

feel that would be the easiest way to keep Ceph versions in sync.

Exactly. It makes things easier.

 

[mihanson]

Thank you @Alwin. I was thinking it would be "overkill" because I would not be using any other functionality of Proxmox other than Ceph monitoring. For the ease of setup I'll give it a shot though. Thank you.

 

[mihanson]

Ok, so I'm a little confused on how to proceed. I have installed, updated and configured Proxmox 6.2 on 3 new computers. Currently, each is it's own standalone node. If I want these 3 new nodes to only be Ceph Monitors for my cluster (cluster "A") how do I proceed? Do these 3 new nodes need to join my cluster (cluster "A")? Do they need to be in their own cluster ("cluster B"), isolated from my cluster (cluster "A")? The docs I've been finding utilize ceph-deploy and the process is a bit murky to me. For reference, my 3 new standalone nodes are all the same. They are Zotac ZBox CI323 Nano with 8GB RAM and 2 x 1G NICs. Thank you for any help on how to proceed.

 

[Alwin]

Ceph and Proxmox VE (corosync in that case) are two different clusters. While you will need to join the three nodes to the current Ceph cluster, they do not need to be part of the Proxmox VE cluster. Hence, the can be in their own Proxmox cluster (easier management) and just host the MONs for Ceph.

 

[mihanson]

@Alwin Thanks for that. What I ended up doing was installing Proxmox 6.2 on the 3 new computers, joined them to my Proxmox cluster and set up Ceph monitors on each of the 3 new computers. I then destroyed the monitor instances on my other cluster computers and allowed the new computers to take over the monitoring of Ceph. So, in the end, I have 6 computers in my cluster (3 for hosting VMs, containers, etc and 3 strictly as Ceph monitors.) It seemed easier for me to do it this way then to have 2 separate Proxmox clusters.

 

[r.jochum]

May I ask why you did that? Just want to know

 

[mihanson]

@r.jochum Which part? Joined all 6 boxes into one cluster or separated the monitors from the 3 computers that will host VMs, containers and Ceph storage?

I joined all 6 together because I can then administer from a single interface. I chose to keep everything on Proxmox to ease in admin. It keeps all software on the same version and again, one interface to deal with.

As far as wanting Ceph monitors on different boxes, it keeps quorum if I reboot a VM/storage node and somewhere in the Ceph docs (sorry don't have it handy) it stated that was best practice. And also because I could. LOL