External Ceph Monitor

adamb

Famous Member
Mar 1, 2012
1,322
72
113
Hey guys we are getting ready to roll out a new production ceph cluster using proxmox.

I wanted to get some input on monitors. Does everyone always use external monitors, or do most use their OSD nodes?

I want to have 3 monitors, but was thinking of maybe 1-2 external monitors and 1 monitor on a OSD node.

Any major issues with that idea?
 
No, there's no "major" issues with "external" monitors -- depending on your definition of "external". Ceph has more or less always supported a monitor separate from an OSD node. It just needs access to the Ceph public network for communication with all OSDs and other MON/MGRs.

Within the PVE community, everyone does what makes sense for their implementation. Personally, we run completely hyper-converged to simplify O/M, but that requires the hardware resources for Ceph, PVE and the VMs/CTs to do so. You could, for instance, make your MON nodes also host your compute for VMs/CTs leaving your OSD nodes dedicated to shared storage only, etc.

The minimum and sufficient instances is 3 monitors for a highly available environment. Also, Ceph managers (MGR) are required (v12+) and it makes sense to run the MGR on the MON node, but it is NOT mandatory. Again, I run more to maintain my hyper-converged architecture, so my MON/MGR daemons in excess of 3 may be considered "wasting resources", but I don't have to care which node(s) is(are) active/master/quorum if I should have a failure while another node is down for maintenance.
 
  • Like
Reactions: Tmanok
No, there's no "major" issues with "external" monitors -- depending on your definition of "external". Ceph has more or less always supported a monitor separate from an OSD node. It just needs access to the Ceph public network for communication with all OSDs and other MON/MGRs.

Within the PVE community, everyone does what makes sense for their implementation. Personally, we run completely hyper-converged to simplify O/M, but that requires the hardware resources for Ceph, PVE and the VMs/CTs to do so. You could, for instance, make your MON nodes also host your compute for VMs/CTs leaving your OSD nodes dedicated to shared storage only, etc.

The minimum and sufficient instances is 3 monitors for a highly available environment. Also, Ceph managers (MGR) are required (v12+) and it makes sense to run the MGR on the MON node, but it is NOT mandatory. Again, I run more to maintain my hyper-converged architecture, so my MON/MGR daemons in excess of 3 may be considered "wasting resources", but I don't have to care which node(s) is(are) active/master/quorum if I should have a failure while another node is down for maintenance.

That is pretty much along the lines I was thinking but I wanted to confirm. Appreciate the input.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!