External Ceph Monitor

[adamb]

Hey guys we are getting ready to roll out a new production ceph cluster using proxmox.

I wanted to get some input on monitors. Does everyone always use external monitors, or do most use their OSD nodes?

I want to have 3 monitors, but was thinking of maybe 1-2 external monitors and 1 monitor on a OSD node.

Any major issues with that idea?

 

[RokaKen]

No, there's no "major" issues with "external" monitors -- depending on your definition of "external". Ceph has more or less always supported a monitor separate from an OSD node. It just needs access to the Ceph public network for communication with all OSDs and other MON/MGRs.

Within the PVE community, everyone does what makes sense for their implementation. Personally, we run completely hyper-converged to simplify O/M, but that requires the hardware resources for Ceph, PVE and the VMs/CTs to do so. You could, for instance, make your MON nodes also host your compute for VMs/CTs leaving your OSD nodes dedicated to shared storage only, etc.

The minimum and sufficient instances is 3 monitors for a highly available environment. Also, Ceph managers (MGR) are required (v12+) and it makes sense to run the MGR on the MON node, but it is NOT mandatory. Again, I run more to maintain my hyper-converged architecture, so my MON/MGR daemons in excess of 3 may be considered "wasting resources", but I don't have to care which node(s) is(are) active/master/quorum if I should have a failure while another node is down for maintenance.

 

[adamb]

No, there's no "major" issues with "external" monitors -- depending on your definition of "external". Ceph has more or less always supported a monitor separate from an OSD node. It just needs access to the Ceph public network for communication with all OSDs and other MON/MGRs.

Within the PVE community, everyone does what makes sense for their implementation. Personally, we run completely hyper-converged to simplify O/M, but that requires the hardware resources for Ceph, PVE and the VMs/CTs to do so. You could, for instance, make your MON nodes also host your compute for VMs/CTs leaving your OSD nodes dedicated to shared storage only, etc.

The minimum and sufficient instances is 3 monitors for a highly available environment. Also, Ceph managers (MGR) are required (v12+) and it makes sense to run the MGR on the MON node, but it is NOT mandatory. Again, I run more to maintain my hyper-converged architecture, so my MON/MGR daemons in excess of 3 may be considered "wasting resources", but I don't have to care which node(s) is(are) active/master/quorum if I should have a failure while another node is down for maintenance.

That is pretty much along the lines I was thinking but I wanted to confirm. Appreciate the input.