With Proxmox being installed on the SATADOM, 2 questions arise:
1) where should Ceph monitors be installed
2) do I need journals
I'm assuming you installed Proxmox onto the DOM so that you can allocate all six SSDs to OSD? This will work, but...
- The MON does have some disk access requirements (limited)
- CEPH logs everything, with log entries every couple of seconds.
These two things together could cause trouble because the DOMs write endurance is quite limited. While not quite as quick to die as a USB stick, they are a lot less tolerant of write activity than even older SSDs.
You might want to reconsider and just deploy the base Proxmox on one of the SSDs. You can leave things on the DOM and relocate the logs onto one of the SSDs but by the time you go through the complexity of doing that you have a very similar outcome to just dedicating one SSD to Proxmox - it still burns one of the SSDs (or a partition of it). Better in my mind to "keep it simple and stupid" (KISS).
As for number of nodes: 4 is just fine. 3 works nominally, but can't really maintain fully redundant operation in the face of a failure (assuming replication of 2 [3 copies]). An odd number is better because you can still recover reliably if the network isolates (i.e., if you get two sets of two nodes that can still see each other). Because of this, you can consider 3 nodes the minimum viable, 4 nodes workable and fine, but 5 nodes is the "ideal small site" for Ceph.
