3 node cluster setup recommendations

[Magneto]

As you will install PVE on a DOM:
Install PVE on DOM. Don't do anything to the drives.
After that you just go by the Documentation:
>https://pve.proxmox.com/wiki/Ceph_Server
Use 3x8 TB as OSDs (why not all 4?) and one SSD as cache:
>https://pve.proxmox.com/wiki/Ceph_Server#Creating_Ceph_OSDs


Regards,
Jonas

That is the setup I have already but I feel I could get better performance, somehow? The way I see it now, each SATA drive acts on it's own. OR does CEPH somehow stripe the data acess the spindles to improve performance?

The 4th drive will be used for backup and to host ISO's. As I couldn't find a way to create a filesystem on CEPH, and Proxmox doesn't allow storing ISO's on CEPH drives, I had to setup GlusterFS for the 4th drive in each server.

 

[tschanness]

Ceph does not "stripe" but distributes all written Blocks (well technically RDB does that I guess but that's besides the point) to different Hard Drives. What performance do you get? What's your ethernet connection right now? Did you interconnect every node with 10g or is the 4th node connected with the 600mbit/s wireless link?

Regarding your question about direct connect: https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server

Regards, Jonas