ceph production with 3 nodes and SAS HDD disks?

[benoitc]

I understand that with 3 nodes I will need at least 2 to have the cluster to be OK. Is this correct? Would it be ok for a smal production server to run on 3 nodes with recent versions of ceph? What should I take care?

Also my disks are not SSD but HPE HDD midline 2TB 12G SAS 7.2K rpm. What kind of performance should I expect over a 25G network? Is this better to run the node just with local storage storaes and then have a ZFS raid mirror ?

 

[jdancer]

It's true that you need a minimum of 3-nodes for Ceph but it's highly recommended to get mores nodes.

With that being said, I do run a 3-node full-mesh broadcast bonded 1GbE Ceph Quincy cluster on 14-year old servers using 8xSAS drives per node (2 of them are used for OS boot drives using ZFS RAID-1). Works surprisingly well.

Then again, it's running Debian underneath.

I use the following optimizations:

Set write cache enable (WCE) to 1 on SAS drives
Set VM cache to none
Set VM to use VirtIO-single SCSI controller and enable IO thread and discard option
Set VM CPU type to 'host'
Set VM CPU NUMA if server has 2 or more physical CPU sockets
Set VM VirtIO Multiqueue to number of cores/vCPUs
Set VM to have qemu-guest-agent software installed
Set Linux VMs IO scheduler to none/noop
Set RBD pool to use the 'krbd' option