Ceph disk planning OSD journal

[Concluant]

I have 3 node (planning 3-node pve cluster with ceph)
2x12c/128Gb ram/2x240Gb ssd sata/6x1.2TB sas 10k/HBA mode raid controller HPE/2x10G eth/4x1G eth
and nfs-storage node
2x12c/64Gb ram/2x240gb ssd sata/10x6TB SATA 7200 Enterprise/HBA mode raid controller HPE/2x10G eth/4x1G eth

I plan deploy 10-20 VM/container.

I have a question about planning ceph

I want to use disks on each node pve-ceph:
2x 240GB ssd zfs raid1 (50GB system and 190GB to journal OSD ceph)
6x1.2TB sas - osd ceph

is it posibble? Or is it better to use 1 disk 240GB for the system, and second disk 240GB for the OSD Journal?

 

[jdancer]

May want to use that SSD also for the Ceph DB as well. Will help with writes to the SAS drives.

The SSD is enterprise class, correct? Ceph eats consumer SSDs like nobody's business.

Here is my Ceph VM optimizations I use:

Set write cache enable (WCE) to 1 on SAS drives
Set VM cache to none
Set VM to use VirtIO-single SCSI controller and enable IO thread and discard option
Set VM CPU type to 'host'
Set VM CPU NUMA if server has 2 or more physical CPU sockets
Set VM VirtIO Multiqueue to number of cores/vCPUs
Set Linux VMs IO scheduler to none/noop
Set RBD pool to use the 'krbd' option

 

[Concluant]

This is an interesting solution. I'll try.
Now i have only samsung 970 evo pro. But, in the future i can replace ssd disks.


What about the question of using zfs? (2 ssd disks zfs mirror for system installation and OSD journal).
Create zfs mirror partition 50GB during installation. (sda1 sdb1)
And create zfs mirror partition 190GB from OSD journal (sda2 sdb2)

I'm not sure that Ceph will accept zfs mirror (from OSD journal) as a separate device.

 

[Concluant]

Ok. Following experimental way: zfs mirror, 2 partition (os and wal/db OSD) - it doesn't work that way.