New Proxmox Cluster Storage suggestion

[a4t3rburn3r]

Hi good people.
I am planning to upgrade existing proxmox cluster.
I have 4 nodes with dual xeon (56 vCores) and 512GB RAM per node.
Proxmox is installed on ZFS mirror from two intel 3510 SSD.
I have used NFS storage for VM's, but now have 4 jbods (with 48x 8TB HDD and 12x 800GB SSD in each jbod).
I think it will be a good idea to make shared nothing storage connecting 1 jbod per 1 virtualization node to get storage HA.
All nodes are connected with each other using 40Gbps ethernet over redundand switches.
The question is: what storage organisation is best or suitable? Ceph / DRBD9 / other?
So at this time i see ceph with crash maps (ssd fast storage + hdd slow storage) or drbd9 with lvmthin. But i have question - what decision will be better? And what organisation structure will be more suitable? Do i need journal on SSD for ceph HDD in proxmox 5.x? Do i need a lot of OSD's for ceph (how much OSD's i need)?

In near feauture i don't plan to extend this cluster nor pv nodes nor storage, at this time we can't get official proxmox support, because my company can't buy support directly from proxmox, we signed the contract with the intermediary about proxmox purchase, but it can't to fulfill the obligations now. Later this year we plan to get official proxmox support. So proxmox will be upgraded to 5.3 version but first time we will be using community repo.

Thanks and good luck.

 

[LnxBil]

... 4 nodes .. dual xeon (56 vCores) .. 512GB RAM ... 4 jbods (with 48x 8TB HDD and 12x 800GB SSD in each jbod).

Wow. The beefiest setup I've ever seen for PVE.

The question is: what storage organisation is best or suitable? Ceph

Yes, Ceph.

Do i need a lot of OSD's for ceph (how much OSD's i need)?

This is kinda idea setup in which you can use a ratio of 1:4 (SSD-to-Spinner), so it'll run fine.

 

[a4t3rburn3r]

Thx. I was looking for DRBD9. But now will try Ceph. I have already 2 proxmox clusters with Ceph, but they are very old - proxmox 3.x. Main question is what is about SSD journal and OSDs - can i simply use 12 SSD as 12 OSD and 48 HDD as 48 OSD per node and configure 2 crash maps - one fast storage on SSD and one slow storage on HDD? Or i need to dedicate SSD to HDD OSDs journal? How much SSD will be enough in this case? Overall is it normal to have 60 OSDs per node?

 

[a4t3rburn3r]

And about pools - is it enough to make one pool on this bunch of OSDs or better to have several poools? On old cluster i simply have 8 HDDs per node and only 3 nodes, so 24 OSDs and one pool.