pve on NVME and partions for ceph journal

[flexyz]

Hi

I have a 4 node server each with one PCI NVME slot and 6 SAS/SATA 2.5" bays pr. node.

Can I install PVE on the NVME and use the same NVME as ceph journal?

NVME is a Intel Solid-State Drive DC P3700 Series - Solid state drive - 400 GB - intern - PCI Express 3.0 x4 (NVMe)"

Or should I instead use a SSD in one of the BAYS for PVE or vice-versa

Thanks

 

[tom]

As long as you use SSD for your OSD, no extra journal is needed. Use bluestore for your SSD only cluster.

 

[flexyz]

I was planning to use 6x2TB spinning disks OSD's pr. node - then I need a SSD journal for each OSD right?

Thanks

 

[tom]

In this case I would install Proxmox VE on a extra SSD (needs to be fast and reliable, as also the ceph monitors will be there), e.g. a Samsung SM863 with 240 GB.

So you can use the P3700 as Ceph journal for your filestore OSDs (HDDs).

 

[flexyz]

Great and the smallest NVME intel have is 400GB - that is maybe overkill, or?

Thanks!

 

[tom]

Great and the smallest NVME intel have is 400GB - that is maybe overkill, or?

Thanks!

yes. I suggest you build a SSD cluster running bluestore. e.g. with Samsung SM863, also available in 2 TB

 

[flexyz]

So no need for dedicated journal SSD when runnning bluestore? - I will dig into bluestore

Thanks again

 

[flexyz]

If I choose to enable "journal" to as SSD with bluestore on PVE, I guess it will move the RockDB etc. there?

 

[flexyz]

I tested some basic creation of OSD on a virtual PVE and when I choose "journal" on another drive it creates a 1G partition - no matter how big the OSD disk. So it looks like some RockDB or? - and is it big enough

 

[Alwin]

As long as the 1GB is not full, the writes to the RocksDB happen to the separate disk/partition. Once it is full, it will continue on the data disk/partition and hence slow down. The size of the RocksDB and WAL can be set in the ceph.conf.