Jumping to share my $0.02 as I've been heavily researching this myself, a newbie as others here are...
First, let's state for the record what exact SSD hardware you need for any and all journal (FileStore), block.db, block.wal, etc device, including just a straight SSD added as an OSD for usage:
Enterprise SSD drives, and nothing else. No, your lot of Samsung 980 Pro M.2 are not enterprise drives nor any mainstream consumer drive you've used - and your performance will be horrible. Enterprise SSDs are a whole different level. Mainly because of one fact that you can dig into the tech details of your SSD to verify how they handle the Power Loss Prevention. Enterprise SSDs have super-capacitors that keep power to the device for a short time after the machine looses power - enough to write everything in its queue to disk before loosing power. Consumer drives, like the Samsung 980 Pro M.2, uses a transaction log.
Why is Power Loss Prevention key to performance for the CEPH cluster, as a backing disk or the
block.db? Because CEPH immediately issues an
flush/f-sync after each write. An Enterprise Drive will acknowledge this
f-sync immediately without waiting to flush, as it has the super-capacitor to guarantee writes. Whereas consumer SSDs, like the Samsung 980 Pro M.2, will not - it will flush everything to NAND first, and then acknowledge. This tiny small handling of
f-sync is paramount to CEPH performance, and is the key to 10 to even 30x the performance. Are you getting horrible IOPS or Bandwidth in your Ceph cluster? This is most likely why, unless your
block.db device is too small.
For the record, you can pickup inexpensive Enterprise drives on eBay. I got over a dozen Samsung PM883 240GB at $30/ea (usually $50-ish) by waiting nearly 6 months for a "Lot of" deal. There's also the older PM863s, and many Intel DC drives. Even SATA2 3Gb/s drives are fine as you want the IOPS and f-sync instance-response, not really the bandwidth. Btw, you want the smaller of the disks, like the 240GB. Read on for why.
Now, some facts from
the official CEPH docs related to the BlueStore Provider (I'm assuming everyone here is using BlueStore).
A quote from the page to clear up DB/WAL:
The BlueStore journal will always be placed on the fastest device available, so using a DB device will provide the same benefit that the WAL device would while also allowing additional metadata to be stored there (if it will fit). This means that if a DB device is specified but an explicit WAL device is not, the WAL will be implicitly colocated with the DB on the faster device.
So, we only need to specify the
block.db device.
About that sizing...
barius said:
The DB/WAL is small compared to the main storage (10% ???)
Byron said:
In CEPH docs, I've seen 4% recommended, I think the recommended minimum is ~30GB per OSD.
Actually,
the CEPH docs recommends:
The general recommendation is to have block.db size in between 1% to 4% of block size. For RGW workloads, it is recommended that the block.db size isn’t smaller than 4% of block, because RGW heavily uses it to store metadata (omap keys). For example, if the block size is 1TB, then block.db shouldn’t be less than 40GB. For RBD workloads, 1% to 2% of block size is usually enough.
So for RGW workloads, aka RADOS Gateway, aka AWS S3, you'd want no less than 4% of the total backing block device.
If you really in a bind and really need RGW but can't afford the Enterprise SSDs for
block.db devices, you could get away with a smaller 1-2% of block size, and use
RGW Data Caching. That's my plan anyways if I ever need RGW.
barius said:
Is it a bad idea to share an SSD for the DB/WAL of several OSDs? My main concern is that if (when!!!) the SSD fails, all the OSDs with their DB/WAL on the same SSD will have to be destroyed and rebuilt. Right?
Yes, that is correct. I took this fact, combined with the 2% sizing requirement of the Enterprise SSDs (no RGW), and the maximum number of OSDs the chassis can install (8x hot swap), and came up with my own formula of:
4x 4TB disks (expandable to 6x disks)
2x 240GB Enterprise SSDs (80GB partitions for each
block.db)
At 2% of 4TB, that's an 80GB partition. 240GB / 80GB = 3x partitions each SSD. So, 2x 240GB SSDs allows me a total of 6x 4TB HDDs in the chassis. Considering I am building 3x chassis, and I already have 12x 4TB drives, this works out for me.
Also, i could swap that 8x 3.5" hot-swap supermicro chassis for an 12x 3.5" how-swap chassis, and get a total of 9x 4TBs with 3x 240GB SSDs. Perfect math and expand-ability.
There's one other redundancy fact hidden in this setup: if I loose an SSD, that means I only loose 2x OSDs (or 3x OSDs if I expand to 6x disks). This is why I chose 240GB, and not the 480GB drives - well, that and cost! To be clear, I purposely bought 2x 240GB instead of one single 480GB, as I plan to partition the 240GB with 2x 80GB partitions for now, leaving open 80GB for any future 4TBs I add. Also, the deal I got on the 240GB drives was killer, and two were much cheaper than a single 480GB. So, win-win all around.
Everyone's systems, nodes, and needs are different. Do the math (2% of not using RGW, 4% if using RGW - or more for heavy workloads, or using RGW cache).
I will be setting them up on Nextcloud for photos/docs/drive sharing internally. Some of you may say, "But, you absolutely want to use RGW with NextCloud, it's built for that using the S3 interface!" Actually, my use case is their 40+ years of digitized photos and old videos already organized in a massive collection. NextCloud has zero ordering/folders/etc in their S3 backend, dumping everything to a root folder with obtuse names. Hence, a straight File System object store, not S3.
---
Remember one final fact: BlueStore will use default/prefer the writes to
block.db device and f-sync. However, there are times BlueStore will bypass your
block.db all together and go straight to the backing device. IIRC, things like streaming.
