To increase the size of the WAL/DB, you need to set the bluestore_block_wal_size/bluestore_block_db_size in the ceph.conf. Then all partitions will be created with that size. Give the DB partition as much as you can, as the DB will write overflow to the OSD data partition if there is not more space left. Also the WAL will be put on the fastest device when you create a separate DB partition.
so we have migrated our storage to BlueStore but it is as slow as with filestore - which is 40MBs read/write.
We have three servers with each having 5x8TB HDDs.
Right now the OSDs per DISK are basically using 100%.
So what would you guys recommend then? To have how much space for the block devices for WAL and DB? And then I assume I need to recreate the OSDs?
I have read that if you have a faster drive you could then put those DB and WAL device on the faster disk - so if I do so, again, what size would you recommend to use here? Thank you.
Depends on the performance your after, a 8TB drive will never get high concurrent speeds due to it's size and still only one head. If you was to replace 1 8TB drive with 4 2TB drive's you would see your performance increase as more OSD's / reading head's generally means higher performance.
With the larger drives you get the $/TB down but your also get the performance penalty of this, however putting in a large cache at bluestore level or placing a cache of SSD's in front helps to reduce the latency peaks and offload some of the spikes.
However it all depends on your use case / what your running on the cluster, a SSD Cache won't help if your continuously adding TB's of data, but will help if your running VM's which read and write to a set of data that you can keep most in the hot SSD Cache.