Where can I tune journal size of Ceph bluestore?

raytracy

Member
Mar 31, 2016
19
0
6
56
I have this statement in [global] config:

osd journal size = 5120

But when I create bluestore OSD with journal, it always partition the SSD with 1GB size only:
upload_2018-5-24_23-45-56.png

How can I enlarge the default 1GB size when I create bluestore OSD?
 

Alwin

Proxmox Staff Member
Staff member
Aug 1, 2017
4,617
443
88
osd journal size = 5120
This is for the older OSD backend, filestore.

To increase the size of the WAL/DB, you need to set the bluestore_block_wal_size/bluestore_block_db_size in the ceph.conf. Then all partitions will be created with that size. Give the DB partition as much as you can, as the DB will write overflow to the OSD data partition if there is not more space left. Also the WAL will be put on the fastest device when you create a separate DB partition.

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-November/022328.html
 
  • Like
Reactions: raytracy

AlexLup

Member
Mar 19, 2018
215
12
23
40
This is for 15gb partitions:

/etc/pve/ceph.conf
bluestore_block_db_size = 16106127360
bluestore_block_wal_size = 16106127360
 

richinbg

Member
Oct 2, 2017
28
3
8
30
Hi,
so we have migrated our storage to BlueStore but it is as slow as with filestore - which is 40MBs read/write.
We have three servers with each having 5x8TB HDDs.
Right now the OSDs per DISK are basically using 100%.

So what would you guys recommend then? To have how much space for the block devices for WAL and DB? And then I assume I need to recreate the OSDs?

I have read that if you have a faster drive you could then put those DB and WAL device on the faster disk - so if I do so, again, what size would you recommend to use here? Thank you.
 

richinbg

Member
Oct 2, 2017
28
3
8
30
Hi
Thanks I will have a look. so even though bluestorage all this has to happen?
 

sg90

Active Member
Sep 21, 2018
273
40
33
32
Hi
Thanks I will have a look. so even though bluestorage all this has to happen?

Depends on the performance your after, a 8TB drive will never get high concurrent speeds due to it's size and still only one head. If you was to replace 1 8TB drive with 4 2TB drive's you would see your performance increase as more OSD's / reading head's generally means higher performance.

With the larger drives you get the $/TB down but your also get the performance penalty of this, however putting in a large cache at bluestore level or placing a cache of SSD's in front helps to reduce the latency peaks and offload some of the spikes.

However it all depends on your use case / what your running on the cluster, a SSD Cache won't help if your continuously adding TB's of data, but will help if your running VM's which read and write to a set of data that you can keep most in the hot SSD Cache.
 
  • Like
Reactions: AlexLup

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!