Howto add DB device to BlueFS

[cmonty14]

Hi,

I have created OSD on HDD w/o putting DB on faster drive.

In order to improve performance I have now a single SSD drive with 3.8TB.

Questions:
How can I add DB device for every single OSD to this new SSD drive?
Which parameter in ceph.conf defines the size for the DB?
Can you confirm that this parameter in ceph.conf will create 100GB for DB for each OSD?
[osd]
osd journal size = 102400
How can I increase the DB size later in case it's insufficient?

THX

 

[RokaKen]

Hi,

I have created OSD on HDD w/o putting DB on faster drive.

In order to improve performance I have now a single SSD drive with 3.8TB.

Questions:
How can I add DB device for every single OSD to this new SSD drive?

I have not validated the procedure, but you can attempt Migrating Bluestore block.db

Which parameter in ceph.conf defines the size for the DB?

bluestore_block_db_size

Can you confirm that this parameter in ceph.conf will create 100GB for DB for each OSD?
[osd]
osd journal size = 102400

I can confirm that WILL NOT do so -- that is for filestore not bluestore.

 

[cmonty14]

OK.
I modified /etc/ceph/ceph.conf by adding this in [global]:
bluestore_block_db_size = 53687091200
This should create RockDB with size 50GB.

Then I wanted to move DB to a new device (SSD) that is not formatted:
root@ld5505:~# ceph-bluestore-tool bluefs-bdev-new-db –-path /var/lib/ceph/osd/ceph-76 --dev-target /dev/sdbk
too many positional options have been specified on the command line

Checking the content of /var/lib/ceph/osd/ceph-76 it appears that there's no link to block.db:
root@ld5505:~# ls -l /var/lib/ceph/osd/ceph-76/
insgesamt 52
-rw-r--r-- 1 ceph ceph 418 Aug 27 11:08 activate.monmap
lrwxrwxrwx 1 ceph ceph 93 Aug 27 11:08 block -> /dev/ceph-8cd045dc-9eb2-47ad-9668-116cf425a66a/osd-block-9c51bde1-3c75-4767-8808-f7e7b58b8f97
-rw-r--r-- 1 ceph ceph 2 Aug 27 11:08 bluefs
-rw-r--r-- 1 ceph ceph 37 Aug 27 11:08 ceph_fsid
-rw-r--r-- 1 ceph ceph 37 Aug 27 11:08 fsid
-rw------- 1 ceph ceph 56 Aug 27 11:08 keyring
-rw-r--r-- 1 ceph ceph 8 Aug 27 11:08 kv_backend
-rw-r--r-- 1 ceph ceph 21 Aug 27 11:08 magic
-rw-r--r-- 1 ceph ceph 4 Aug 27 11:08 mkfs_done
-rw-r--r-- 1 ceph ceph 41 Aug 27 11:08 osd_key
-rw-r--r-- 1 ceph ceph 6 Aug 27 11:08 ready
-rw-r--r-- 1 ceph ceph 3 Aug 27 11:08 require_osd_release
-rw-r--r-- 1 ceph ceph 10 Aug 27 11:08 type
-rw-r--r-- 1 ceph ceph 3 Aug 27 11:08 whoami

root@ld5505:~# more /var/lib/ceph/osd/ceph-76/bluefs
1

How can I create DB on SSD device for existing OSDs?

THX

 

[Alwin]

Some of the prospects, I have written on your other post [0].

To create a new DB on another device, use your command with an prefixed variable assignment [1]. Your command seems to only have the wrong dash assigned (–-path).

Besides the addition of the DB, you need to make sure that the LV tag for the ceph.db_device is updated, otherwise the OSD won't start.

As said in the other thread, re-creating the OSD might be easier and it will make sure that the on-disk format for the OSDs are current.

[0] https://forum.proxmox.com/threads/bluefs-spillover-detected-on-30-osd-s.56230/#post-264623
[1] https://www.spinics.net/lists/ceph-users/msg54305.html