Ceph OSD

TecScott

Member
Mar 30, 2017
17
0
6
29
Currently have filestore OSD's on Proxmox with the journal partition on a SSD drive, this worked fine through the GUI and would create a new partition on the SSD.

I'm trying to re-create the OSD's using bluestore now, so I've deleted one of the OSD's and in the GUI tried to create a new OSD using the disk and SSD as the DB disk however I get prompted 'device is already in use and has no LVM on it (500)'

The SSD has the Proxmox OS installed in one partition, then another partition for the journal for a different OSD.

Is there a way to bypass this error? Do I need to create a partition and assign it using CLI?
 

sg90

Active Member
Sep 21, 2018
240
33
28
30
Yes by default bluestore will try and use the whole physical disk for both the OSD / DB Disk, if you want to use just a partition you need to do this via the CLI as from my understanding/experience it's not supported by Proxmox.

However, one thing to note, with Bluestore the benefit of an SSD is less than with Filestore, if you have multiple OSD's running off a single SSD for the DB you will probably be better off just colocating the DB on the OSD as you remove the single point of failure/performance limit the single SSD causes.
 

TecScott

Member
Mar 30, 2017
17
0
6
29
Does the SSD not provide faster cache/journal still though?

With filestore we had multiple 7.2k 4TB disks per node and a 20GB journal on SSD. If we then move to bluestore and put both OSD + DB on the slower disk I'd imagine performance would be impacted?

Also any idea what the CLI commands would be to set this?
 

sg90

Active Member
Sep 21, 2018
240
33
28
30
Does the SSD not provide faster cache/journal still though?

With filestore we had multiple 7.2k 4TB disks per node and a 20GB journal on SSD. If we then move to bluestore and put both OSD + DB on the slower disk I'd imagine performance would be impacted?

Also any idea what the CLI commands would be to set this?
It provides a much less of a benefit, also the DB works differently to how the journal did. Ideally the min space you will want for a DB partition is 31GB.

I guess your node / replication can survive a whole host OSD failure? Fact a single SSD will cause the whole host of OSD's to go offline.
 

TecScott

Member
Mar 30, 2017
17
0
6
29
Yeah we can tolerate a node failure so in the event an SSD died we'd look to replace the SSD or evict the node from the cluster and re-balance on the remaining OSD's.

Although it's less of a benefit I'd imagine it's still worthwhile over pure slow disks, and I'd imagine the performance of bluestore is still better than filestore so it's worth me upgrading now rather than leaving it on filestore?
 

sg90

Active Member
Sep 21, 2018
240
33
28
30
Yes bluestore is definitely the way to go, and has been around long enough now to be pretty stable and the new default for any new clusters.

However as previously said if you can't get at least a 30ish GB partition per an OSD Bluestore will end up moving all the DB data to the slow disk anyway, the DB works differently than the filestore journal.
 

TecScott

Member
Mar 30, 2017
17
0
6
29
The SSD's are 400GB with 2 OSD's per node, I think around 100GB is allocated for OS + SWAP so should be fine for 30GB per OSD.

Do you know the commands to be able to create this? As the Web UI won't allow me to select the SSD as the DB partition.
 

sg90

Active Member
Sep 21, 2018
240
33
28
30
The SSD's are 400GB with 2 OSD's per node, I think around 100GB is allocated for OS + SWAP so should be fine for 30GB per OSD.

Do you know the commands to be able to create this? As the Web UI won't allow me to select the SSD as the DB partition.
I don't think proxmox supports it even via CLI, you will have to do it using your own separate CEPH cluster and ceph-deploy.

But I may be wrong and it may have been added / support in the Proxmox CLI.
 

TecScott

Member
Mar 30, 2017
17
0
6
29
Struggling to understand the Proxmox logic then.

Their KB states recommendation is to put DB/WAL on SSD or NVRAM for better performance, so are they suggested an individual SSD for every OSD?

Sounds like my best option is to stick with filestore...
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!