Ceph block.db and block.wal

[JackScreecher]

Hello, I'm looking over the Proxmox documentation for building a Ceph cluster here...

https://pve.proxmox.com/wiki/Deploy_Hyper-Converged_Ceph_Cluster

There is a small section entitled block.db and block.wal which says...

If you want to use a separate DB/WAL device for your OSDs, you can specify it through the -db_dev and -wal_dev options. The WAL is placed with the DB, if not specified separately.

I was wondering if anyone knows how much of a performance advantage is gained by doing this? The NVME drives I'm using are Seagate Firecuda 530's and they are PCIv4 7000+ MB/s. Do you think it's still a good idea to have an additional drive for the journal? My Ceph cluster will be dedicated - so not used as local storage. The HV's will be separate.

 

[ness1602]

Unfortunately, Firecuda is a consumer not an enterprise product,so i guess you won't get much performance out of it

 

[JackScreecher]

Unfortunately, Firecuda is a consumer not an enterprise product,so i guess you won't get much performance out of it

I appreciate what you say, but I think It's the perfect drive for this type of project. This drive uses the high end Phison PS5018-E18 Controller. It's TLC not QLC and it has a 5100TBW endurance rating. It's a very high performance drive. Yes it may be marketed at the desktop market but it's still a good choice for a system like this due to the redundancy of Ceph. I'd say these are as good, if not better than many enterprise drives on the market right now. As it's PCIe 4 the drive is already twice as fast as any PCie 3 drive. It has LDPC, ECC and End-to-End Data Path Protection. Ticks all the boxes for me.

Anyway, my question relates to how necessary it is to use a separate device for DB/WAL and whether this is something people would always recommend.

Perhaps someone with experience of using Proxmox and Ceph could spare a moment to comment. Many thanks.

 

[ness1602]

Yes, it is always recommended to use db/wal, proxmox in the gui uses some percent for it automatically.

 

[JackScreecher]

Yes, it is always recommended to use db/wal, proxmox in the gui uses some percent for it automatically.

Sorry my question was whether to use a separate drive for this. I'm looking to see other people's experience of running db/wal from the osd's compared to using an additional partitioned drive.

It isn't necessary to do it but the documentation suggests there is a performance benefit, although it isn't clear how much. As I'm using the fastest NVME drives available, capable of 7000MB/s I wonder if I really need to run the journal on an additional drive.

I'm hoping to hear someone else's experience of this.

 

[mvs]

This drive uses the high end Phison PS5018-E18 Controller. It's TLC not QLC and it has a 5100TBW endurance rating. It's a very high performance drive. Yes it may be marketed at the desktop market but it's still a good choice for a system like this due to the redundancy of Ceph.

Consumer SSDs don't have power loss protection (PLP). Such SSDs are either slow at sync writes or their controller treats all writes as async internally (may cause data corruption).

Anyway, my question relates to how necessary it is to use a separate device for DB/WAL and whether this is something people would always recommend.

You may achieve better performance, if you locate DB/WAL on a drive with much faster low queue depth sync writes (like Intel Optane NVMe or NVDIMM/PMEM drive). But Ceph has many bottlenecks, so there is no guarantee. You can test this scenario with a ramdisk, if you have sufficient RAM on your servers.
On the other hand separate DB/WAL increases complexity and a probability of an OSD failure (single OSD depends on multiple drives).

 

[JackScreecher]

Thanks for your replies.

The Ceph documentation does say It is only useful to use a WAL / DB device if the device is faster than the primary device (e.g., when it is on an SSD and the primary device is an HDD).

I'll do some testing.

 

[ness1602]

OR if you have enterprise ssd and regular ssd.

 

[JackScreecher]

OR if you have enterprise ssd and regular ssd.

Yes. Do you know of an enterprise NVME drive that will be fast enough to make a difference?

 

[ness1602]

Intel 3710 lets say

 

[JackScreecher]

Intel 3710 lets say

That's a SATA drive.

 

[ness1602]

SSD D7-P5510​

This one should do it for nvme

 

[filipealvarez]

Please share the results with Firecuda 530, I buy 4 KC3000 for dB/wall too

 

[JackScreecher]

The KC3000 is almost identical to the 530 so I imagine it will be very similar. The Intel D7-P5510 is a good enterprise drive but it has lower write performance and most likley wouldn't be fast enough to make a difference. I'll do some testing. Thanks for your efforts ness1602

 

[ness1602]

I would never trust kingston in enterprise environment, but thats just my experience.

 

[james58899]

Using a separate DB device saved nearly half the IOPS in my previous tests, but IOPS is usually not the bottleneck for SSD.
So I don't think SSD needs separate DB/WAL devices, which increases the possibility of failure.

 

[gummiost]

Did any of you test the performance difference between having the DB/WAL on a separate drive and having the OSD and DB/WAL on the same drive?