Improving small write performance, safe to use write-back caching with Ceph on HDD?

[proxale]

Users had been complaining about laggy VM performace on our HDD-based Ceph pool. With many users VMs having many applications that are logging to disk, I see a lot of small writes with write IOPS > 5x read IOPS.

Reading up here and elsewhere, it seems that Write-back may yield better performance than the default no-cache.
The concern here is that it had been long ingrained in me that write caching is not safe without BBU. However, with Ceph having a WAL on PLP NVMe for the pool, would I be right to assume that this would make it safe since the WAL on NVMe serves as a crash-resilient write cache and PLP is effectively BBU for the NVMe?

 

[Alwin Antreich]

The disk cache and the Qemu writeback cache are two different things.

Usually disabling the disk cache yields better latency for requests (aside from RBD on HDDs not being fast at any rate). While you have PLP for your DB/WAL the HDDs may not (some of them do though). The risk of losing data is relative, as you usually have multiple copies on other nodes. Though the OSD may see data in an inconsistent state (a repair is necessary).

On VM level the writeback cache [0] will keep dirty pages in memory till they are flushed to disk. When the node crashes these pages are lost but the data on the disk is crash consistent. The performance will increase for the VM as write operations can return earlier, if they aren't sync writes (or overly frequent flushes). Since Ceph Pacific there is a write-around cache option which will only cache writes but reads are directly from the storage. This is the default and used when you set cache=writeback on the VM.

[0] https://pve.proxmox.com/wiki/Performance_Tweaks