SSD drive cache

[uk_user]

Hi,

We have a 25 node Proxmox cluster using Ceph as the storage with 200 Samsung PM883 1.92TB SSD drives as OSD's. We are troubleshooting an issue with lower than expected IOPS which isn't a major issue day to day but has caused us problems if Ceph needs to rebalance (eg drive swap).

While troubleshooting it has been discovered that write cache is turned on (hdparm -W /dev/XXX). I suspect this used to be disabled but has changed since the nodes got rebooted a while ago.

1. From reading around it seems to be recommended that write cache is disabled across all drives - is this still the recommendation?

2. Is there a way of setting this so it continues to apply after reboots?

3. Is it safe to disable cache on all drives (hdparm -W 0 /dev/XXX) within a live cluster outside of a maintenance window, ie is there any chance of service interruption?

Thanks

 

[Maximiliano]

Hello, could you provide us more context please, what Proxmox VE and Ceph versions are you using? Whats your Ceph setup (size, OSDs per node, etc),

Regarding disabling write cache, I am not sure to be honest. I only found an issue [1] with a fix [2] adding to the documentation a recommendation to try disabling the write cache and benchmark the changes to see if it improves performance. Do note that the issue only affect certain drives according to the bug report. You can also take a look at [3].

[1] https://tracker.ceph.com/issues/53161
[2] https://github.com/ceph/ceph/pull/4...eef3856f535d094658b61cc8ca66bfa567afb582cfebc
[3] https://forum.proxmox.com/threads/disabling-write-cache-on-ssds-with-ceph.112767/

 

[uk_user]

Thanks for your reply. Our drop in Ceph IOPS is quite a long running issue (we have had a ticket with Proxmox in the past) so I may start a new thread with all versions and metrics we have been gathering to show the issue.

My main question was regarding the SSD write cache and whether it was safe to disable write cache using "hdparm -W 0 /dev/xxx" on a live ceph cluster or is there a chance this will cause ceph/storage issues?

Thanks again.