Ceph Pacific - High Latency on Large Disks

[mcdowellster]

Hello There,

I've had a cluster running since 2016 and been upgrading storage. The SSD pool has been excellent since day one but my spinning disks are causing me some serious headaches.

My largest drives are two 14TB WDC WD140EDFZ. These are almost ALWAYS at 250MS. They are on the same HBA as the SSDs in the cluster. This is really only happening since moving from Nautilus.

Any tuning / thoughts?

 

[aaron]

You could try to recreate them one by one and see if this helps. Are they already bluestore OSDs or still filestore?

 

[mcdowellster]

Last night I moved all my data off the spinning disks, blew away the spinning disk pools, spent several hours troubleshooting random weird problems.

After finally getting everything going zapping and wiping the spinning discs and re-adding them I'm having a tremendous performance improvement. Not entirely sure why. Everything has been bluestores for years. One thing I did notice though it seemed like there was always a scrub going on all the time.

I guess we'll see soon once the scrubs start happening on the spinning discs if the same problem happens again.

 

[aaron]

Okay. Interesting. Another approach could have been to set one HDD OSD to out, wait until Ceph becomes healthy again, then destroy it (don't forget to check if the "Cleanup disk" checkbox is enable, or if you use pveceph osd destroy, to set the parameter. Then recreate the OSD, wait for Ceph to become healthy again -> continue on to the next OSD.

 

[spirit]

I known than on pacific,
bluestore_min_alloc_size_hdd has been changed from 64k to 4k,
with the new avl allocator.

Maybe it could have impact on some workload (I can't tell as I'm only use nvme/ssd)

it's still possible to revert to old param in ceph.conf

[OSD]
bluestore_allocator = bitmap
bluefs_allocator = bitmap
bluestore_min_alloc_size_hdd=65536