But this honestly would only be a problem in write back mode vs write through. I dealt with this issue back in the DOS days on my Mylex DAC960SUI, and the solution was quite easy--don't use write-back.Well… ZFS - and that goes for Ceph as well, both ensure that the data has made it onto the disks correctly. If you do have a (caching) HW-RAID controller between the disks and ZFS/Ceph, it may be that some operations are flagged to be on stable storage, when they in fact aren't. It may not be of much importance in a lab setup, but I'd never trust my data to any HW RAID controller these days. Pressure on price/performance has been so great, that the vendors seem to have settled on shady compromises regarding safe programming. Especially a series of LSI controllers have been infamous for causing all sorts of trouble and LSI is, what still drives most of the HBAs.
So basically all write operations are a verify read after write (awesome), but that that caching can effectively 'fool' that and can cause issues when data has been written incorrectly--correct?