We have 7 nodes with 6 having 7 ssd's each. the ssd's are Intel S3610 .
We have cronscripts to check for issues that in the past have led to data corruption. there is always a time issue in the vm before corruption. so the script checks for time discrepancies . shortly before those occur there are lines like the following in logs / dmesg :
The drives are always part of the ceph pool and not rpool or zfs.
For some reason in the last 3 weeks we have had timing / scsi issues about 3 times per week. In the 4 months before that we had 3 time issue emails.
From searches on "task abort: SUCCESS scmd" - some suggest firmware upgrade.
Now we can not easily upgrade the firmware in these drives as they came from Dell systems and we are using Supermicro. The Intel firmware upgrade program will not work with anything but Intel retail drives [ non oem ]. And the dell uprade program only works when the drives are in a supported dell system [ we no longer have any dell systems ].
So in our case having a very reliable and fast NVMe PCIe journal may alleviate the issue we have.
Using a journal drive is a SPOF. however with 7 nodes and relatively little storage traffic [ mainly accounting programs with 30 concurrent users ] we are OK for some time if a node dies. [ We have shutdown a node to replace motherboards a few times].
On each node , the lsi hba also deal with traffic from a zfs mirror used for video recordings. Before adding the hournal to ceph I am doing things like changing settings to sync=disabled . If that does not help i may use a single drive formatted to xfs for recordings.
So I would still like to know how to add a journal drive to an existing ceph pool in case we decide to do so.