If you'd like a technical explanation:
With O_DIRECT the kernel doesn't cache the data to be written. Instead it passes the pointer down to the storage layer. The software raid in turn passes the same pointer down to the handler for each individual disk. Each of these handlers then uses that pointer to read the data from the buffer independently and asynchronously.
This means that if while one thread is waiting for the write() to finish another one is modifying the data being written, each raid disk - since they are performing the copying independently - might be writing a different state of the buffer.
While this seems like a bad thing to do there are two important things to remember:
a) Any random unprivileged user program in a guest VM can do this, causing the host's RAID to be considered inconsistent/broken.
b) There are cases where you legitimately know before the write() finishes that you won't be needing the data so you don't care about writing a consistent state. The most obvious one (and number 1 reason for the corruption) being swap space: If the kernel starts swapping out memory of, for example, a program which is just about to exit(), the data currently in-flight effectively becomes "useless", and the kernel starts recycling that memory block before the write() finishes, causing the same kind of corruption since the kernel doesn't know that the single physical hard drive it actually sees and thinks its writing the data to is in fact part of a software raid on a hypervisor.
Kernel bug entry:
https://bugzilla.kernel.org/show_bug.cgi?id=99171
The problem is: using an intermediate buffer is basically the opposite of what O_DIRECT is meant to do (although the documentation does state it only makes the kernel
try to skip any such buffers), and locking out other threads sharing the same memory would be a performance killer.
While my personal stance on this is that the
whole point of a RAID1 system is to write the same data to all underlying disks, other people might argue that O_DIRECT is an exception here. The documentation warns that it has to be used with caution, and the open(2) manpage contains a nice quote from Linus:
man 2 open said:
"The thing that has always disturbed me about O_DIRECT is that the whole interface is just stupid, and was probably designed by a deranged monkey on some serious mind-controlling substances."—Linus
