Hi,
I noticed that the only way to get low iowaits with ceph is to settle VM disk cache to writeback.
But it's not enough, with mysql (innodb) we still have high iowaits on high load. We have to disable barrier on ext4 mount options. After that, disk performance is OK.
On a 5 nodes cluster...