Ceph - RBD caching

[David Herselman]

How do I establish whether or not RBD access is running in writeback mode?

We define VMs with default caching (none) and have validated that Ceph is configured with writeback caching (using defaults):

[admin@kvm5a ~]# for f in /var/run/ceph/ceph-osd.*.asok; do ceph --admin-daemon $f config show; done | grep 'rbd_cache"\|rbd_cache_writethrough'
    "rbd_cache": "true",
    "rbd_cache_writethrough_until_flush": "true",
    "rbd_cache": "true",
    "rbd_cache_writethrough_until_flush": "true",
    "rbd_cache": "true",
    "rbd_cache_writethrough_until_flush": "true",
    "rbd_cache": "true",
    "rbd_cache_writethrough_until_flush": "true",

Ceph documentation (http://docs.ceph.com/docs/master/rbd/rbd-config-ref/) indicates that only Linux kernel 2.6.32 or later sends flush instructions but RedHat appear to have backported 'VIRTIO_BLK_F_FLUSH' to RHEL/CentOS 5.6 in 2010.

We run virtual Check Point vSEC gateways, which are based on RedHat Enterprise Linux 5. They appear to operate very slowly and I'm attempting to troubleshoot this and make necessary changes but wish to avoid possible data corruption problems.


Is there a way I can validate whether or not a VM is running with writeback caching or not?

 

[Alwin]

Qemu overrides ceph cache options: http://docs.ceph.com/docs/master/rbd/qemu-rbd/#qemu-cache-options
You don't really have stats here, you need to run a test, with the different options to see what is faster for you. Not only the kernel needs to support the flush, also the FS needs to be able to cope with it (eg. VM reset).