Bad performance on VM disk with Ceph

[Samuele Bistoletti]

Hi,
I made some benchmark on our cluster, for some reason we have very slow disk perfomance inside the VM if compared with rados bench test performed in the same pool.

That is the result of rados bench 30 write -b 4M -t 16 -p test_pool


And that is a dd write performed inside a VM running on the same pool:


Pool is mounted as RBD without KRBD flag on Proxmox.
Proxmox running on version 6.4-13.
Ceph running on version 15.2.14.
We are running 10Gb network with dedicated NIC for Ceph public and cluster net.
Cephx is disabled.
No other VM is running on that pool.

Any idea why VM is so slow?
We have the same problem with "SSD" pool (the speed on VM is approximately 1/4 of the speed reported by rados bench).

 

[mira]

Try krbd, it might be faster.
Also, do you use `writeback` cache on the disks?

 

[Samuele Bistoletti]

Already tried KRBD but with no gain in terms of read performance.

The VM disk is in Default (no cache) mode.

 

[mira]

How is the performance with a single thread when running rados bench? (remove the `-t` option from the command)
That should be the performance you can expect in the guest (minus a bit of overhead).

 

[Samuele Bistoletti]

I think the default thread is 16. I made the test with a single thread, performance is comparable with the VM:



Is it safe to enable writeback mode on VM cache? What kind of side effects could we have in case of power loss or problems with the node running the VM?

Also KRBD si safe to be enabled in production? I noticed a huge improvement in terms of writing performance (800 MB/s with SSD pool compared to the 190 MB/s without KRBD enabled)..

 

[Samuele Bistoletti]

We made some tests, performances are really better with KRBD, but the questions is: is it safe to be enabled in production (how is the cache managed in case of power outage)?

Many thanks

 

[mira]

Enabling KRBD is safe, but enabling `Writeback` cache can lead to data loss.