(Only) Windows bad diskperformance on PVE 5/Ceph Bluestore

Aug 20, 2015
35
5
73
Germany
I am running a PVE 5-Cluster with Ceph bluestore OSDs. They are only HDD OSDs and connected over 2x1GBit bonds. Don't get me wrong here, it isn't productive yet and I don't expect any fancy performance out of this setup.
I am highly impressed with the performance of it under Linux-KVMs and when I monitor the OSDs with atop on the repective hosts they can quite busy when I run diskintense things on linuxhosts. The cephpanel in the webfrontend gives quite nice IOPS and good read and writeperformance.
I installed a windows server 2016 and tried different settings. The virtio-scsi with cache=none gave me about 20-30MB writeperformance, which is much less than with the linuxclients and looking at atop on the servers they are only about 30 % busy. When I use cache=writeback and copy something onto the windowsmachine it does this with fullspeed of the networklink for some gigabytes but the it stalls for about the time it had run before at 0 MB/s.
Are there any best settings for windowsmachines or is there a bug in the actual luminousversion? Some advice would be great.
 
Windows handles IO differently then linux does, so to have a better test, you should do a test with fio (also for windows available) inside those VMs. You also should use the latest VirtIO drivers, as they usually give better performance. Also ceph has its own cache that can be set through the /etc/pve/ceph.conf file, leave the qemu cache with none (default) and let ceph handle the caching. http://docs.ceph.com/docs/luminous/rbd/rbd-config-ref/

fio: https://github.com/axboe/fio
 
you don't need to edit ceph.conf (anyway, qemu cache option override it)

qemu cache=none -> rbd_cache=false
qemu cache=writeback -> rbd_cache=true
@aderumier, thanks, but a link would have been nice. ;)

You're right, I checked the docs, since qemu 2.4 the behavior changed (support added in qemu 1.2). http://docs.ceph.com/docs/master/rbd/qemu-rbd/

Anyway, if you need to change any of the defaults, then you still need to go and add it to the ceph.conf.
 
I am running a PVE 5-Cluster with Ceph bluestore OSDs. They are only HDD OSDs and connected over 2x1GBit bonds. Don't get me wrong here, it isn't productive yet and I don't expect any fancy performance out of this setup.
I am highly impressed with the performance of it under Linux-KVMs and when I monitor the OSDs with atop on the repective hosts they can quite busy when I run diskintense things on linuxhosts. The cephpanel in the webfrontend gives quite nice IOPS and good read and writeperformance.
I installed a windows server 2016 and tried different settings. The virtio-scsi with cache=none gave me about 20-30MB writeperformance, which is much less than with the linuxclients and looking at atop on the servers they are only about 30 % busy. When I use cache=writeback and copy something onto the windowsmachine it does this with fullspeed of the networklink for some gigabytes but the it stalls for about the time it had run before at 0 MB/s.
Are there any best settings for windowsmachines or is there a bug in the actual luminousversion? Some advice would be great.

Hmm this is exactly what I am noticing aswell....
Did you ever find a solution?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!