(Only) Windows bad diskperformance on PVE 5/Ceph Bluestore

[nibblerrick]

I am running a PVE 5-Cluster with Ceph bluestore OSDs. They are only HDD OSDs and connected over 2x1GBit bonds. Don't get me wrong here, it isn't productive yet and I don't expect any fancy performance out of this setup.
I am highly impressed with the performance of it under Linux-KVMs and when I monitor the OSDs with atop on the repective hosts they can quite busy when I run diskintense things on linuxhosts. The cephpanel in the webfrontend gives quite nice IOPS and good read and writeperformance.
I installed a windows server 2016 and tried different settings. The virtio-scsi with cache=none gave me about 20-30MB writeperformance, which is much less than with the linuxclients and looking at atop on the servers they are only about 30 % busy. When I use cache=writeback and copy something onto the windowsmachine it does this with fullspeed of the networklink for some gigabytes but the it stalls for about the time it had run before at 0 MB/s.
Are there any best settings for windowsmachines or is there a bug in the actual luminousversion? Some advice would be great.

 

[Alwin]

Windows handles IO differently then linux does, so to have a better test, you should do a test with fio (also for windows available) inside those VMs. You also should use the latest VirtIO drivers, as they usually give better performance. Also ceph has its own cache that can be set through the /etc/pve/ceph.conf file, leave the qemu cache with none (default) and let ceph handle the caching. http://docs.ceph.com/docs/luminous/rbd/rbd-config-ref/

fio: https://github.com/axboe/fio

 

[aderumier]

Also ceph has its own cache that can be set through the /etc/pve/ceph.conf file, leave the qemu cache with none (default) and let ceph handle the caching. http://docs.ceph.com/docs/luminous/rbd/rbd-config-ref/

you don't need to edit ceph.conf (anyway, qemu cache option override it)

qemu cache=none -> rbd_cache=false
qemu cache=writeback -> rbd_cache=true

 

[Alwin]

you don't need to edit ceph.conf (anyway, qemu cache option override it)

qemu cache=none -> rbd_cache=false
qemu cache=writeback -> rbd_cache=true

@aderumier, thanks, but a link would have been nice.

You're right, I checked the docs, since qemu 2.4 the behavior changed (support added in qemu 1.2). http://docs.ceph.com/docs/master/rbd/qemu-rbd/

Anyway, if you need to change any of the defaults, then you still need to go and add it to the ceph.conf.

 

[Norman Uittenbogaart]

I am running a PVE 5-Cluster with Ceph bluestore OSDs. They are only HDD OSDs and connected over 2x1GBit bonds. Don't get me wrong here, it isn't productive yet and I don't expect any fancy performance out of this setup.
I am highly impressed with the performance of it under Linux-KVMs and when I monitor the OSDs with atop on the repective hosts they can quite busy when I run diskintense things on linuxhosts. The cephpanel in the webfrontend gives quite nice IOPS and good read and writeperformance.
I installed a windows server 2016 and tried different settings. The virtio-scsi with cache=none gave me about 20-30MB writeperformance, which is much less than with the linuxclients and looking at atop on the servers they are only about 30 % busy. When I use cache=writeback and copy something onto the windowsmachine it does this with fullspeed of the networklink for some gigabytes but the it stalls for about the time it had run before at 0 MB/s.
Are there any best settings for windowsmachines or is there a bug in the actual luminousversion? Some advice would be great.

Hmm this is exactly what I am noticing aswell....
Did you ever find a solution?