[SOLVED] Abysmal Windows VM performance.

[SUPERgluedCHAIN]

I am getting abysmal performance from within a Windows VM on my Promox cluster. Installing Windows is very slow, taking upwards of an hour to get past even the first stage of the process. After it is done and while idle on the desktop disk, latency can be as much as 3000ms, and it is pretty regularly sitting at 100% usage, and under load, the write speeds sit in the KB/s.

The current setup for the cluster is as follows:

• Three identical nodes - E5-2673 v3, 64GB RAM, SSD containing Proxmox, 2x 16TB Exos drives (ST16000NM001G-2KK103), 10Gb NIC
• Ceph has been set up using the Exos drives

I am new to Ceph, so it's possible, and probable, that there is a configuration issue that I am unaware of.

 

[epytir]

Have you installed the Windows Drivers:
https://fedorapeople.org/groups/vir...ownloads/archive-virtio/virtio-win-0.1.215-1/

Do you use VirtIO SCSI Controller with scsi Disks (you need the drivers for this) ? Because Sata and IDE are super slow.

Also you can test your Ceph Pool on one host with:
Write Test:
rados –p <pool> bench –b <blockSize> <seconds> <write> -t <threads> --no-cleanup
Read Test:
root@node:/# rados –p <pool> bench –b <blockSize> <seconds> <seq> -t <threads>

 

[SUPERgluedCHAIN]

I created a new VM and made sure the drives were set up as SCSI as you have suggested and it hasn't made a huge amount of difference. I seem to be getting 1Mb/s now which is an improvement.

Write test (rados bench 60 write -b 4M -t 16 -p cephfs_data --no-cleanup)

Total time run:         60.6326
Total writes made:      2191
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     144.543
Stddev Bandwidth:       34.2958
Max bandwidth (MB/sec): 212
Min bandwidth (MB/sec): 84
Average IOPS:           36
Stddev IOPS:            8.57396
Max IOPS:               53
Min IOPS:               21
Average Latency(s):     0.440449
Stddev Latency(s):      0.404642
Max latency(s):         3.01181
Min latency(s):         0.052812

Read test (rados bench 60 seq -t 16 -p cephfs_data)

Total time run:       15.3111
Total reads made:     2191
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   572.395
Average IOPS:         143
Stddev IOPS:          18.8498
Max IOPS:             180
Min IOPS:             107
Average Latency(s):   0.11033
Max latency(s):       0.709589
Min latency(s):       0.0199876

Let me if you want me to add any other details.

 

[epytir]

OK I used Ceph for some months and actually had trhe same issues on a 3 node Cluster because of too slow SSDs. Ceph is a monster when is comes to network and diskio. (thats why I switched back to a classic storage).
The problem is that ceph shines when using massive network speed 25G+ and super fast PCIE SSDs and more that 3 nodes but that stuff is much to expensive for my company.

What SSDs you are using?
Is is full flash or also HDDs ?
Do you use Enterprise SSDs ? (I tested a lot with consumer SSDs because I wanted to get a cheap HA storage bug without any good results it was a catastrophy)
Even with nvme disks with 3000MB/s write and 4000MB/s read I only got around 200MB/s in the Ceph with 10Gbit interface. Thats why I stopped using Ceph.

Last have you set to pool to autoscale yes and Target Ratio 1.0 (1.0 means that all the space is planned to be used in this pool so if you only have 1 pool this is perfect)

Greetings Epytir

 

[Dunuin]

• 2x 16TB Exos drives (ST16000NM001G-2KK103), 10Gb NIC
• Ceph has been set up using the Exos drives

So only HDDs.

 

[SUPERgluedCHAIN]

So its using HDDs.

Yeah, they are "spinning rust", as some people lovingly refer to them—Enterprise-grade, but still HDD's.

I have tried the Target Ratio setting but to no avail. I am wondering if following this guide would work:
https://forum.proxmox.com/threads/ceph-bad-performances-only-from-vm-and-ct.62222/

It means rebuilding the storage cluster, but if it works, then I can do it.

 

[epytir]

So in my tests HDDs without SSD Cache is not fast enough and Ceph normally requires at least 3 Hosts with 3 Disks:

To build a hyper-converged Proxmox + Ceph Cluster, you must use at least three (preferably) identical servers for the setup.

https://pve.proxmox.com/wiki/Deploy_Hyper-Converged_Ceph_Cluster

Also in the Guide is mentioned that more disks are better then some big disks.
The only Ceph cluster I have installed for a customer that worked fine with 3 nodes was 8 times 2 TB HDDs per Host with 2 times 1 TB SSD PCIe Cache per Host for DB/WAL to speed up the HDD. Speed was around 300 MB/s what was ok for that scenario but is not that fast for all the hardware that was used (all Enterprise grade stuff).

Unfortunately my experience with Ceph was not great at all because you lose a lot of space because every obejct is stored 3 times and its not that fast.
In the end i use NFS for bigger clusters where we got the money for a HA Storage and ZFS Replication with 2 Hosts for the "cheap" clusters.
With ZFS Replication you can have replica every 5 minutes (with good disks) so in case of a crash you only lose 5 minutes and have HA for the VMs and have the full speed of the disks.

 

[SUPERgluedCHAIN]

After a load of testing I have managed to solve the problem, at least for now. By default the template for Windows does not use a writeback cache. Setting this up and using the Windows hardware settings suggested here mean speeds are much more reasonable, seeing speeds of over 30MB/s and things are far more responsive.

I am not entirely happy with this solution as it seems a little unclear where this cache located before being written to the disk. I am going to be looking into getting some enterprise SSD's and setting up a Ceph cache, but it isn't critical right now.

 

[Dunuin]

I am not entirely happy with this solution as it seems a little unclear where this cache located before being written to the disk.

In RAM. So using a UPS would be recommended then using writeback.