Slow random reads in Window VMs

[Goldklump]

Hi all,

I have a Proxmox 6 cluster with 3 nodes, and the read performance on the Windows VMs is slow.
Each node has 192GB RAM and 4 SSD. Two SATA SSD for the OS and two fast NVME U2 SSD for the VM data.
I first tried with ZFS, but the performance was not great. So I tried CEPH. Sequential read/writes are somewhat better than with ZFS, but the random reads are still slow. See screenshot.
The Linux VMs get about 50-65% of the host speed and their reads are faster than writes. I tried various performance tweaks and suggestions I found for proxmox, but so far no success. The Windows VMs use the latest stable VirtIO drivers, the images are in "raw" format, the disks are VirtIO with WriteBack.
Any idea what to check next?

 

[apoc]

what exactly is your concern? The 4.77 MB/s on 4k read?
How does the tool actually work?

Is it only one thread (as I read from the next it uses 64 threads)? Or is it talking about the queue-depth of the test?
It might be the case that this is your limit. If you have one outstanding IO in your pipe, you are waiting for the IO to be generated, not the execution of that IO. This is the reason why I hate such tests.
Easy to setup, but basically the numbers are worth nothing, because you can't specify the test and measure what is of interest.

Throughput -> you need queuedepth to get the max number from the system. Large Blocks
Number of IO -> small IO, large queuedepth again.
latency? -> measure individual IO - minimal queuedepth

Also it seems you measure your C-Drive. That means typically you have a filesystem-layer in between which will totally ruin your test. Caches, overhead, etc.

If you test, do it right. But not with a consumer utility

 

[Goldklump]

You are right, that the tool is too simple.
I used it, because I saw it a few times on the forum.

The solution was to disable C-states in BIOS and Linux.