[SOLVED] Slow write I/O for Windows guests

[godzilla]

Hi,

PVE 7.4 here. There's something wrong with my Windows guests. No matter the underlying hardware, I have very poor performance in I/O for write operations. Like, 1.3GB/s in read and about 100MB/s in write. I tested using both my Ceph storage and local SSDs.

I have latest VirtIO drivers installed on the guest and tested with lot of different disk/controller setups (VirtIO, SCSI, with/without iothreads and so on). Always same results.

As a comparison, a Linux guest on the same cluster makes about 2GB/s writes depending on the dd parameters.

What would you do if you were in my place? Maybe CPU model/flags? I'm using qemu64 for compatibility right now with default flags: I can try a different one but I can't go by trial and error.

Thanks!

 

[godzilla]

Hi, please ignore this thread. I found the root cause was that my benchmark tool wrote random data instead of plain zeroes, and it seems the guest OS is very slow in generating random numbers.

I'm going to find a clue about the slow random data generation and open a new thread in case I need help.

Thank you!

 

[leesteken]

Hi, please ignore this thread. I found the root cause was that my benchmark tool wrote random data instead of plain zeroes, and it seems the guest OS is very slow in generating random numbers.

I'm going to find a clue about the slow random data generation and open a new thread in case I need help.

Apologies for answering here, but VMs can have a problem with getting enough entropy (for random numbers) because they typically have very little hardware devices and no interactive users. Try adding a virtual random number generator: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_virtio_rng. That can help a lot with adding entropy from the host to the VM.
EDIT: You might also get an improvement by using host (or something similar) as the CPU Type, if it's still at kvm64 or similar, because it enables more (efficient) CPU instructions.

 

[Dunuin]

Apologies for answering here, but VMs can have a problem with getting enough entropy (for random numbers) because they typically have very little hardware devices and no interactive users. Try adding a virtual random number generator: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_virtio_rng. That can help a lot with adding entropy from the host to the VM.
EDIT: You might also get an improvement by using host (or something similar) as the CPU Type, if it's still at kvm64 or similar, because it enables more (efficient) CPU instructions.

Do you know if the guestOS will automatically use the virtio RNG or has there something to be done inside the Win/Linux guestOS to actually make use of it? I've got some VMs where I added it but I got no idea if it is actually working.

 

[leesteken]

Do you know if the guestOS will automatically use the virtio RNG or has there something to be done inside the Win/Linux guestOS to actually make use of it? I've got some VMs where I added it but I got no idea if it is actually working.

Windows does need a VirtIO driver. QEMU wiki states that you can check with cat /sys/devices/virtual/misc/hw_random/rng_current on Linux. For me it worked out of the box, but I have not noticed it doing much. Maybe because most of my VMs use a CPU Type with a hardware RNG instruction available?

 

[godzilla]

Apologies for answering here, but VMs can have a problem with getting enough entropy (for random numbers) because they typically have very little hardware devices and no interactive users. Try adding a virtual random number generator: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_virtio_rng. That can help a lot with adding entropy from the host to the VM.
EDIT: You might also get an improvement by using host (or something similar) as the CPU Type, if it's still at kvm64 or similar, because it enables more (efficient) CPU instructions.

No need to apologize! Yes I attached /dev/urandom to a virtio-RNG device for my VM, but noticed no significant improvements (all virtIO drivers installed ofc).

Unfortunately I need to use the most generic CPU type because I have a cluster with (sligthly) different nodes, I can't risk not being able to live-migrate.

Anyway, all is well that ends well

 

[leesteken]

Unfortunately I need to use the most generic CPU type because I have a cluster with (sligthly) different nodes, I can't risk not being able to live-migrate.

Using the new x86--64-v2-AES (or higher) might improve the pseudo-random generation quite a bit. It was added to be a good common denominator (to prevent migration issues). What CPUs are you using?