Terrible performance in Windows VM

[cossali98]

Hello guys, after trying every trick that I found on Internet, I'm going to open a thread...
I'm having terrible terrible terrible performance in Windows VM. The system is really slow, every action that I do it takes a lot of time. I noticed an increment of IO Delay in Proxmox Dashboard when I turn on the Windows VM. Sometimes it arrives even at 60% and this make the VM really really slow.

Actually I'm using proxmox at home but I'm doing some tests in order to use it for some customers.

Host configuration:
HP DL380 G9
2x Intel Xeon E5-2640 v4 10c/20t
128GB DDR4
HP Smart Array P440ar controller in IT mode
4x SSD Fanxiang S101 512Gb (before I was using HDD Enterprise 15k, same result)

Storage configuration:
SSD configured in raidz1
zpool create -f -o ashift=12 -m /mnt/ZFS_VM/ ZFS_VM raidz1 /dev/sdd /dev/sde /dev/sdf /dev/sdg cache /dev/sdb2 log /dev/sdb1
Autotrim=on
Compression: lz4

I tried to configured the VM following the tips indicated in the Wiki:
https://pve.proxmox.com/wiki/Windows_10_guest_best_practices
https://pve.proxmox.com/wiki/Performance_Tweaks

Windows VM configuration
agent: 1 balloon: 0 bios: ovmf boot: order=ide0;scsi0;net0 cores: 2 cpu: host efidisk0: ZFS_VM:vm-110-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M hotplug: disk,network,usb,cpu ide0: none,media=cdrom machine: pc-i440fx-7.0 memory: 8192 meta: creation-qemu=7.0.0,ctime=1665304616 name: Win01 net0: virtio=FA:2E:D7:6D:2E:F3,bridge=vmbr0,tag=30 numa: 1 ostype: win10 scsi0: ZFS_VM:vm-110-disk-1,cache=writeback,discard=on,iothread=1,size=90G scsihw: virtio-scsi-pci smbios1: uuid=b54439b7-a96d-4f3d-92f5-86335f7777bf sockets: 1 tablet: 0 tags: prod vga: qxl vmgenid: 0796e401-d9f0-4696-82f8-1cb12e49993d

The main problems detect are:
- General slow doing normal operation
- Disk percentage that rise up to 100%
- Slow copies from network and between VMs (tested with Windows ISO file)
- High IO delay in proxmox

I tried to using even LVM but the results seem worse...

The linux VMs don't seem to be affected.

Please help me to solve this issue that is driving me crazy.

 

[_gabriel]

ZFS on consumer SSD topic of this week !
last week
other week

@ pve devs : Please , to prevent flood forum , add a warning directly in the PVE installer "Be sure to use Entreprise SSD with power protection otherwise ZFS will be slow. You've be warned"

Regular ext4 (lvmthin will used for vm disks), can be faster but less secure in case of power failure or crash. Like regular desktop/laptop.
But with theses cheap SSD with QLC nand, write performance is terrible as you said.
Benchmarking Storage will confirm.

 

[cossali98]

ZFS on consumer SSD topic of this week !
last week
other week

@ pve devs : Please , to prevent flood forum , add a warning directly in the PVE installer "Be sure to use Entreprise SSD with power protection otherwise ZFS will be slow. You've be warned"

Regular ext4 (lvmthin will used for vm disks), will be faster but less secure in case of power failure or crash. Like regular desktop/laptop.

As I wrote, I already tried to use LVM but the performance are the same.
So I'm not understanding if it's a problem related to storage type or to bad configuration of VM

 

[_gabriel]

Nothing to understand.
How cheap SSD can be fast ??
Benchmarking Storage will confirm

edited : seems TLC nand not the worst QLC. but I doubt they are faster than a not recommended Samsung QVO...

 

[mircolino]

If your Windows VM is running Windows Security, disable Device Security -> Core Isolation -> Memory Integrity
Then in Control Panel -> Programs and Features, turn off the Windows Hypervisor Platform feature

The following is the qm config I use:

agent: 1
bios: ovmf
boot: order=scsi0;scsi1
cores: 2
cpu: qemu64
efidisk0: local-zfs:vm-102-disk-1,efitype=4m,pre-enrolled-keys=1,size=1M
machine: pc-q35-7.1
memory: 4096
meta: creation-qemu=7.1.0,ctime=1676549155
name: xxxx
net0: virtio=xx:xx:xx:xx:xx:xx,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win11
scsi0: local-zfs:vm-102-disk-0,cache=writeback,discard=on,size=64G
scsi1: none,media=cdrom
scsihw: virtio-scsi-pci
smbios1: uuid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
sockets: 1
tpmstate0: local-zfs:vm-102-disk-2,size=4M,version=v2.0
vga: virtio

 

[cossali98]

If your Windows VM is running Windows Security, disable Device Security -> Core Isolation -> Memory Integrity
Then in Control Panel -> Programs and Features, turn off the Windows Hypervisor Platform feature

The following is the qm config I use:

agent: 1
bios: ovmf
boot: order=scsi0;scsi1
cores: 2
cpu: qemu64
efidisk0: local-zfs:vm-102-disk-1,efitype=4m,pre-enrolled-keys=1,size=1M
machine: pc-q35-7.1
memory: 4096
meta: creation-qemu=7.1.0,ctime=1676549155
name: xxxx
net0: virtio=xx:xx:xx:xx:xx:xx,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win11
scsi0: local-zfs:vm-102-disk-0,cache=writeback,discard=on,size=64G
scsi1: none,media=cdrom
scsihw: virtio-scsi-pci
smbios1: uuid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
sockets: 1
tpmstate0: local-zfs:vm-102-disk-2,size=4M,version=v2.0
vga: virtio

The two features are already disabled.
Can I ask you which disks are you using?

 

[cossali98]

Nothing to understand.
How cheap SSD can be fast ??
Benchmarking Storage will confirm

edited : seems TLC nand not the worst QLC. but I doubt they are faster than a not recommended Samsung QVO...

I understand that these SSD aren't the fastest on the market, but we're talking about a simple copy over network from the NAS... I don't think that this simple operation could cause a so high IO delay.
I'm using it in my homelab and I don't think that I need to buy enterprise SSDs to test ZFS...

 

[mircolino]

Can I ask you which disks are you using?

I installed PVE on an Intel NUC 11 (i7-1165G7) with 64GB of RAM. The disk is a 2TB Samsung 980 PRO formatted with ZFS.

My Windows 11 VM (with the latest virtio drivers installed) runs really well and it's very snappy.
Since I'm running a single node I also disabled all the clustering and replication services. I routinely monitor the SSD wear with smartctl -a /dev/nvme0n1 and I find it very reasonable.