Huge IO performance degradation between proxmox ZFS host and WS19 VM

[Progratron]

For more than a week, I am trying to determine the reason for the following IO performance degradation between proxmox host and a Windows Server 2019 VM(s).

I have to ask for your help guys because I've run out of ideas.

Environment data:

• Single proxmox host, no cluster, pve 6.1-8 with ZFS
• A few WS19 VMs all having this issue, very low load, SOHO-usage
• ZFS sync=disabled, volblocksize for VM disks = 4k
• VM has all the latest VirtIO drivers (0.1.173)

IO test on both VM and host with the following fio command:

fio --filename=test --sync=1 --rw=$TYPE --bs=$BLOCKSIZE --numjobs=1 --iodepth=4 --group_reporting --name=test --filesize=1G --runtime=30

Results:

	Host	VM virtio scsi single, cache=none, discard
4K random read	573,00 MiB/s	62,50 MiB/s
64K random read	1.508,00 MiB/s	831,00 MiB/s
1M random read	1.796,00 MiB/s	1.730,00 MiB/s
4K random write	131,00 MiB/s	14,10 MiB/s
64 random write	596,00 MiB/s	62,50 MiB/s
1M random write	1.081,00 MiB/s	75,40 MiB/s
4K sequential read	793,00 MiB/s	56,20 MiB/s
64K sequential read	1.631,00 MiB/s	547,00 MiB/s
1M sequential read	1.542,00 MiB/s	2.036,00 MiB/s (?)
4K sequential write	240,00 MiB/s	3,42 MiB/s
64K sequential write	698,00 MiB/s	43,80 MiB/s
1M sequential write	1.088,00 MiB/s	223,00 MiB/s

As you can see almost all IO is drastically affected, the only exception is 1M reads. Writing performance is horrible.

Here is it graphically:

Spoiler

Writes:



Reads:

What I have tried so far: different volblocksizes on ZFS, different ZFS sync setting (left it on disabled, since the host is in DC), virtio-blk vs virtio scsi single (not much difference), writeback cache (became even worse).

Any suggestions what am I missing?

 

[aaron]

What is the config of the VM? qm config <vmid>

different ZFS sync setting (left it on disabled

If you like your data I would not do that :/

 

[Progratron]

What is the config of the VM? qm config <vmid>

agent: 1
balloon: 8192
bootdisk: virtio0
cores: 2
cpu: host,flags=+pcid
ide2: local:iso/virtio-win-0.1.173-9.iso,media=cdrom,size=384670K
memory: 16384
name: test-ru
net0: virtio=0A:63:FA:82:9F:91,bridge=vmbr0
net1: virtio=3E:44:3D:88:E4:54,bridge=vmbr0
numa: 1
ostype: win10
scsi0: local-zfs_64k:vm-210-disk-1,discard=on,iothread=1,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=3c47d662-6ae3-48b9-b665-af53544b94b6
sockets: 2
vga: std,memory=512
virtio0: local-zfs_4k:vm-210-disk-0,discard=on,size=96G
virtio1: local-zfs_4k:vm-210-disk-1,discard=on,iothread=1,size=32G
vmgenid: bd48ba99-ca3e-419c-9799-ed01d93b9e01

There are different disks to see from my tests.

If you like your data I would not do that :/

Well, I thought all I am risking is 5 seconds of data... Even somewhere here I've read that when a server is in DC and has redundant power supply it should be fine. Am I missing something?

 

[guletz]

Even somewhere here I've read that when a server is in DC and has redundant power supply it should be fine. Am I missing something?

... and do you have an UPS(and the coresponding tool in PMX so the PMX can do a clean shutdown)

 

[guletz]

A few WS19 VMs all having this issue, very low load, SOHO-usage

Hi,

Can you define what is "A few WS19 VMs" (how many and exactly what OS) and what is running in this VMs(maybe a DataBase, a speciall application, whatever)?
In wich condition do you make this tests witch fio?
How do you install the WS19(=Windows Server 2019?), you have format IN ADVANCE your virtual HD or not?
Also it would be interesting to show:

1. hardware server(cpu/RAM/HDDs/nvme/....)
2. zpool status -v
3. zpool history|grep create
4. zpool history|grep set
5. arc_summary


Good luck / Bafta !