Windows performance on Proxmox - sanity check

[dkong]

Hello STH forum, I have a question about running Windows on Proxmox because this is giving me a bad time so far (mostly related to storage IO performance).

Some relevant specs of the box I'm running this on:
Intel server motherboard
Xeon E3-1230 v3
32 GB ECC UDIMM
2x 120GB Kingston SSD as a ZFS mirror for boot
2x 250GB Samsung Pro 840 as two separate ZFS zpools for VM's
2x 1TB spinning rust as a ZFS mirror for backups and general data

Right now I have one zpool with one dataset in it, for VM's, mounted like this: /ssd1/vms
This dataset is imported in Proxmox via the GUI as a directory in contrast to as a ZFS location. My reason for this is that I wanted to use a qcow2 disk.

That brings me to the VM settings. I'm running Windows 10 Pro (ISO for the 1709 version installation). The VM has one disk (75GB) as qcow2 on the SSD dataset. The storage driver is VirtIO block, I had to import the driver for this into the Windows install, I've used the most recent stable VirtIO ISO from the Fedora project site (version 0.1.141). The virtual disk is configured to use writeback cache because I've followed some recommendations, even though I don't like using writeback I can accept the risks for this Windows VM.

Now my issue is this: storage performance sucks. I've fiddled with the above parameters a bit and it's gotten better, but still unusable as far as I'm concerned. Now what I don't understand is that running crystaldiskmark inside the VM gives very good looking figures. Updating Windows 10 to 1803 slows the whole thing down to a crawl though, never seeing faster write speeds to the disk (according to task manager) than 20 MB/s.. BTW this box is running nothing else at this time, just this single Windows VM (configured with 8GB RAM and 2 cores).

Does anybody know what I'm doing wrong? I can think of these different things to try:
Use the latest VirtIO drivers instead of the most recent stable ones
Have proxmox consume the ZFS dataset "directly" instead of as a directory
Use a raw disk instead of a qcow2 one
Skip ZFS for VM's altogether and just run LVM + EXT4 or XFS on the VM SSD's
I can of course try each different combination but it will take me weeks and I'd like to get this box running. Ideally, I can get acceptable storage performance without needing to have any form of cache on the virtual disk.

One thing I forgot to add is that when viewing the node summary, I also noticed the I/O delay get up to 30% during the Windows update installation

 

[guletz]

You forgot many things

The first thing is why do you want to use zfs? What is your goal regarding storage? Do you start to read about storage/virtualisation/proxmox?

Some stupid things of mine
- if you use zfs, you must use raw format and no others like qcow2
- if you use zfs for win , you must use a dataset with volume block size like 32-64 k.

Because you have 2x spinning drive, I will create a raid0 zfs pool with 2×ssd 240 G. I will also create another zfs pool for 2x 1 Tb disks(mirror). Then I will replicate any vm from ssd pool to the hdd pool. So in case that ssd pool wil be broken then you could start your vm on hdd pool.

 

[dkong]

Thank you guletz. I have general virtualization and storage knowledge (used to work with VMware professionally a lot). I've also done some reading and experimenting on ZFS.

The block size you talk about, is that the thing you set upon zpool creation with the ashift setting? (I believe it defaults to 12 on modern systems)
I am actually on ZFS right using qcow2, because I've added the ZFS location that's mounted on the host as a directory in Proxmox, in contrast to just adding the ZFS target directly.

My main reason for using ZFS for the VM SSD's is consistency, the boot drives are a ZFS mirror and of course the spinning disks too. I wouldn't have a problem moving the VM SSD's to another FS, I am just curious to discover my I/O bottleneck and if it's possible to get decent performance out of the SSD's using ZFS.

I wouldn't like to use RAID0, the performance would be nice and of course I can back up stuff, but it impacts availability as well. Maybe it wouldn't be too bad though, I appreciate the idea and might use it anyway. But it's not a solution to the strange performance bottleneck I'm experiencing now. In this case I can move all my stuff off of Windows and go Linux only, but I'm using this experience as a Proxmox evaluation too and would like to use it for some of my customers in the future, where Windows would be a requirement.. So unless I solve this I have to stick to VMware.

 

[guletz]

The block size you talk about, is that the thing you set upon zpool creation with the ashift setting? (I believe it defaults to 12 on modern systems)

No, it is not about ashift. It is a dataset/zvol proprety:

zfs set recordsize=16k rpool/16K

I am actually on ZFS right using qcow2, because I've added the ZFS location that's mounted on the host as a directory in Proxmox, in contrast to just adding the ZFS target directly.

qcow2 is not OK if you use a zfs storage, because you will have 2 different storage layers who are COW(copy on write). Because of this your guest will have bad I/O.

 

[dkong]

No, it is not about ashift. It is a dataset/zvol proprety:

zfs set recordsize=16k rpool/16K



qcow2 is not OK if you use a zfs storage, because you will have 2 different storage layers who are COW(copy on write). Because of this your guest will have bad I/O.

OK, I set the record size of my ssd1/vms dataset to 32k. When I mount the dataset in the Proxmox storage GUI I get to choose a block size too. The default is 8k. I have tested two scenario's, leaving this to the default and setting it to 32k (all with the dataset at 32k as well). I'm also using a RAW disk now, as it's the only thing supported for a ZFS dataset mounted this way. So far, no improvements yet. I can already tell during the windows installation when it says "getting files ready" the IO delay on the Proxmox box is already going above 20% at least..

-edit- I've also tried the different versions of VirtIO now, and using no cache or using cache as write-back. Nothing seems to improve the I/O delay.

 

[guletz]

Ok. Do like this:
- remove from DataCenter/Storge your ssd1/vms(and save any VM that you have)
- destroy your ssd1/vms (save any VM that you have)
- create again the same dataset(zfs create ssd1/vms)
- then import/add this new dataset in PMX(Datacenter/Storage), and when you will fill the form, you will see in that window a field for block size(put here 32k)
- check the config file from path(using command line) /etc/zfs/storage.cfg, it must look like this for this new dataset:

....
zfspool: local-32k-zfs
pool ssd1/vms
blocksize 32k
content images,rootdir
sparse 0
....

 

[dkong]

Ok. Do like this:
- remove from DataCenter/Storge your ssd1/vms(and save any VM that you have)
- destroy your ssd1/vms (save any VM that you have)
- create again the same dataset(zfs create ssd1/vms)
- then import/add this new dataset in PMX(Datacenter/Storage), and when you will fill the form, you will see in that window a field for block size(put here 32k)
- check the config file from path(using command line) /etc/zfs/storage.cfg, it must look like this for this new dataset:

....
zfspool: local-32k-zfs
pool ssd1/vms
blocksize 32k
content images,rootdir
sparse 0
....

Thanks, mine is looking like this now:
zfspool: ssd1-vms
pool ssd1/vms
blocksize 32k
content images
sparse 0

I tried two more tests, first with thin provisioning and then without. Doesn't the recordsize for the new dataset need to be adjusted to 32k blocks too in your last example?
Sadly, so far still above 20% IO delay in all test cases.

 

[guletz]

OK, how old is your SSD? Do you used before you start create zfs pool?

 

[dkong]

OK, how old is your SSD? Do you used before you start create zfs pool?

The SSD is a few years old, it served as a datastore in my ESXi box before. It didn't see heavy usage, some passive FreeBSD VM's I only used for things like IRC clients, and one Windows VM.

In the meantime I have removed all ZFS partitions from the SSD and tried EXT4 just to see the difference. Now I have no issues at all with I/O latency or bad performance. For my current use case I can live with this but I still really want to know what went wrong. I have some time and some resources available to try out more things, if other people can run Windows VM's on ZFS I should be able too right?

 

[guletz]

In the meantime I have removed all ZFS partitions from the SSD and tried EXT4 just to see the difference

... and after you put the ext4 on SSD, plese run:

fstrim -v -a

Also can you post your Proxmox Win config file?

 

[dkong]

... and after you put the ext4 on SSD, plese run:

fstrim -v -a

I did this, it gave the following output:

root@proxmain:/# fstrim -v -a
/ssd1: 212.3 GiB (227918135296 bytes) trimmed

Is this something I should throw in a monthly cron job or something?

 

[guletz]

Is this something I should throw in a monthly cron job or something?

Yes. Anyway, when you use for the sirst time any SSD for a zfs pool, is highly recomended to format with ext4 and then run a fstrim on it.
After the fstrim you could use it in a zfs pool. Generally speking, a non-trimmed SSD can lead to very low I/O in time!

 

[dkong]

Thanks again, I'm learning lots of useful things from you, I won't let the new knowledge go to waste. Do ZFS-partitioned SSD's need/support the fstrim operation too? Or is this something the ZFS fs will handle for me?

I will try ZFS one more time with a trimmed disk to see if it makes any performance difference.

 

[guletz]

Thx. for your good thoughts about me

zfs can not do fstrim(maybe in the future it will be), so in my case, I remove 1 SSD from a zfs mirror, and I make a ext4 and a fstrim. After this operations, I put back the SSD.

 

[dkong]

I ran the ZFS test again with a trimmed SSD. Performance doubled but was still worse than EXT4. I keep getting spikes above 20% IO delay, but not all the time like before doing fstrim. I think it's improved enough to work with, but it's still just one VM and I like my EXT4 results better for this.

This must mean that there is still something causing the big difference between ZFS and EXT4.

 

[guletz]

You must understend that you can not compare apple with oranges. zfs have a lot of others functions(checksums, zfs send/recive, auto healing, compression, and so on) that will not find in EXT4/whatever. Also keep in mind that after your VMs is start, the zfs caching will be better(hit rate).
Your spikes could be also hardware corelated somehow(SSD firmware, server controller, and so on).
In my own case I run some VMs( win2012-server ) who are running DBs on a HDD mirror, but I do not see spikes like in your case. You could make a test in your case, using the hdd zfs pool(make a backup before) for the same VM, and then see if you get this spikes!