ZFS: .raw better than zvol?

[cpzengel]

Hi,

we just tried the new Server 2016 with ZFS on Proxmox 4.4.5

While zvol (ZFS Dataset) is recommended we figured out that a .raw File on a ZFS Dataset has better write Performance.
Both Disks were configured with Write Back Cache in vm.conf

While the virtual Disk with .raw wrote about 3GB contuniously on Windows and Proxmox, the zvol had some interruptions on writing.

So what is the official current reccomodation?

Hint: If you are still using qcow2 with Server2008 and R2, beware of horrible system hangs while RPC Usage like Filesharing on SMB, RDP and Exchange!

Any Ideas?

proxmox-ve: 4.4-78 (running kernel: 4.4.35-2-pve)
pve-manager: 4.4-5 (running version: 4.4-5/c43015a5)
pve-kernel-4.4.13-1-pve: 4.4.13-56
pve-kernel-4.2.6-1-pve: 4.2.6-36
pve-kernel-4.4.13-2-pve: 4.4.13-58
pve-kernel-4.4.35-2-pve: 4.4.35-78
pve-kernel-4.4.21-1-pve: 4.4.21-71
pve-kernel-4.2.8-1-pve: 4.2.8-41
pve-kernel-4.4.24-1-pve: 4.4.24-72
pve-kernel-4.4.19-1-pve: 4.4.19-66
lvm2: 2.02.116-pve3
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-48
qemu-server: 4.0-102
pve-firmware: 1.1-10
libpve-common-perl: 4.0-85
libpve-access-control: 4.0-19
libpve-storage-perl: 4.0-71
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-docs: 4.4-1
pve-qemu-kvm: 2.7.0-10
pve-container: 1.0-90
pve-firewall: 2.0-33
pve-ha-manager: 1.0-38
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 2.0.6-5
lxcfs: 2.0.5-pve2
criu: 1.6.0-1
novnc-pve: 0.5-8
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.8-pve13~bpo80​

 

[wolfgang]

Hi,

use zvol without cache on the VM disk setting.
ZFS has it own cache.

 

[cpzengel]

now zvol seems ok, but some lags on write
i´ll keep an eye on it
thanks wolfgang

 

[cpzengel]

So today we had very bad Performance on zvol without cache. Turned to Writethrough and System worked ok.
Whats wrong there?

 

[wolfgang]

Have you enabled dedup?
This is often a reason for bad write performance.

 

[cpzengel]

no, we have horrible performance at many clients with many settings. we need optimal setup for the future!

 

[cpzengel]

perhaps you have 10 mins for a short phone call? +496021200888

 

[wolfgang]

Can you send the output of

zpool status

 

[cpzengel]

we just removed the cache ssd and same problem and at some other customers suffer from horrible zfs performance in windows vms



root@KGPM236:~# zpool status

pool: rpool

state: ONLINE

scan: resilvered 9.84G in 0h2m with 0 errors on Tue Mar 29 17:15:41 2016

config:


NAME STATE READ WRITE CKSUM

rpool ONLINE 0 0 0

mirror-0 ONLINE 0 0 0

sdb2 ONLINE 0 0 0

sdc2 ONLINE 0 0 0

mirror-1 ONLINE 0 0 0

sdd ONLINE 0 0 0

sdf ONLINE 0 0 0

mirror-2 ONLINE 0 0 0

sdg ONLINE 0 0 0

sdh ONLINE 0 0 0

 

[cpzengel]

same on my machine with 4 x 2tb constellation on zvol with no cache. host is freezing for seconds.

 

[wolfgang]

Do you have enough memory for ZFS?
How many VM do you have on this pool?

 

[cpzengel]

we have write issues on any customer at the moment

The current Machine has

128GB RAM, dual Octacore E5-2630v3, Intel Board
4 Windows Machines running, 3 SRV2008, 1 SRV2016
70GB Memory are free
ARC is set to 16GB
Cache SSD is currently removed because of testing, normally uses as log and cache
PVE is current, see above

My Machine has same issues

.RAW with writeback seems to work okay, but not good.
zvol with horrible write rate

i am willed to pay a one time ticket for all our custumers
is this possible
my number is above

we have many customers with issues since newer versions

 

[mir]

scan: resilvered 9.84G in 0h2m with 0 errors on Tue Mar 29 17:15:41 2016

Seems to be in order for running a scrub on your pool!!!

 

[cpzengel]

this was just a test, what should a scrub do on performance

 

[wolfgang]

We have no ticket to buy.

see
http://www.proxmox.com/en/proxmox-ve/pricing

 

[cpzengel]

we are using virtio latest fedora drivers for windows

so wolfgang
what is the current zfs recommoteation for windows?

 

[cpzengel]

https://pve.proxmox.com/wiki/Windows_2012_guest_best_practices

steht zum beispiel man soll writeback nehmen, gilt das auch für zfs?

 

[cpzengel]

Solution found
In my honest opionion its a problem by design and now a problem of my integration
so pve-1 and vms are located at same raid
on high load inside a guest or a benchmark situation at the host, the guests and the host are freezing for seconds.

i solved it by installing proxmox to a mirror of cheap ssds in the system

now the guests are not failing at all while high load occurs at the same zfs raid

feature request
please consider to put the os itself to a kind of ramdisk and only hold /etc/pve on pve-1 while system is up