PVE 4.3 / zfs storage bottlenecks?

[axy]

Hi All,

I'm seeing very slow apt-get updates inside a Debian 8 KVM guest (iotop displays 100kb write but 99% io load / io wait), dd speed tests (yes I know it's not perfect) is also very slow:

# dd if=/dev/zero of=test bs=64k count=1k oflag=dsync
^C15+0 records in
15+0 records out
983040 bytes (983 kB) copied, 3.52654 s, 279 kB/s​

The guest is used as our DB server and sometimes it takes a very long time to complete a simple query.

Issue only started appearing after upgrading from PVE 4.2 to 4.3.

I can get more than 100mb/s on the host with the same command.

Host runs on ZFS RAID 10 with SSD for L2ARC.

# pveperf
CPU BOGOMIPS: 56002.16
REGEX/SECOND: 2450469
HD SIZE: 1789.49 GB (rpool/ROOT/pve-1)
FSYNCS/SECOND: 14257.64

# pveversion -v

Spoiler

proxmox-ve: 4.3-66 (running kernel: 4.4.21-1-pve)
pve-manager: 4.3-10 (running version: 4.3-10/7230e60f)
pve-kernel-4.4.6-1-pve: 4.4.6-48
pve-kernel-4.4.16-1-pve: 4.4.16-64
pve-kernel-4.4.19-1-pve: 4.4.19-66
lvm2: 2.02.116-pve3
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-47
qemu-server: 4.0-94
pve-firmware: 1.1-10
libpve-common-perl: 4.0-80
libpve-access-control: 4.0-19
libpve-storage-perl: 4.0-68
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-docs: 4.3-14
pve-qemu-kvm: 2.7.0-6
pve-container: 1.0-81
pve-firewall: 2.0-31
pve-ha-manager: 1.0-35
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 2.0.5-1
lxcfs: 2.0.4-pve2
criu: 1.6.0-1
novnc-pve: 0.5-8
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.7-pve10~bpo80

Thank you for your time to look at this problem.

 

[axy]

Might be related: https://github.com/zfsonlinux/zfs/issues/5187

 

[axy]

Update 2 - changing from ZFS directory to ZFS plugin / zvol made speeds better, but still curious why ZFS directory would be this slow?

 

[wolfgang]

Hi,

Update 2 - changing from ZFS directory to ZFS plugin / zvol made speeds better, but still curious why ZFS directory would be this slow?

Do you use qcow2 fileformat?
If yes then the problem is that cow on a cow fs never perform good.