KVM guest I/O much slower than host

[axy]

Hi all,

I'm running ZFS on Proxmox 4.1 and for some reasons the I/O inside KVM guest is much slower than on host. The results are fairly consistent.

Host:

Spoiler

root@host:/# dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 0.451532 s, 2.4 GB/s
root@host:/# dd if=/dev/zero of=test bs=64k count=16k oflag=dsync
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 0.456314 s, 2.4 GB/s

Guest:

Spoiler

[root@kvm ~]# dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 2.26477 s, 474 MB/s
[root@kvm ~]# dd if=/dev/zero of=test bs=64k count=16k oflag=dsync
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 10.0685 s, 107 MB/s

Storage configuration on host is the following

Spoiler

root@host:/# cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
maxfiles 0
content images,rootdir,iso,vztmpl

zfspool: storage
pool rpool/zfsdisks
content rootdir,images
sparse

The guest is running on raw (the only disk option available) with no cache. It has 2 CPU cores and 2GB RAM. Hostnode runs on 6x HDD raid 10 and 1 SSD each for cache and log, as you can see below. Should you be wondering why dsync IO is so high, it's because sync is set to disabled.

Spoiler

root@host:/# zpool status
  pool: rpool
 state: ONLINE
  scan: none requested
config:

NAME        STATE     READ WRITE CKSUM
rpool       ONLINE       0     0     0
  mirror-0  ONLINE       0     0     0
    sda2    ONLINE       0     0     0
    sdb2    ONLINE       0     0     0
  mirror-1  ONLINE       0     0     0
    sdc     ONLINE       0     0     0
    sdd     ONLINE       0     0     0
  mirror-2  ONLINE       0     0     0
    sde     ONLINE       0     0     0
    sdf     ONLINE       0     0     0
logs
  sdh       ONLINE       0     0     0
cache
  sdg       ONLINE       0     0     0
root@host:/# pveperf | egrep '(CPU|SECOND)'
CPU BOGOMIPS:      166416.32
REGEX/SECOND:      1593360
FSYNCS/SECOND:     19405.40

I created another guest (on local storage, not ZFS plugin; with qcow2 and writeback cache, and got similar if not worse results).

Any help or advice would be appreciated.

Thanks!

 

[LnxBil]

Sorry, but writing zeros is no test - it's wrong on so many different levels!

Try a real I/O testsuite like fio for workloads.

 

[dabtech]

As LnxBil says, that's not a good benchmark, especially if you're using ZFS compression. Writing zeros to a ZFS dataset using compression will yield overly exaggerated good results.

 

[axy]

Yes I understand there are a myriad of other benchmarking tools available. However they were writing pretty much the same set of data, why would the results differ so much?

 

[fabian]

Because different storage technologies/formats/file systems treat the zeroes you are copying differently. Some might compress or optimize them on the fly, some might not, metadata allocation will differ, etc. Unless your real world use case is copying a lot of zero blocks from A to B, don't treat copying from /dev/zero as a valid benchmark.

 

[axy]

So you are saying ext4 in the guest is the bottleneck? Do you recommend any other FS?

 

[fabian]

No, I am saying you need to establish a valid baseline for your host I/O performance before comparing it to the same benchmark inside a guest. Otherwise you are comparing apples to oranges. Once you have a valid benchmark on the host (again: writing zeroes is not a valid benchmark), you can try different guest (storage) configurations and see what works for your setup and needs and optimize accordingly.