Ubuntu Xenial/Bionic kernel 4.13/4.15 virtio-scsi direct sync 4k write performance regression

[chrone]

Hi Proxmoxers,

Just found out today that when using Ubuntu Xenial kernel 4.13, the direct synchronous write performance is 93% slower than on kernel 4.4. Kernel 4.13 direct sync 4KB write could only achieve 303KB/s while kernel 4.4 could achieve 4.5MB/s.

The VM is using ext4 filesystem and running on top of ZFS ZVOL on ZFS RAID1 with qemu disk cache none and qemu disk discard enabled. The underlying physical disks are Samsung Enterprise SSD SM863a 480GB series.

The reason I need to run kernel 4.13 to mount CephFS Luminous with kernel client. Hopefully CephFS Luminous kernel client will be back ported to kernel 4.4 in the near future.

Is this from Ubuntu Xenial kernel 4.13 performance regression or from Proxmox VE 5.1 kernel 4.13 virtio-scsi performance regression?

I submitted but to Ubuntu linux-hwe as well at https://bugs.launchpad.net/ubuntu/+source/linux-hwe/+bug/1758621

The test results:

##
# ubuntu xenial kernel 4.13. the direct synchronous write performance is 93% slower than kernel 4.4
##
root@localhost:~# uname -a
Linux localhost 4.13.0-37-generic #42~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

root@localhost:~# dd if=/dev/zero of=/mnt/test/test bs=4K count=256 oflag=direct,sync status=progress; rm /mnt/test/test
1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.46369 s, 303 kB/s

##
# ubuntu xenial kernel 4.4. the direct synchronous write performance is 14x faster than kernel 4.13
##
root@localhost:~# uname -a
Linux localhost 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

root@localhost:~# dd if=/dev/zero of=/mnt/test/test bs=4K count=256 oflag=direct,sync status=progress; rm /mnt/test/test
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231176 s, 4.5 MB/s

##
# proxmox 5.1
##
root@proxmox1:~# uname -a
Linux proxmox1 4.13.13-6-pve #1 SMP PVE 4.13.13-42 (Fri, 9 Mar 2018 11:55:18 +0100) x86_64 GNU/Linux

root@proxmox1:~# pveversion -v
proxmox-ve: 5.1-42 (running kernel: 4.13.13-6-pve)
pve-manager: 5.1-46 (running version: 5.1-46/ae8241d4)
pve-kernel-4.13: 5.1-42
pve-kernel-4.13.13-6-pve: 4.13.13-42
pve-kernel-4.13.13-2-pve: 4.13.13-33
corosync: 2.4.2-pve3
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-common-perl: 5.0-28
libpve-guest-common-perl: 2.0-14
libpve-http-server-perl: 2.0-8
libpve-storage-perl: 5.0-17
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 2.1.1-3
lxcfs: 2.0.8-2
novnc-pve: 0.6-4
proxmox-widget-toolkit: 1.0-11
pve-cluster: 5.0-20
pve-container: 2.0-19
pve-docs: 5.1-16
pve-firewall: 3.0-5
pve-firmware: 2.0-4
pve-ha-manager: 2.0-5
pve-i18n: 1.0-4
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.9.1-9
pve-xtermjs: 1.0-2
qemu-server: 5.0-22
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.6-pve1~bpo9

root@proxmox1:~# zpool status -v
  pool: rpool
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sda2    ONLINE       0     0     0
            sdb2    ONLINE       0     0     0

errors: No known data errors


root@proxmox1:~# pveperf /
CPU BOGOMIPS:      134101.44
REGEX/SECOND:      2697203
HD SIZE:           389.89 GB (rpool/ROOT/pve-1)
FSYNCS/SECOND:     4479.87
DNS EXT:           49.29 ms
DNS INT:           66.89 ms

 

[chrone]

Just tested again on new Ubuntu 18.04 with kernel 4.15. The direct synchronous small 4K write performance is still slower than kernel 4.4 on Ubuntu 16.04.

Are kernel 4.13 and 4.15 having write performance issue on qemu 2.11.x?

##
# ubuntu bionic kernel 4.15. the direct synchronous write performance is still 93% slower than kernel 4.4 on ubuntu xenial.
##

root@localhost:~# dd if=/dev/zero of=/tmp/test bs=4K count=256 oflag=direct,sync status=progress; rm /tmp/test
1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.504 s, 299 kB/s
root@localhost:~# dd if=/dev/zero of=/tmp/test bs=4M count=256 oflag=direct,sync status=progress; rm /tmp/test
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.00597 s, 357 MB/s
root@localhost:~# uname -a
Linux localhost 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:16:15 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux