Poor disk performance on Linux VM

[andrea.cservice]

Hi,

After reading many threads, I decided to open one to try to solve a performance problem that only affects Linux VMs.
I have a Proxmox VE 6.5-5 installed on Dell R720 with 2 Xeon E5-2660 v2 2.20GHz, 128 GB RAM and RAID controller PERC H710 Mini (integrated).
RAID1 volume for the system and RAID5 volume for the VMs with 3TB SAS disks with battery backup e writecache option enabled.

This is the pveperf test on root (RAID1)
root@pve1:~# pveperf CPU BOGOMIPS: 176038.20 REGEX/SECOND: 1359212 HD SIZE: 93.99 GB (/dev/mapper/pve-root) BUFFERED READS: 134.80 MB/sec AVERAGE SEEK TIME: 7.56 ms FSYNCS/SECOND: 1991.33

This is pveperf on RAID5
root@pve1:~# pveperf /dev/raid5-lvm/ CPU BOGOMIPS: 176038.20 REGEX/SECOND: 1365320 HD SIZE: 62.90 GB (udev) FSYNCS/SECOND: 41061.86

The server hosted
- 3 Windows VMs (WS 2019) with the Virtio-SCSI driver and WriteCache option enabled and have no performance problems.
- a Linux VM (Centos7) with the same hardware features and disk performance is very low.

This is the test of RAID5 LVM:



Thi is th VM config:

agent: 1 balloon: 0 boot: order=scsi0;ide2;net0 cores: 4 cpu: host ide2: none,media=cdrom memory: 16384 name: SRVLINUX net0: virtio=CE:62:9C:85:93:09,bridge=vmbr0 numa: 0 onboot: 1 ostype: l26 scsi0: raid5-lvm:vm-104-disk-0,cache=directsync,discard=on,size=600G scsihw: virtio-scsi-pci smbios1: uuid=b8902838-ba75-4137-a57b-1899ec08a623 sockets: 2 startup: order=4 vmgenid: e0bea4db-6714-48ed-a98a-c09b60559d1a

I tried different mod on Centos
- upgrade Kernel to version 5.16
- setting noop on file /sys/block/sda/queue/scheduler
- disable swap
and tried different setting on disk e memory configuration on Proxmox.

Several combinations same results:



The dd command was given by a second company that manages the software installed on the VM: a result of between 300 and 400 MB/s indicates the correct functionality of their software.

What can I do?

Thanks

 

[oguz]

hi,

scsi0: raid5-lvm:vm-104-disk-0,cache=directsync,discard=on,size=600G

try disabling directsync cache option [0], use a faster one

(safest but slowest (relative to the others))

cache=directsync.

    host don't do cache.
     guest disk cache mode is writethrough
     similar to writethrough, a fsync is made for each write.

   This mode causes qemu-kvm to interact with the disk image file or block device with both O_DSYNC and O_DIRECT semantics, 
     where writes are reported as completed only when the data has been committed to the storage device, and when it is also desirable to bypass the host page cache.
     Like cache=writethrough, it is helpful to guests that do not send flushes when needed.
     It was the last cache mode added, completing the possible combinations of caching and direct access semantics.

hope this helps!


[0]: https://pve.proxmox.com/wiki/Performance_Tweaks

 

[spirit]

directsync bypass all cache, including the cache from your raid controller (even with writeback enabled + battery)

 

[andrea.cservice]

I also tried setting writethrough, writeback, directsync, unsafe. Activate SSD emulation and Discard with the same result.
The above configuration was the one that gave the best results.

I don't know if it is correct to get this answer from the host with my HW configuration: