Strong i/o delays on ZFS VMs

[frankz]

Hello everyone, I have a VM in particular Zoneminder. I noticed that when I make backups or commit the machine to some operations, the system has very strong trills.

I have long used a maximum configuration for the ZFS cache defined in /etc/modprobe.d/zfs.conf:

Options zfs zfs_arc_max=8589934592

Here are the results of use with arcstat and arc_summary:




I wanted to get an opinion if I made a mistake or you can improve the performance of the VM.

 

[Dunuin]

ZFS got alot of overhead and the low IOPS performance of the HDDs can easily be the bottleneck causing a very high IO delay. Especially if you use a raidz1/2/3 instead of a striped mirror as there the IOPS performance won't scale with the number of disks.

What does your pool look like (zpool status)?

You could enable relatime (zfs set relatime=on YourPool) so not every read will cause an additional write. This might help with the IOPS a bit.
Enabeling lz4 compression is also nearly always useful. And your volblocksize should match your pool layout or you will get padding overhead.
Adding SSDs to store the metadata ("special device") might help too to reduce the IOPS that will hit th HDDs.

If you want to look if your ARC is big enough you can run arc_summary to get detailed statistics.

 

[frankz]

ZFS got alot of overhead and the low IOPS performance of the HDDs can easily be the bottleneck causing a very high IO delay. Especially if you use a raidz1/2/3 instead of a striped mirror as there the IOPS performance won't scale with the number of disks.

What does your pool look like (zpool status)?

You could enable relatime (zfs set relatime=on YourPool) so not every read will cause an additional write. This might help with the IOPS a bit.
Enabeling lz4 compression is also nearly always useful. And your volblocksize should match your pool layout or you will get padding overhead.
Adding SSDs to store the metadata ("special device") might help too to reduce the IOPS that will hit th HDDs.

If you want to look if your ARC is big enough you can run arc_summary to get detailed statistics.

Thank you for responding.

This is what I know:

zpool status
pool: zshare
state: ONLINE
scan: scrub repaired 0B in 01:11:52 with 0 errors on Sun Feb 13 01:35:54 2022
config:

NAME STATE READ WRITE CKSUM
zshare ONLINE 0 0 0
ata-ST1000DM010-2EP102_W9AKDG5L ONLINE 0 0 0
errors: No known data errors


root@pve4:~# zfs get all zshare |grep compression
zshare compression on local


The configuration is not in mirror, but at the moment a single disk in stripe .
As for arc_summary, I attach file .
However, unfortunately, the disc is mechanical, so performance is limited compared to an SSD.

 

[RolandK]

>the system has very strong trills.

could you describe more exact what's meant by that?

 

[Dunuin]

ARC looks fine and HDD is CMR so no idea why your IO wait is that high. You could install sysstat and run iostat to check how hard the HDD is hit by IOPS. Maybe your HDD is simply to slow and the IOPS performance is bottlenecking.

 

[RolandK]

also mind that if your workload is sync writes performance may suffer when there is no SLOG installed

furthermor, if you see your VM lagging, getting hiccup and ping got jitter, you can set your VM controller/disk to different mode, which makes behaviour of the VM better under high IO load (virtio-scsi-single/aio=threads/iothread)

 

[frankz]

ARC looks fine and HDD is CMR so no idea why your IO wait is that high. You could install sysstat and run iostat to check how hard the HDD is hit by IOPS. Maybe your HDD is simply to slow and the IOPS performance is bottlenecking.

Sorry, it almost seems like the record is faster or I misunderstood? ??

 

[frankz]

also mind that if your workload is sync writes performance may suffer when there is no SLOG installed

furthermor, if you see your VM lagging, getting hiccup and ping got jitter, you can set your VM controller/disk to different mode, which makes behaviour of the VM better under high IO load (virtio-scsi-single/aio=threads/iothread)

Here are the VM settings:

 

[Tmanok]

Something else to note, you are swapping 1GB in your first screenshot. Your system does not appear to have enough memory, the PVE memory bar in summary does not show arc cache, if you open HTOP you will see all your memory is wired and cached.

Arc Cache behaves a little more hardcore than Linux Cache. Considering increasing your memory or limiting Arc Cache to something realistic for the amount of memory for your host (like 8-12GB). The default for PVE is for ZFS to use half of host memory, you can configure this here: https://pve.proxmox.com/wiki/ZFS_on...xt=As a general rule of,of memory for the ARC.

Thanks,

Tmanok

 

[frankz]

Something else to note, you are swapping 1GB in your first screenshot. Your system does not appear to have enough memory, the PVE memory bar in summary does not show arc cache, if you open HTOP you will see all your memory is wired and cached.

Arc Cache behaves a little more hardcore than Linux Cache. Considering increasing your memory or limiting Arc Cache to something realistic for the amount of memory for your host (like 8-12GB). The default for PVE is for ZFS to use half of host memory, you can configure this here: https://pve.proxmox.com/wiki/ZFS_on_Linux#:~:text=Limit ZFS Memory Usage,-ZFS uses 50&text=As a general rule of,of memory for the ARC.

Thanks,

Tmanok

Hi. And thank you for your suggestions. However, I investigated, the problem of the severe delay was due to the Zoneminder VM which overengaged the record. I changed some parameters, as far as I could I filed some disk options. Now I find that the disk is not under stress as before, even if during the backup of the same it remains a little slow.