High IO delay

[NeODarK]

Hi, I have some issues with high IO delay. I have a mSATA 60Gb drive to store the base system

I tried with two 2,5" barracuda compute 1Tb hard disk in mirror with ZFS (for VM, CT, backups, templates and isos) the last week and now (the pic) is only one barracuda compute to store VM and CT, isos and templates (in ext4 format) and a second hard disc 250Gb to store backups (ext4 too).

Why I have this high IO delays?

I tried iotop and no more than 500 K/s of write data, usual 35-50 K/s

This is a write speed of the 1Tb hard disk (where VM and CT are stored)

root@proxmox:/dev/Disco1Tb# dd count=1000 bs=1M if=/dev/urandom of=/dev/Disco1Tb/random.out
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 3.42414 s, 306 MB/s

And a read speed of the 1Tb hard disk

root@proxmox:/dev/Disco1Tb# hdparm -t /dev/sda

/dev/sda:
 Timing buffered disk reads: 218 MB in  3.24 seconds =  67.19 MB/sec

I have the system up to date 6.4.6 pve-manager/6.4-6/be2fa32c (running kernel: 5.4.114-1-pve)

proxmox-ve: 6.4-1 (running kernel: 5.4.114-1-pve)
pve-manager: 6.4-6 (running version: 6.4-6/be2fa32c)
pve-kernel-5.4: 6.4-2
pve-kernel-helper: 6.4-2
pve-kernel-5.4.114-1-pve: 5.4.114-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.4-1
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-3
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-2
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.6-2
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-5
pve-cluster: 6.4-1
pve-container: 3.3-5
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-3
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1

 

[aaron]

HDDs are quite bad when it comes to how many IOPS they can handle. Depending on how large those IO operations are, you might end up with a very low bandwidth usage.

IOPS is usually the problem once you have multiple VMs using the same disk as backing storage. This is because most HDDs are able to handle about 120 IOPS and not more.

Which HDD model exactly do you use?

Using mode HDDs in a RAID10 like setup (zpool with multiple mirror vdevs) will improve the situation to a certain degree as write and read operations can be spread out over multiple disks. But in the end, SSDs are much better suited for multiple very different IO patterns as the access time is basically zero, compared to HDDs where the platter and RW-head need to line up first.

 

[mmidgett]

Here are some of my thoughts.

If you must use hard drives ensure that they are SAS not SATA and have 10K+ RPM's. 15K's are better. The faster you can spin the disk the more io it can handle. It still will not be much as HDDs are just slow. You would need to throw more HDD's to get better performance as your reading off a single disk and writes are delayed while reading. Bottom line is you need more HDD's if your going to use them.

You should also turn on the advanced options for the VM HD's and set the IO limits on them so you don't have one VM take all the resources.

 

[NeODarK]

HDDs are quite bad when it comes to how many IOPS they can handle. Depending on how large those IO operations are, you might end up with a very low bandwidth usage.

IOPS is usually the problem once you have multiple VMs using the same disk as backing storage. This is because most HDDs are able to handle about 120 IOPS and not more.

Which HDD model exactly do you use?

Using mode HDDs in a RAID10 like setup (zpool with multiple mirror vdevs) will improve the situation to a certain degree as write and read operations can be spread out over multiple disks. But in the end, SSDs are much better suited for multiple very different IO patterns as the access time is basically zero, compared to HDDs where the platter and RW-head need to line up first.

Hi, yes!! I can confirm... the CTs and VMs that cause this delay are two CTs with Cacti (network monitoring) and one VM with Mikrotik SNMP monitoring... before I move from hard disk to mSATA SSD the delay goes to minimum....

Thanks!!