Proxmox VE: New Server high IO delay

[SomerHimson]

My new Proxmox server has at the momenzt only three VM's running (at least he tries to), all of them doing nothing except waiting for work.


Used file system is ZFS RAID 1 with 2x 2 TB WD red on a Asus Z170-P board.

Has anyone an idea where this high IO delay could come from?

 

[dcsapak]

you have 2 slow disks with only 8gb of ram (of which maximum 4gb will be used for zfs; which is very low)
for zfs its recommended 4gb + 1Gb/Tb of storage only (vms use extra memory)

 

[SomerHimson]

Thanks for you quick answer.
So you suggest 16 GB of RAM and then it should run again?
Why isn't the RAM statistics showing that it's busted?

 

[dcsapak]

Why isn't the RAM statistics showing that it's busted?

its not 'busted' but under load (and this is indicated by the high io wait)

So you suggest 16 GB of RAM and then it should run again?

it can improve things, but depending on what io load you impose on your 2 spinning disk it may not be much

typical 7200 rpm hdds have about 75-100 iops [1] and you have 2 disk but a zfs on top (which will issue more than 1 write for each write in the guest, depending on many factors such as blocksize etc.)

so if each of your 3 running vms produce 50 iops load, you are simply at the limits of your disks

1: https://www.symantec.com/connect/articles/getting-hang-iops-v13

 

[SomerHimson]

Thanks for the explanation.
I'm checking now the link to learn more.

 

[SomerHimson]

Just an additional question:
Is it possible, that with my new machine and it's low (relatively) RAM the host boots due to to full RAM?
It just did it 10 minutes ago, while I was reading the article... :-(

 

[dcsapak]

Is it possible, that with my new machine and it's low (relatively) RAM the host boots due to to full RAM?

check the syslog/journal (journalctl) for clues

 

[SomerHimson]

I can't find anything in the logs, at least the typical ones under /var/log. All log simply end at certain time and some time later the normal starting entries were done.
It looks like the system was simply booting, like if the mains suddenly failed.

journalctl starts like this:

-- Logs begin at Wed 2018-01-10 13:10:18 CET, end at Wed 2018-01-10 13:48:46 CET. --
Jan 10 13:10:18 Sonne kernel: random: get_random_bytes called from start_kernel+0x42/0x4f3 with crng_init=0
Jan 10 13:10:18 Sonne kernel: Linux version 4.13.13-2-pve (root@nora) (gcc version 6.3.0 20170516 (Debian 6.3.0-18)) #1 SMP PVE 4.13.13-32 (Thu, 21 Dec 2017 09:02:14 +0100) ()
Jan 10 13:10:18 Sonne kernel: Command line: BOOT_IMAGE=/ROOT/pve-1@/boot/vmlinuz-4.13.13-2-pve root=ZFS=rpool/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet
Jan 10 13:10:18 Sonne kernel: KERNEL supported cpus:
Jan 10 13:10:18 Sonne kernel: Intel GenuineIntel
Jan 10 13:10:18 Sonne kernel: AMD AuthenticAMD
Jan 10 13:10:18 Sonne kernel: Centaur CentaurHauls

The only red lines are:

Jan 10 13:10:18 Sonne kernel: [Firmware Bug]: TSC_DEADLINE disabled due to Errata; please update microcode to version: 0x52 (or later)

and that now aliases.db was found (I haven't configured it up to now).

The behaviour looks to me pretty much the same as with my big machine where I have a task here open (in German):
https://forum.proxmox.com/threads/zufällige-und-unkontrollierte-neustarts.33911

 

[SomerHimson]

Okay, I went on in my analysis. Assuming the RAM causes the trouble and together with my experience with my big server, I got the idea of all the strange trouble is cause by low RAM. So after the new machine ran smoothly all night long, I started an additional VM and after less than 15 minute the system did a reboot. Nothing in the logs to find, at least where I looked.

Just now, the additional RAM for the new machine arrived, I will do some test with busting RAM, to validate whether it is my problem or not. Afterwards I will do the same tests with the additional RAM installed. I will publish me results in the other task.

 

[wimp9849]

Okay, I went on in my analysis. Assuming the RAM causes the trouble and together with my experience with my big server, I got the idea of all the strange trouble is cause by low RAM. So after the new machine ran smoothly all night long, I started an additional VM and after less than 15 minute the system did a reboot. Nothing in the logs to find, at least where I looked.

Just now, the additional RAM for the new machine arrived, I will do some test with busting RAM, to validate whether it is my problem or not. Afterwards I will do the same tests with the additional RAM installed. I will publish me results in the other task.

Hi @SomerHimson, i'm having the same problem, same amount of disk and ram. Has adding extra ram solved your problem?

PS. the reboot is because you have to set arc min and arc max, the swappines and these setting on the swap partition

zfs set primarycache=metadata rpool/swap
zfs set secondarycache=metadata rpool/swap
zfs set compression=zle rpool/swap
zfs set checksum=off rpool/swap
zfs set sync=always rpool/swap
zfs set logbias=throughput rpool/swap

 

[SomerHimson]

Thanks mate,

I got the hint for ARC limiting and defined it with max 2GB. This let to the situation that the system was running stable but quite slow.
After upgrading to 16GB of RAM, I enlarged the ARC to 4GB, what made working significantly more comfortable. But I'm afraid it's only for four maybe five VMs with low workload.
The swappiness I haven't tuned up to now, but I will try to do so. Thanks for the hint.

I'm working on my big machine where I decided to go to 64GB (coming from 32) and introducing Samsung SM863a SSDs for ZIL and L2ARC. This should then accelerate the system dramatically. I hate spending all the money, but the big machine will get ZFS running, by any means. ;-)
The small one I will most probably try to setup a Debian 9 using software RAID1 and then install Proxmox on top.
I have an old server running Debian 4 with RAID1 powering 7 VMs. This old machine is using VMware Server since 2008 with 2 GB of RAM, which is sufficient in performance. So I think for small machines, ZFS is a too mighty FS eating too much resources. All the nice management features and intelligence is maybe a bit extreme overkill to little "baby servers"... :-(

Cheers,
Christian

 

[wimp9849]

Thanks mate,

I got the hint for ARC limiting and defined it with max 2GB. This let to the situation that the system was running stable but quite slow.
After upgrading to 16GB of RAM, I enlarged the ARC to 4GB, what made working significantly more comfortable. But I'm afraid it's only for four maybe five VMs with low workload.
The swappiness I haven't tuned up to now, but I will try to do so. Thanks for the hint.

I'm working on my big machine where I decided to go to 64GB (coming from 32) and introducing Samsung SM863a SSDs for ZIL and L2ARC. This should then accelerate the system dramatically. I hate spending all the money, but the big machine will get ZFS running, by any means. ;-)
The small one I will most probably try to setup a Debian 9 using software RAID1 and then install Proxmox on top.
I have an old server running Debian 4 with RAID1 powering 7 VMs. This old machine is using VMware Server since 2008 with 2 GB of RAM, which is sufficient in performance. So I think for small machines, ZFS is a too mighty FS eating too much resources. All the nice management features and intelligence is maybe a bit extreme overkill to little "baby servers"... :-(

Cheers,
Christian

Thanks for your feedback, the only reason i'm testing zfs is because of the ability to do a vm replica in a second node as a disaster recovery but without zfs is not possible( If i understood correct)

 

[SomerHimson]

So I just added 5G SLOG and 128G L2ARC on a SM863a at the big machine. The acceleration is dramatical from the first second on. IO delay is even with 3 VM booting at the same time less than 10% most of the time even less than 2%. Perfect!
I think I give the smal machine a try with a M2-SSD and no mirror. I think for the small one this will be okay money-wise and maybe the performance is then sufficient with an ARC of 4GB.