[SOLVED] Hypervisor and all VMs hungs during high IO loads

[jhr]

Perhaps, but I have 32GB physical RAM and runs one VM with 16GB RAM and it seems to not enough for ZFS writes.
After reboot I saw about 16-12GB available in arcstat, but when my VM starts a database backup, its down to even -285MB.
If its not crashed it stayed on about 300-400MB available. But it mostly crashed during a database backup.
I know I could limit size of RAM for ARC, but I was affraid of two things:
1. it will crashing more often
2. performance of ZFS will go down and IO will be high

Am I wrong ?

 

[Neobin]

Adding more RAM solved my problem with crashing. No need to change HDDs.
ZFS eats so much RAM

Do not get me wrong; I am really glad to hear you solved it...at least for now. But:

With more RAM and therefore a bigger ARC, the system can now compensate the bad performance of the storage with your current workload.
If your workload does not increase in the future to a certain point, this is probably totally fine and all is good.
But, if your workload increases in the future, there will for sure be a point, where it again can not compensate it anymore and you will end up where you were before.

As I said, I am glad to hear it works for you now and I hope it will be in the future too.

 

[LnxBil]

2. performance of ZFS will go down and IO will be high

Read I/O yes.

1. it will crashing more often

can be. Maybe you should consider adding a dedicated non-ZFS-based swap device/partion for those low-memory conditions or just add more RAM.

 

[jhr]

Do not get me wrong; I am really glad to hear you solved it...at least for now. But:

With more RAM and therefore a bigger ARC, the system can now compensate the bad performance of the storage with your current workload.
If your workload does not increase in the future to a certain point, this is probably totally fine and all is good.
But, if your workload increases in the future, there will for sure be a point, where it again can not compensate it anymore and you will end up where you were before.

As I said, I am glad to hear it works for you now and I hope it will be in the future too.

I fully understand what you mean. Now my Proxmox seems stable, arcstat shows about 10-16GB available. Workload is stable too, except time when the backups are in progress. I am aware of potencialy the same crashed could happen in the future, if my workload will rise.
Unfortunately I don't have any comparision of Proxmox with ZFS R10 and Proxmox without ZFS R10, but my guess is without ZFS I wouldn't need adding a memory. It's a pity that Proxmox's installer not supporting SW-RAID (I know I can install Debian on SW-RAID and then install Proxmox over it, but I didn't tried yet). All my Proxmox's installations (7) running with ZFS.

 

[LnxBil]

Unfortunately I don't have any comparision of Proxmox with ZFS R10 and Proxmox without ZFS R10, but my guess is without ZFS I wouldn't need adding a memory.

With respect to the crash, you may be right, but performance is in both setups better if you have more buffer cache for the filesystem.
The only difference is that the buffer cache (e.g. ext4) can be easier and faster evacuated than in ZFS with ARC and that is the main problem in any OOM situations (besides slabs). Therefore you need swap to "take the edge of" the memory requirements.

 

[hvisage]

But it mostly crashed during a database backup.
I know I could limit size of RAM for ARC, but I was affraid of two things:
1. it will crashing more often


Am I wrong ?

That is the reason to force the arc smaller, perhaps even drop caches (sending 3 to vm.drop_caches) just before the expected RAM pressures.
I've seen similar with a system that had a very high number of files (LXC) that caused ARC pressure problems during backups, as the ARC was under pressure to evacuate loads of recently used while trying to cope with the mass reading.
The problem with Linux's OOM killer idea, is that ARC can't cater quick enough, as it's not a first class kernel buffer/cache citizen - Linux's NIHS. If it was, and Linux wasn't so brain dead w.r.t. OOM/overcommit, these problems would've been neglingent if ever happening.