My VMs are shutting down unexpectedly

[FaisalALi92]

I have a single proxmox server configured with ZFS the servers resources are 8 CPU 16 GB RAM around 1 TB disk

I am hosting two VMs one 2 CPU with 2GB RAM 100 GB disk and another VM 2 CPU 4GB RAM 50 GB disk both machines are turning off by themselves?!!

I read about a similar issue here I found this link https://pve.proxmox.com/wiki/ZFS_on_Linux#sysadmin_zfs_limit_memory_usage which I honestly did not undestand

I checked it and below are outputs I got:

root@stage-ve-01:~# arc_summary | grep zfs_arc_max
zfs_arc_max 0
root@stage-ve-01:~# arc_summary | grep zfs_arc_min
zfs_arc_min
Can someone help?
Note: I am not really bothered by performance now I just dont want to keep turning on the VMs constatntly 0

 

[apoc]

ZFS uses up to 50% memory if not configured otherwise. I guess your host runs into a out of memory condition and finally kills your VMs.
Your output shows that you have not configured the min/Max.
Review the complete output and you will see it uses up to 8GB.
With 1TB of total storage you should be fine with 2-4 Gb arc memory according to my experience. There might be some more guidance out there but the source of your issue seems obvious to me.

 

[Dunuin]

Min, max and currect ARC usage can be shown by running arc_summary and looking at these lines:

ARC size (current):                                   100.1 %    8.0 GiB
        Target size (adaptive):                       100.0 %    8.0 GiB
        Min size (hard limit):                         50.0 %    4.0 GiB

Your ARC is too small when the hit rates are low or your dnode/metadata cache sizes are running out of space:

        Metadata cache size (hard limit):              75.0 %    6.0 GiB
        Metadata cache size (current):                 21.3 %    1.3 GiB
        Dnode cache size (hard limit):                 10.0 %  614.4 MiB
        Dnode cache size (current):                    14.8 %   90.8 MiB

ARC total accesses (hits + misses):                                33.7M
        Cache hit ratio:                               78.0 %      26.3M
        Cache miss ratio:                              22.0 %       7.4M
        Actual hit ratio (MFU + MRU hits):             77.9 %      26.2M
        Data demand efficiency:                        48.9 %      11.1M
        Data prefetch efficiency:                       2.0 %       1.7M

To check if really the ARC was the problem run cat /var/log/syslog | grep oom. If it return some lines OOM killed those guests because you were running out of RAM. Then limiting the ARC size might help.

 

[FaisalALi92]

ZFS uses up to 50% memory if not configured otherwise. I guess your host runs into a out of memory condition and finally kills your VMs.
Your output shows that you have not configured the min/Max.
Review the complete output and you will see it uses up to 8GB.
With 1TB of total storage you should be fine with 2-4 Gb arc memory according to my experience. There might be some more guidance out there but the source of your issue seems obvious to me.

Hi thank you for your quick response,

I checked this command arc_summary I found the below
zfs_arc_average_blocksize 8192

How can I configure it to only use between 2-4 GB of RAM?

Thank you in advance

 

[Dunuin]

How can I configure it to only use between 2-4 GB of RAM?

# create ZFS config file if not already existing (usually its not)
touch /etc/modprobe.d/zfs.conf
# set min arc size to 2GB
echo "options zfs zfs_arc_min=$[2 * 1024*1024*1024]" >> /etc/modprobe.d/zfs.conf
# set max arc size to 4GB
echo "options zfs zfs_arc_max=$[4 * 1024*1024*1024]" >> /etc/modprobe.d/zfs.conf
# update initramfs so ZFS will use the new configs
update-initramfs -u
# do a reboot so new ARC limits will be used
reboot

 

[FaisalALi92]

# create ZFS config file if not already existing (usually its not)
touch /etc/modprobe.d/zfs.conf
# set min arc size to 2GB
echo "options zfs zfs_arc_min=$[2 * 1024*1024*1024]" >> /etc/modprobe.d/zfs.conf
# set max arc size to 4GB
echo "options zfs zfs_arc_max=$[4 * 1024*1024*1024]" >> /etc/modprobe.d/zfs.conf
# update initramfs so ZFS will use the new configs
update-initramfs -u
# do a reboot so new ARC limits will be used
reboot

Hello I ran the above commands yet my VM still turned off unexpectedly

 

[Dunuin]

Did you check the syslog for OOM messages like I explained above?

Trying the old 4.13 or newer 4.19 kernel might also be an idea.

 

[apoc]

Check that the settings are actually applied.
Use arcstat and arcsummary

Edit:
Something else: are you using with dows guests? Win 10?
If yes check the powersafe settings. My guests once did suspend themselves after some time of inactivity