Disable ZFS ARC or limiting it

[Valutin]

Hello,
I would like to know if it's possible to disable ZFS ARC? or limiting it as much as possible and how much would be enough.
My config is that I created a mirror pool for Proxmox system and that's it. On the Rpool, only proxmox and the Windows 10/Winvirtio drivers ISO are residing. The other drives I have are directly passed through to the VMs, so besides rpool, no other disk are part of a ZFS pool. rpool is 30GB in total and already on SSD. I have a total of 32GB of RAM and would like to give as much as possible my 2 VMs. Is 512MB/1GB, min/max enough? (You might asked me why did I chose zfs... well, this is my second Proxmox setup)
I noticed yesterday that proxmox was using more than 4GB of RAM, this prevented me to run 2 VMs at 2x14GB configs. I was under the impression that Proxmox could use 1GB of RAM and since ZFS is only serving the host, I could just take the RAM off its available "RAM pool" (no pun intended)
Thanks

 

[Valutin]

Found an article about this, applied control on min and max Arc, applied 512MB and 1GB limits.

 

[jarekmor]

The documentation:

2.3.3 ZFS Performance Tips
ZFS works best with a lot of memory. If you intend to use ZFS make sure to have enough RAM available for it.
A good calculation is 4GB plus 1GB RAM for each TB RAW disk space.

3.8.7 Limit ZFS Memory Usage
It is good to use at most 50 percent (which is the default) of the system memory for ZFS ARC to prevent per- formance shortage of the host.

Use your preferred editor to change the configuration in /etc/modprobe.d/zfs and insert:
options zfs zfs_arc_max=8589934592
This example setting limits the usage to 8GB.

Note:
If your root file system is ZFS you must update your initramfs every time this value changes:
# update-initramfs -u

 

[leesteken]

options zfs zfs_arc_max=8589934592

Don't forget to also set eth zfs_arc_min on the same line, because sometimes the default for zfs_arc_min is larger than the specified zfs_arc_max and ZFS will use more than you specified.

 

[mbosma]

You can limit the arc live without reboot (not persistent) or persistent:

Set arc min and max persistent on reboots:

$ vim /etc/modprobe.d/zfs.conf
>>>
options zfs zfs_arc_max=34359738368
options zfs zfs_arc_min=17179869184
<<<
update-initramfs -u

Set arc min max realtime (non-persistent)

$ echo 17179869184>> /sys/module/zfs/parameters/zfs_arc_min
$ echo 34359738368>> /sys/module/zfs/parameters/zfs_arc_max

These examples are for 16G and 32G.

 

[jarekmor]

In my home case I have 32GB RAM and 3TB of the storage and my ZFS's settings are (4GB + 1GB of RAM for each 1TB of the storage):

root@pve:~# cat /etc/modprobe.d/zfs.conf options zfs zfs_arc_min=4294967296 options zfs zfs_arc_max=8589934592

I also using following command in cronetab once a 24 hours :

echo 3 > /proc/sys/vm/drop_caches

to clear the read-cache and make the RAM available again as ZFS’s ARC (disk cache) is growing in size and Proxmox ARC is not releasing this process automatically.

 

[Valutin]

The documentation:

2.3.3 ZFS Performance Tips
ZFS works best with a lot of memory. If you intend to use ZFS make sure to have enough RAM available for it.
A good calculation is 4GB plus 1GB RAM for each TB RAW disk space.

3.8.7 Limit ZFS Memory Usage
It is good to use at most 50 percent (which is the default) of the system memory for ZFS ARC to prevent per- formance shortage of the host.

Use your preferred editor to change the configuration in /etc/modprobe.d/zfs and insert:
options zfs zfs_arc_max=8589934592
This example setting limits the usage to 8GB.

Note:
If your root file system is ZFS you must update your initramfs every time this value changes:
# update-initramfs -u

Thanks, I don't plan on adding more drives in ZFS, it's only to build a 2 men workstation, so all devices are passed through.

Don't forget to also set eth zfs_arc_min on the same line, because sometimes the default for zfs_arc_min is larger than the specified zfs_arc_max and ZFS will use more than you specified.

Rebooted a few times since then and it seems correctly set, always using 1024GB, Tempted to give less as the current arcsize is just 24MB, I'll let the users use the workstation first and check the arcsize in a week to further take off from it.

In my home case I have 32GB RAM and 3TB of the storage and my ZFS's settings are (4GB + 1GB of RAM for each 1TB of the storage):

root@pve:~# cat /etc/modprobe.d/zfs.conf options zfs zfs_arc_min=4294967296 options zfs zfs_arc_max=8589934592

I also using following command in cronetab once a 24 hours :

echo 3 > /proc/sys/vm/drop_caches

to clear the read-cache and make the RAM available again as ZFS’s ARC (disk cache) is growing in size and Proxmox ARC is not releasing this process automatically.

I need to read more on that, I do have another box, server side with few VMs and I only have 64GB of RAM there.

You can limit the arc live without reboot (not persistent) or persistent:

Set arc min and max persistent on reboots:

$ vim /etc/modprobe.d/zfs.conf
>>>
options zfs zfs_arc_max=34359738368
options zfs zfs_arc_min=17179869184
<<<
update-initramfs -u

Set arc min max realtime (non-persistent)

$ echo 17179869184>> /sys/module/zfs/parameters/zfs_arc_min
$ echo 34359738368>> /sys/module/zfs/parameters/zfs_arc_max

These examples are for 16G and 32G.

Thanks for the tip for the non persistent, I'll have to dig that one in.


All in all, thanks for the community help and tips. That's great help here.

 

[Ovidiu]

Don't forget to also set eth zfs_arc_min on the same line, because sometimes the default for zfs_arc_min is larger than the specified zfs_arc_max and ZFS will use more than you specified.

What exactly do you mean by "on the same line"?

Something like this?

options zfs zfs_arc_min=4294967296 zfs_arc_max=8589934592

 

[Dunuin]

What exactly do you mean by "on the same line"?

Something like this?

options zfs zfs_arc_min=4294967296 zfs_arc_max=8589934592

Not sure if that is possible. But creating a /etc/modprobe.d/zfs.conf and adding...

options zfs zfs_arc_min=4294967296
options zfs zfs_arc_max=8589934592

...will work.