Rule of thumb, how much ARC for ZFS ?

[Q-wulf]

I have a 512 GB pool based on SSD's.

• Compression on
• Deduplication on
• Used only for VM OS Data

How much ARC do I realistically need with Proxmox's implementation of ZFS ?


Edit:
Background info (not relevant to the problem): The system sits on a LVM-thin SSD POOL. VM-data sits on a Raid-6 with large Cache and BBU. This has all been cobbled together from used parts; no intention to buy/upgrade.

 

[LnxBil]

Hi Q-wulf, only to understand it correctly: You're running ZFS on a RAID-6 with cache and bbu?

 

[Q-wulf]

Nope.

Storage-Pools:

1. I have a ZFS pool called SSD running on SSD's that is used for VM OS Data. <-- need advice for ARC allocation.
2. VM-Data resides on a Raid 6 HDD Pool provided by a hardware RC with Cache + BBU.
3. Proxmox sits on a SSD LVM

Edit: just want to know how much Ram i'll need to allocate to ARC (min) for this 512 GB ZFS pool if I use Compression and Dedup AND still would like good performance.

If I am not totally mistaken from ZFS on FreeBSD that would be 8 GB (base) + 0.5 GB (Pool size)+ 0.5 GB (dedup metadata). Is it the same on ZOL or is it different?

Rule of thumb will suffice.

 

[LnxBil]

Thanks for the clearification. Dedup RAM usage is normally (rule of thumb of average 64K blocks) 5 GB per 1 TB of disk storage, so in your case add 2.5 GB for the dedup stuff (to the base minimum of 2 GB according to the ZoL website). So in general, your (FreeBSD advised) amount of 9 GB should be more than enough for the minimum ARC size. What is your planned maximum size?

In my opinion, you will not "feel" a not so big minimum value on an SSD due to the good random i/o performance.

 

[Q-wulf]

well, I was trying to minimize the Ram footprint of the ZFS-Pool.


So Rule of thumb then is:
[base of 2GB] + [1 GB/TB of storage] + [5GB/TB for Dedupe]

In my Case that would be 5GB of Arc.

What is your planned maximum size?

What is your planned maximum size?

In my opinion, you will not "feel" a not so big minimum value on an SSD due to the good random i/o performance.

That is actually what I meant. Should have asked "What is the minimum ARC size i can get away with".


As such, I was thinking of doing this:

• edit /etc/modprobe.d/zfs.conf to apply several tuning options for high performance servers:

# ZFS tuning for a proxmox machine that reserves 64GB for ZFS
#
# Don't let ZFS use less than 5GB and more than 5GB
options zfs zfs_arc_min=5368709120
options zfs zfs_arc_max=5368709120

Then use the Rest of my 27 GB of Ram for VM's/Proxmox.

 

[dswartz]

Note also that this depends on the cache policy for your VMs. If you use writeback, ARC will be very little used, since the linux page cache will handle things, so in that case, you can leave it at the default. My 128GB server:

root@pve:~# free -m
total used free shared buffers cached
Mem: 128826 128403 422 93 48372 35307
-/+ buffers/cache: 44723 84102
Swap: 8191 119 8072

ARC Size: 13.03% 8.20 GiB
Target Size: (Adaptive) 13.08% 8.23 GiB
Min Size (Hard Limit): 0.05% 32.00 MiB
Max Size (High Water): 2012:1 62.90 GiB

 

[Q-wulf]

That is good to know.
But i use cache= none almost exclusively on that machine.

For anyone wondering, here is the wiki article:
https://pve.proxmox.com/wiki/Performance_Tweaks#Disk_Cache

 

[guletz]

That is good to know.
But i use cache= none almost exclusively on that machine.

For anyone wondering, here is the wiki article:
https://pve.proxmox.com/wiki/Performance_Tweaks#Disk_Cache

Because you do not want to cache the same data in zfs and in the guest.



Pay atention on dedup. If dedup db can fit entire in ram, is ok. If not, your performance will be very bad, because zfs will need to read it from disks(rotational not ssd, ssd can help for some load until reach a level) . So if you do not add new files in the future, or your existing files are mostly unchanged, it is ok to use dedup. In any other situations it is a bad ideea to use dedup. Also the combination of dedup and compression it not so wise, because many SSD use internal compression (firmware). You need to test so you can be sure what your SSD realy do in the background

How many RAM do you want to alocate to zfs depens of the storage size, but this is not the only thing that you take in accounting . I faced the situation when with the same storage size with the same zfs memory allocation, one server was good and other was bad in terms of performances. The difference was the load that this different servers must deal with it. Also I can mention, that I do not understood why for you is important the guest OS data (witch is easy to recover in case of .... so you use zfs) but VM data is not so important (no zfs). Or maybe my bad english understanding; )

 

[Rhinox]

For anyone wondering, here is the wiki article:
https://pve.proxmox.com/wiki/Performance_Tweaks#Disk_Cache

I have read that wiki-page a few times and yet I'm not sure I understand it correctly. It seems to me to be a little bit simplified, because in case of using ZFS for VM-data on PVE-host, there are at least 3 different disk-related software-caches: system-cache on host (free -m), ARC on host, and system-cache on vm-guest (free -m). Actually, VM could use ZFS too, that is one more ARC...