Lower ZFS ARC max size

[CS-]

Hello,

I have PVE hosts with 125,6 GiB RAM. We are using ZFS to run our VMs. The workload running on these hosts are pfSense FW's.
The maximum ZFS is 10% (12,6 GiB). Is it safe to lower this maximum size?

What maximum size do you recommend? 4GiB?


arc_summary | grep -E 'Current size|Target size'
Target size (adaptive): 100.0 % 12.6 GiB
Current size: 100.0 % 12.6 GiB

arc_summary -s arc
------------------------------------------------------------------------
ZFS Subsystem Report Thu Sep 10 15:57:54 2026
Linux 7.0.14-12-pve 2.4.3-pve1
Machine: pve101 (x86_64) 2.4.3-pve1

ARC status:
Total memory size: 125.6 GiB
Min target size: 3.1 % 3.9 GiB
Max target size: 10.0 % 12.6 GiB
Target size (adaptive): 100.1 % 12.6 GiB
Current size: 100.1 % 12.6 GiB
Free memory size: 25.2 GiB
Available memory size: 21.0 GiB

ARC structural breakdown (current size): 12.6 GiB
Compressed size: 92.1 % 11.6 GiB
Overhead size: 4.5 % 585.4 MiB
Bonus size: 0.2 % 31.9 MiB
Dnode size: 0.8 % 102.1 MiB
Dbuf size: 0.4 % 45.8 MiB
Header size: 1.8 % 237.5 MiB
L2 header size: 0.0 % 0 Bytes
ABD chunk waste size: 0.1 % 14.9 MiB

ARC types breakdown (compressed + overhead): 12.1 GiB
Data size: 89.0 % 10.8 GiB
Metadata size: 11.0 % 1.3 GiB

ARC states breakdown (compressed + overhead): 12.1 GiB
Anonymous data size: 0.1 % 10.7 MiB
Anonymous metadata size: < 0.1 % 514.0 KiB
MFU data target: 31.5 % 3.8 GiB
MFU data size: 4.6 % 573.1 MiB
MFU evictable data size: 4.6 % 568.7 MiB
MFU ghost data size: 0 Bytes
MFU metadata target: 12.2 % 1.5 GiB
MFU metadata size: 1.6 % 194.5 MiB
MFU evictable metadata size: 1.3 % 156.5 MiB
MFU ghost metadata size: 0 Bytes
MRU data target: 44.1 % 5.4 GiB
MRU data size: 83.5 % 10.1 GiB
MRU evictable data size: 80.0 % 9.7 GiB
MRU ghost data size: 970.6 MiB
MRU metadata target: 12.2 % 1.5 GiB
MRU metadata size: 9.4 % 1.1 GiB
MRU evictable metadata size: 7.4 % 925.4 MiB
MRU ghost metadata size: 0 Bytes
Uncached data size: 0.8 % 103.2 MiB
Uncached metadata size: 0.0 % 0 Bytes

ARC hash breakdown:
Elements: 1.0M
Collisions: 3.3M
Chain max: 5
Chains: 30.7k

ARC misc:
Uncompressed size: 168.0 % 19.4 GiB
Memory throttles: 0
Memory direct reclaims: 0
Memory indirect reclaims: 0
Deleted: 25.7M
Mutex misses: 2.7k
Eviction skips: 577
Eviction skips due to L2 writes: 0
L2 cached evictions: 0 Bytes
L2 eligible evictions: 396.6 GiB
L2 eligible MFU evictions: < 0.1 % 64.0 KiB
L2 eligible MRU evictions: 100.0 % 396.6 GiB
L2 ineligible evictions: 142.5 MiB


Thanks.

 

[alma21]

Hi,

https://pve.proxmox.com/wiki/ZFS_on_Linux#sysadmin_zfs_limit_memory_usage

>>>> As a general rule of thumb, allocate at least 2 GiB Base + 1GiB/TiB-Storage

 

[xybo]

I agree that the 2 GiB + 1 GiB/TiB rule of thumb is probably too simplistic for this use case.


ARC size should be based more on the actual workload and memory pressure than on the total ZFS pool size. For example, I have a ZFS storage with around 15 TB and ~30 VMs, while ARC is limited to 16 GB. The ARC is usually close to that limit, but when I shut down the main PostgreSQL database, ARC usage drops to around 3.4 GB.


So in my opinion, the amount of storage alone isn't a good reason to allocate a large ARC. The working set and workload are much more relevant.


For a Proxmox host with 125 GB RAM running mainly pfSense VMs, I'd probably start with an ARC limit of 6–8 GB and monitor ARC hit rate and overall memory pressure. If there is no measurable performance difference, it could likely be reduced further, e.g. to 4 GB.

 

[UdoB]

The maximum ZFS is 10% (12,6 GiB). Is it safe to lower this maximum size?

Well, it already is lowered. The "classic" limit was 50 % of the system Ram for many years; the compile time limit is 90 %, afaik ;-)

What maximum size do you recommend?

@xybo gave the correct hint: check the actual hitrate while running your specific workload.