[SOLVED] PVE 6.3-4 and ZFS 2.0 ignores zfs_arc_max

[cpzengel]

zfs_arc_max seems to be ignored in

root@pve1:~# pveversion

pve-manager/6.3-6/2184247e (running kernel: 5.4.103-1-pve)

root@pve1:~# zfs -V
zfs-2.0.3-pve2
zfs-kmod-2.0.3-pve2

As far I can see it manages itself and allways keeping 2GB left at the end

root@pve1:~# arcstat && free -h
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  size     c  avail
14:07:10  2.4K     2      0     2    0     0    0     0    0   10G   10G    27G
              total        used        free      shared  buff/cache   available
Mem:           94Gi        63Gi        30Gi        90Mi       838Mi        30Gi
Swap:            0B          0B          0B

 

[mhaluska]

Your ARC max size looks fine = 10G. Check details from arc_summary command like this:

ARC size (current):                                    97.9 %   62.6 GiB
        Target size (adaptive):                       100.0 %   64.0 GiB
        Min size (hard limit):                        100.0 %   64.0 GiB
        Max size (high water):                            1:1   64.0 GiB
        Most Frequently Used (MFU) cache size:         37.7 %   22.2 GiB
        Most Recently Used (MRU) cache size:           62.3 %   36.6 GiB
        Metadata cache size (hard limit):              75.0 %   48.0 GiB
        Metadata cache size (current):                 12.6 %    6.1 GiB
        Dnode cache size (hard limit):                 10.0 %    4.8 GiB
        Dnode cache size (current):                    22.3 %    1.1 GiB

 

[cpzengel]

Your ARC max size looks fine = 10G. Check details from arc_summary command like this:

ARC size (current):                                    97.9 %   62.6 GiB
        Target size (adaptive):                       100.0 %   64.0 GiB
        Min size (hard limit):                        100.0 %   64.0 GiB
        Max size (high water):                            1:1   64.0 GiB
        Most Frequently Used (MFU) cache size:         37.7 %   22.2 GiB
        Most Recently Used (MRU) cache size:           62.3 %   36.6 GiB
        Metadata cache size (hard limit):              75.0 %   48.0 GiB
        Metadata cache size (current):                 12.6 %    6.1 GiB
        Dnode cache size (hard limit):                 10.0 %    4.8 GiB
        Dnode cache size (current):                    22.3 %    1.1 GiB

its ignoring the max value!
arcstat always sets upper limit to : ram - 2G!

 

[mhaluska]

Since OpenZFS 2.0 arcstat looks little bit different, size is current used size and c is max ARC size defined, where avail is free/available memory on system.

# arcstat
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  size     c  avail
16:38:47     0     0      0     0    0     0    0     0    0   62G   64G   219G

# free -h
              total        used        free      shared  buff/cache   available
Mem:          503Gi       210Gi        42Gi       5.0Gi       250Gi       285Gi
Swap:            0B          0B          0B

Edit: c is ARC target size

 

[cpzengel]

so finally

/etc/modprobe.d/zfs.conf

has to look like this

options zfs zfs_arc_min=6442450944
options zfs zfs_arc_max=10737418240

important is to set the min lower than the max, only setting max did not work if the defautlts min 1/32 of ram is higher than max value

follwed by a

update-initramfs -u

to fix a running system use it that way to free caches first

echo 10737418240> /sys/module/zfs/parameters/zfs_arc_max
echo 6442450944> /sys/module/zfs/parameters/zfs_arc_min
free -h && sync && echo 3 > /proc/sys/vm/drop_caches && free -h

and fix settings like mentioned above

chriz