zfs_arc_max does not seem to work

shapecharge · Mar 6, 2015

Would like to see if anyone else has had this issue where zfs_arc_max does not seem to limit zfs.

currently awaiting a server I ordered to replace a desktop class machine I use for a lab. Decided to test zfs before that arrives and in my testing it does not seem like zfs_arc_max is being followed. Installed 3.4 on a test system 16gb RAM 1 hdd. install option was zfs raid 0, again lab so no need to worry much about a crash.

Restored 3 test systems from current lab backup files. set zfs_arc_max at 4gb for initial test. While restoring the machines (stored on nfs backup location) system memory ballooned to 5.5-6gb. Starting the machines (1 openvz at 512mb RAM, 1 KVM ubuntu server 2gb RAM, and 1 Win7 KVM 2gb Ram, these are the max memory available to those machines each at idle are using close to half from the gui)

memory usage on the server is now 12.5gb which is 4gb over what I would expect with simple math 4gb for zfs, 512 for openvz, and 4gb for the two KVM servers.

alatteri · Mar 7, 2015

I agree.. Something is wrong. If you do `grep c_max /proc/spl/kstat/zfs/arcstats` the value returned is 6299265024, which is half my physical RAM on install. I created /etc/modprobe.d/zfs.conf and add the line options zfs zfs_arc_max=4294967296 which is being ignored.

Nemesiz · Mar 9, 2015

# aptitude search zfs
i debian-zfs - Native ZFS filesystem metapackage for Debian.
p libzfs-dev - Native ZFS filesystem development files for Linux
p libzfs1 - Native ZFS filesystem library for Linux
p libzfs1-dbg - Debugging symbols for libzfs1
i A libzfs2 - Native OpenZFS filesystem library for Linux
v lzfs -
v lzfs-dkms -
p zfs-dbg - Debugging symbols for ZFS userland libraries and tools
i zfs-dkms - Native ZFS filesystem kernel modules for Linux
i A zfs-doc - Native OpenZFS filesystem documentation and examples.
p zfs-fuse - ZFS on FUSE
v zfs-grub -
v zfs-grub-pc -
v zfs-grub-pc-bin -
i zfs-initramfs - Native OpenZFS root filesystem capabilities for Linux
v zfs-modules -
i zfsonlinux - archive.zfsonlinux.org trust package
i zfsutils - Native OpenZFS management utilities for Linux
p zfsutils-dbg - Debugging symbols for zfsutils

# cat /etc/modprobe.d/zfs.conf
# ARC
#options zfs zfs_arc_max=4294967296

# 6 gb
#options zfs zfs_arc_max=6442450944

# 10 gb
options zfs zfs_arc_max=10737418240

# metadata limit
# 5 gb
options zfs zfs_arc_meta_limit=5368709120

# L2ARC
options zfs l2arc_noprefetch=0
options zfs l2arc_write_max=104857600
options zfs l2arc_write_boost=104857600

# grep c_max /proc/spl/kstat/zfs/arcstats
c_max 4 10737418240

alatteri · Mar 9, 2015

root@XXXXX:~# cat /etc/modprobe.d/zfs.conf
# ZFS tuning for a proxmox machine that reserves 64GB for ZFS
#
# Don't let ZFS use less than 4GB and more than 64GB
#options zfs zfs_arc_min=2147483648
#options zfs zfs_arc_max=4294967296
options zfs zfs_arc_max=2147483648

root@XXXXX:/etc/modprobe.d# grep c_max /proc/spl/kstat/zfs/arcstats
c_max 4 6299265024

Nemesiz · Mar 9, 2015

I have not tested Proxmox ZFS libs. I`m using zfsonlinux deb dkms version and I use it from proxmox 2.x version

alatteri · Mar 10, 2015

It seems that proxmox official ZFS is ignoring the options zfs zfs_arc_max=2147483648 value. I added an additional for 4 gigs of RAM and ARC swallowed that too.

root@licvault01:~# grep c_max /proc/spl/kstat/zfs/arcstats
c_max 4 8413184000

wolfgang · Mar 10, 2015

Hi,
thanks you for showing this problem.
after changing this value you must update the init ram disk.

update-initramfs -u

zfs modules will load when the zfs.conf is not mounted.
so it will ignored.
that's why you must update it.

We will update the wiki.

alatteri · Mar 10, 2015

i have done this, it made no difference.

wolfgang said:
Hi,
thanks you for showing this problem.
after changing this value you must update the init ram disk.

update-initramfs -u

zfs modules will load when the zfs.conf is not mounted.
so it will ignored.
that's why you must update it.

We will update the wiki.

wolfgang · Mar 10, 2015

you have a new PVE 3.4 installation with ZFS as root?
and did you reboot the Host?

alatteri · Mar 10, 2015

Yes, fresh 3.4 install on dual SSD ZFS Raid0. I rebooted the host.

wolfgang · Mar 10, 2015

sorry but here i can't reproduce it.
I test it with kernel 3.10-7 and 2.32-37
and it works.

alatteri · Mar 11, 2015

I went back to plain old EXT4... much easier, and everything just works.

thesubmitter · Nov 12, 2015

Having this same issue"

cat /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=8299967296

__

cat /proc/spl/kstat/zfs/arcstats |grep c_
c_min 4 33554432
c_max 4 33722855424
arc_no_grow 4 0
arc_tempreserve 4 0
arc_loaned_bytes 4 0
arc_prune 4 0
arc_meta_used 4 7662630224
arc_meta_limit 4 25292141568
arc_meta_max 4 9375715920
arc_meta_min 4 16777216
arc_need_free 4 0
arc_sys_free 4 1053835264

__

cat /sys/module/zfs/parameters/zfs_arc_max
0

___

cat /proc/spl/kmem/slab | tail -n +3 | awk '{ print $3 }' | tr "\n" "+" | sed "s/$/0/" 10:25AM
49152+32768+0+1793088+9031680+49172480+8404992+77746176+9584640+100106240+10166272+91197440+12029952+151285760+14192640+122142720+14868480+118947840+29741056+106618880+22151168+190361600+32931840+211587072+33402880+181329920+13025280+26572656640+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0

28Gb used....

_

I made the zfs.conf changed, ran update-initramfs -u & rebooted:

My proxmox machines are crashing irregularly due to what seems to be contention between the zfs memory requirements and virtualization requirements which cause some sort of contention.

I am also using SSD disks as a primary fast store with partition cut off for CACHE and ZIL for a 2 disk array of spinning disks.

No dedup

Compression on.

High IO

Nemesiz · Nov 12, 2015

Try to change it manually.

# 10GB ARC
echo 10737418240 > /sys/module/zfs/parameters/zfs_arc_max

razvanc.mobile · Nov 12, 2015

i can confirm, after upgrading to 4.0.57 setting arc max in zfs.conf is ignored, even after reboot. it worked on previous version for sure, i think it was 4.0.40.( or .44, i forget, i'd have to check to see which version is the iso i use)

setting it like Nemesiz suggested works though.

Sent from my SM-N9005

Erk · Nov 12, 2015

razvanc.mobile said:
i can confirm, after upgrading to 4.0.57 setting arc max in zfs.conf is ignored, even after reboot. it worked on previous version for sure, i think it was 4.0.40.( or .44, i forget, i'd have to check to see which version is the iso i use)

setting it like Nemesiz suggested works though.

Sent from my SM-N9005

It works fine for me on a machine I only build last week with the latest proxmox 4.0-57

Did you remember to run update-initramfs -u after editing the zfs.conf file? If not then cat /sys/module/zfs/parameters/zfs_arc_max will return 0

Nemesiz · Nov 12, 2015

/etc/rc.local helps always

thesubmitter · Nov 13, 2015

Nemesiz said:
/etc/rc.local helps always

Ugly hack, anybody know about the memory reclaim situation?

Nemesiz · Nov 13, 2015

zfs_arc_max is dynamic parameter so its not a bad way to change it from rc.local

sigxcpu · Nov 13, 2015

Code:

root@hypervisor:~# uname -a
Linux hypervisor 4.2.3-1-pve #1 SMP Wed Oct 21 14:40:52 CEST 2015 x86_64 GNU/Linux

root@hypervisor:~# grep arc_max /etc/modprobe.d/zfs.conf
options zfs zfs_resilver_delay=0 zfs_resilver_min_time_ms=5000 zfs_top_maxinflight=128 zfs_scrub_delay=0 zfs_arc_max=17179869184 zfs_arc_meta_limit=8589934592 zfs_arc_shrink_shift=8 l2arc_write_max=67108864 l2arc_write_boost=67108864 l2arc_noprefetch=0

root@hypervisor:~# cat /proc/spl/kstat/zfs/arcstats | grep c_max
c_max                           4    17179869184

zfs_arc_max does not seem to work

New Member

Renowned Member

Renowned Member

Renowned Member

Renowned Member

Renowned Member

Proxmox Retired Staff

Renowned Member

Proxmox Retired Staff

Renowned Member

Proxmox Retired Staff

Renowned Member

Active Member

Renowned Member

Member

Renowned Member

Renowned Member

Active Member

Renowned Member

Well-Known Member