zfs_arc_max does not seem to work

shapecharge

New Member
Mar 6, 2015
2
0
1
Would like to see if anyone else has had this issue where zfs_arc_max does not seem to limit zfs.

currently awaiting a server I ordered to replace a desktop class machine I use for a lab. Decided to test zfs before that arrives and in my testing it does not seem like zfs_arc_max is being followed. Installed 3.4 on a test system 16gb RAM 1 hdd. install option was zfs raid 0, again lab so no need to worry much about a crash.

Restored 3 test systems from current lab backup files. set zfs_arc_max at 4gb for initial test. While restoring the machines (stored on nfs backup location) system memory ballooned to 5.5-6gb. Starting the machines (1 openvz at 512mb RAM, 1 KVM ubuntu server 2gb RAM, and 1 Win7 KVM 2gb Ram, these are the max memory available to those machines each at idle are using close to half from the gui)

memory usage on the server is now 12.5gb which is 4gb over what I would expect with simple math 4gb for zfs, 512 for openvz, and 4gb for the two KVM servers.
 
I agree.. Something is wrong. If you do `grep c_max /proc/spl/kstat/zfs/arcstats` the value returned is 6299265024, which is half my physical RAM on install. I created /etc/modprobe.d/zfs.conf and add the line options zfs zfs_arc_max=4294967296 which is being ignored.
 
# aptitude search zfs
i debian-zfs - Native ZFS filesystem metapackage for Debian.
p libzfs-dev - Native ZFS filesystem development files for Linux
p libzfs1 - Native ZFS filesystem library for Linux
p libzfs1-dbg - Debugging symbols for libzfs1
i A libzfs2 - Native OpenZFS filesystem library for Linux
v lzfs -
v lzfs-dkms -
p zfs-dbg - Debugging symbols for ZFS userland libraries and tools
i zfs-dkms - Native ZFS filesystem kernel modules for Linux
i A zfs-doc - Native OpenZFS filesystem documentation and examples.
p zfs-fuse - ZFS on FUSE
v zfs-grub -
v zfs-grub-pc -
v zfs-grub-pc-bin -
i zfs-initramfs - Native OpenZFS root filesystem capabilities for Linux
v zfs-modules -
i zfsonlinux - archive.zfsonlinux.org trust package
i zfsutils - Native OpenZFS management utilities for Linux
p zfsutils-dbg - Debugging symbols for zfsutils

# cat /etc/modprobe.d/zfs.conf
# ARC
#options zfs zfs_arc_max=4294967296

# 6 gb
#options zfs zfs_arc_max=6442450944

# 10 gb
options zfs zfs_arc_max=10737418240

# metadata limit
# 5 gb
options zfs zfs_arc_meta_limit=5368709120


# L2ARC
options zfs l2arc_noprefetch=0
options zfs l2arc_write_max=104857600
options zfs l2arc_write_boost=104857600

# grep c_max /proc/spl/kstat/zfs/arcstats
c_max 4 10737418240
 
root@XXXXX:~# cat /etc/modprobe.d/zfs.conf​
# ZFS tuning for a proxmox machine that reserves 64GB for ZFS
#
# Don't let ZFS use less than 4GB and more than 64GB
#options zfs zfs_arc_min=2147483648
#options zfs zfs_arc_max=4294967296
options zfs zfs_arc_max=2147483648




root@XXXXX:/etc/modprobe.d# grep c_max /proc/spl/kstat/zfs/arcstats
c_max 4 6299265024
 
Last edited:
I have not tested Proxmox ZFS libs. I`m using zfsonlinux deb dkms version and I use it from proxmox 2.x version
 
It seems that proxmox official ZFS is ignoring the options zfs zfs_arc_max=2147483648 value. I added an additional for 4 gigs of RAM and ARC swallowed that too.

root@licvault01:~# grep c_max /proc/spl/kstat/zfs/arcstats
c_max 4 8413184000
 
Hi,
thanks you for showing this problem.
after changing this value you must update the init ram disk.

update-initramfs -u

zfs modules will load when the zfs.conf is not mounted.
so it will ignored.
that's why you must update it.

We will update the wiki.
 
you have a new PVE 3.4 installation with ZFS as root?
and did you reboot the Host?
 
Last edited:
sorry but here i can't reproduce it.
I test it with kernel 3.10-7 and 2.32-37
and it works.
 
Having this same issue"

cat /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=8299967296

__

cat /proc/spl/kstat/zfs/arcstats |grep c_
c_min 4 33554432
c_max 4 33722855424
arc_no_grow 4 0
arc_tempreserve 4 0
arc_loaned_bytes 4 0
arc_prune 4 0
arc_meta_used 4 7662630224
arc_meta_limit 4 25292141568
arc_meta_max 4 9375715920
arc_meta_min 4 16777216
arc_need_free 4 0
arc_sys_free 4 1053835264

__

cat /sys/module/zfs/parameters/zfs_arc_max
0

___

cat /proc/spl/kmem/slab | tail -n +3 | awk '{ print $3 }' | tr "\n" "+" | sed "s/$/0/" 10:25AM
49152+32768+0+1793088+9031680+49172480+8404992+77746176+9584640+100106240+10166272+91197440+12029952+151285760+14192640+122142720+14868480+118947840+29741056+106618880+22151168+190361600+32931840+211587072+33402880+181329920+13025280+26572656640+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0

28Gb used....

_

I made the zfs.conf changed, ran update-initramfs -u & rebooted:


My proxmox machines are crashing irregularly due to what seems to be contention between the zfs memory requirements and virtualization requirements which cause some sort of contention.

I am also using SSD disks as a primary fast store with partition cut off for CACHE and ZIL for a 2 disk array of spinning disks.

No dedup

Compression on.

High IO
 
Last edited:
i can confirm, after upgrading to 4.0.57 setting arc max in zfs.conf is ignored, even after reboot. it worked on previous version for sure, i think it was 4.0.40.( or .44, i forget, i'd have to check to see which version is the iso i use)

setting it like Nemesiz suggested works though.

Sent from my SM-N9005
 
Last edited:
i can confirm, after upgrading to 4.0.57 setting arc max in zfs.conf is ignored, even after reboot. it worked on previous version for sure, i think it was 4.0.40.( or .44, i forget, i'd have to check to see which version is the iso i use)

setting it like Nemesiz suggested works though.

Sent from my SM-N9005

It works fine for me on a machine I only build last week with the latest proxmox 4.0-57

Did you remember to run update-initramfs -u after editing the zfs.conf file? If not then cat /sys/module/zfs/parameters/zfs_arc_max will return 0
 
Code:
root@hypervisor:~# uname -a
Linux hypervisor 4.2.3-1-pve #1 SMP Wed Oct 21 14:40:52 CEST 2015 x86_64 GNU/Linux

root@hypervisor:~# grep arc_max /etc/modprobe.d/zfs.conf
options zfs zfs_resilver_delay=0 zfs_resilver_min_time_ms=5000 zfs_top_maxinflight=128 zfs_scrub_delay=0 zfs_arc_max=17179869184 zfs_arc_meta_limit=8589934592 zfs_arc_shrink_shift=8 l2arc_write_max=67108864 l2arc_write_boost=67108864 l2arc_noprefetch=0

root@hypervisor:~# cat /proc/spl/kstat/zfs/arcstats | grep c_max
c_max                           4    17179869184
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!