ZFS arc growing without limit - pve 7.3.3, kernel 5.15.74.1

[PigLover]

Over the last few days I had some odd failures and I started monitoring memory use much more closely. I've noticed that on all of my servers that do not have explicit arc limit set the ZFS arc growing to consume 100% of ram and then I start losing processes due to OOM. I don't know exactly when this started but I recently did some housekeeping and got everything updated to current versions. I believe it probably started shortly after that. All of the affected servers are running pve 7.3.3 and kernel 5.15.74.1.

Previously, arc was automatically limited to 25% of system ram on systems without an explicit arc limit. I am running a couple of machines on the 5.19 kernel but they are small servers with limited ram and I already had explicit arc limits set.

Setting a value in /sys/module/zfs/parameters/zfs_arc_max as described here to limit the arc size DOES seem to be respected and appears to fix the problem.

 

[sterzy]

Hey,

could you post the output of the following two commands? That might give some more insight into the problem:

• cat /etc/modprobe.d/zfs.conf | grep zfs_arc
• arc_summary

Please put the output between [code][/code] tags otherwise it won't be formatted properly.

 

[PigLover]

I can do that - but as I've already updated zfs.conf to force the ARC limit you won't see the condition when it was causing problems.

I'll need to find a good time to revert one of the servers and see if I can get it to reproduce.

I really offer my post as a caution to others - explicitly setting the arc_max should be considered a best practice.

 

[Dunuin]

Previously, arc was automatically limited to 25% of system ram on systems without an explicit arc limit.

Not sure which PVE Versions you were running, but PVE 6.X and 7.X always used 50% of RAM as default for the ARC.