OOM out of memory. zfs ARC size (current) more than zfs Max size (high water)

[Dmitrius7]

Proxmox suddenly started freezing due to OOM (Out of Memory).
After diagnosis and investigation, I found that ZFS ARC is consuming all the memory. There is a limit set for Max size (high water) at 5.0 GiB, but the system ignores this limit. The server has 64GB of RAM. The swap was 16GB; I added another 30GB, but it is ignored, filling only 200-300MB before the system freezes due to OOM.

This problem started yesterday. I have spent more than 10 hours but have not found a solution. I see that after booting, the Anonymous metadata size starts to grow and occupies all the RAM, ignoring the Max size (high water) limit.

I recorded a full video where the problem is clearly visible: https://youtu.be/4eLd8Po541o?si=Ya6akZPmmuEIxSWZ&t=382

I found on github.com openzfs. A user is reporting the exact same issue as mine. Notably, the PVE versions are the same, and the post was made 17 hours ago, around the same time I encountered this problem.
https://github.com/openzfs/zfs/issues/16322

I really ask for help! The server is dead, nothing works! Thank you in advance!

Command outputs:

arc 5G max RAM
root@erpband-pve1:~# cat /sys/module/zfs/parameters/zfs_arc_min
5368709119
root@erpband-pve1:~# cat /sys/module/zfs/parameters/zfs_arc_max
5368709120

arc_summary | grep -E 'ARC size \(current\)|Target size \(adaptive\)|Min size \(hard limit\)|Max size \(high water\)|Anonymous metadata size'
ARC size (current): 1043.4 % 52.2 GiB
Target size (adaptive): 100.0 % 5.0 GiB
Min size (hard limit): 100.0 % 5.0 GiB
Max size (high water): 1:1 5.0 GiB
Anonymous metadata size: 99.4 % 50.1 GiB

# pveversion -v
proxmox-ve: 8.2.0 (running kernel: 6.8.8-2-pve)
pve-manager: 8.2.4 (running version: 8.2.4/faa83925c9641325)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.8-2
proxmox-kernel-6.8.8-2-pve-signed: 6.8.8-2
proxmox-kernel-6.8.4-3-pve-signed: 6.8.4-3
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.7
libpve-cluster-perl: 8.0.7
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.3
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.9
libpve-storage-perl: 8.2.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.7-1
proxmox-backup-file-restore: 3.2.7-1
proxmox-firewall: 0.4.2
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.7
pve-container: 5.1.12
pve-docs: 8.2.2
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.1
pve-firewall: 5.0.7
pve-firmware: 3.12-1
pve-ha-manager: 4.0.5
pve-i18n: 3.2.2
pve-qemu-kvm: 9.0.0-5
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.4-pve1

 

[Dmitrius7]

Important note! If I import the pool as read-only, everything works fine, and the RAM is not consumed.
zpool import -o readonly=on -d /dev/disk/by-partlabel/ zp-erp1-hdd

Today I reinstalled proxmox with previous version. It didn' help. The problem is still exist.

Now:
# zfs version
zfs-2.2.3-pve1
zfs-kmod-2.2.3-pve1

was before reinstall

# zfs version
zfs-2.2.4-pve1
zfs-kmod-2.2.4-pve1

 

[Dmitrius7]

I am transferring all the data and will delete the pool. After it I will use mdadm + LVM. There have been no such failures over the years.

It's a pity, I really like ZFS.

 

[Dmitrius7]

I installed FreeBSD 14 and imported this pool. It works fine!

 

[leesteken]

I read about ARC size higher than max once before on this forum but that also was not resolved or explained. It's really weird.

 

[waltar]

Doing arc_min and arc_max almost the same isn't goog idea either. Better try 2g min and 10g max.

 

[Dmitrius7]

Doing arc_min and arc_max almost the same isn't goog idea either. Better try 2g min and 10g max.

I tryed it too, it didn't help.