[SOLVED] Proxmox 6.4 & Proxmox Backup Server 2.0: unable to backup a 120GB LXC container / arc_prune 99% CPU

[albans]

Hi,

I've 16GB of RAM on my proxmox host and 1TB zfs root, with containers using approx. 70%/11GB of RAM:

              total        used        free      shared  buff/cache   available
Mem:       16366624    11155840     2862816     1282280     2347968     3578468
Swap:             0           0           0

I use PBS 2 on another dedicated server as a storage for my LXC containers backup - it works perfectly, except for 1 LXC container which is about 120GB in size.

I've read this thread: https://forum.proxmox.com/threads/b...ne-to-nearly-freeze-system.55921/#post-258075

I've ended up using the following parameters for ZFS:
zfs_arc_meta_limit_percent=90
zfs_arc_dnode_limit_percent=40
zfs_arc_max=4294967296

I made sure zfs_arc_meta_limit > arc_meta_used & zfs_arc_dnode_limit > dnode_size.

The behavior is as follow: everything running smooth for about 3hrs, and then suddenly arc_prune is using 99% of the CPU and I've to stop the backup job. Initially, prior to tuning the zfs option, it was happening after 1hr constantly...

I understand I may be short on RAM: nevertheless, I wish to understand how to calculate what RAM shall be free to enable a backup of 120GB.
Only reference to arc sizing vs storage as per wiki https://pve.proxmox.com/wiki/ZFS_on_Linux says "allocate at least 2 GiB Base + 1 GiB/TiB-Storage". Which is what I had initially: 3GB for 1TB of storage. Now I've 4GB for 1TB of storage, and I'm still facing the same issue.

Also, what are the other options to backup this LXC container without running into this problem?

1. running proxmox backup agent within the lxc container itself?
2. using vzdump with some parameters? (ionice, bwlimit, within /etc/vzdump.conf, etc...)

Replication via ZFS to another proxmox host works like a charm.... Hence I don't understand why the backup task itself is such a headache.
Would it be possible to backup the container on the basis of the replicated zfs subvolume on the other proxmox host (where I've much more available RAM)?

Thanks for your thoughts.

 

[albans]

Replying to myself: issue resolved!
I ended up with the following settings:

UNTIL REBOOT / LIVE:
echo 4669439573 >/sys/module/zfs/parameters/zfs_arc_max
echo 100 >/sys/module/zfs/parameters/zfs_arc_meta_limit_percent
echo 60 >/sys/module/zfs/parameters/zfs_arc_dnode_limit_percent

PERMANENT:
=> vi /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=4669439573
options zfs zfs_arc_meta_limit_percent=100
options zfs zfs_arc_dnode_limit_percent=60
=> update-initramfs -u
=> REBOOT

DEBUG path:
watch the following values while the backup is running:
=> grep dnode /proc/spl/kstat/zfs/arcstats; grep arc_meta /proc/spl/kstat/zfs/arcstats
despite any of the meta_limit / dnode_limit you've set, the dnode_size and arc_meta_used will increase sooner or later to higher values than the limits. So keep increasing "live" the zfs_arc_meta_limit_percent & zfs_arc_dnode_limit_percent & zfs_arc_max , while keeping an eye on your available memory. You'll end up identifying the right values avoiding arc_prune to panic and enabling the backup to complete.

I have also noted that the backup_proxmox_client itself uses a lot of ram while running, which may explain as well the differences encountered vs local backups.

Finally, I suggest an edit of the reference to arc sizing vs storage as per wiki https://pve.proxmox.com/wiki/ZFS_on_Linux says "allocate at least 2 GiB Base + 1 GiB/TiB-Storage". I don't believe you can actually run fine with an integration of Proxmox PVE and PBS with less than 4-5GB for zfs_arc_max on the PVE host, unless you have small containers to backup (not 120GB as I have in this instance).