Backup failed on VM template : RTNETLINK answers: Cannot allocate memory

[guillaumev]

Hi,

This is happening on a proxmox 7.4-17, trying to backup to a PBS 3.0-2. When I'm doing a backup of a VM template, I have the following error :

INFO: starting new backup job: vzdump 325 --mode snapshot --notes-template '{{guestname}}' --storage poivron-potager-backups --node manioc --remove 0
INFO: Starting Backup of VM 325 (qemu)
INFO: Backup started at 2024-02-05 09:29:35
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: template-debian12-cemea
INFO: include disk 'virtio0' 'donnees:base-325-disk-0' 32G
INFO: creating Proxmox Backup Server archive 'vm/325/2024-02-05T08:29:35Z'
INFO: starting kvm to execute backup task
RTNETLINK answers: Cannot allocate memory
can't create interface fwln325i0 - command '/sbin/ip link add name fwln325i0 mtu 1500 type veth peer name fwpr325p0 mtu 1500' failed: exit code 2

kvm: -netdev type=tap,id=net0,ifname=tap325i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on: network script /var/lib/qemu-server/pve-bridge failed with status 512
ERROR: Backup of VM 325 failed - start failed: QEMU exited with code 1
INFO: Failed at 2024-02-05 09:29:36
INFO: Backup job finished with errors
TASK ERROR: job errors

The server has 8Gio of available RAM, which *should* be more than enough to create a network (the template itself has 2Gio RAM attributed).

 root@hyper.manioc:~# free -m
               total        used        free      shared  buff/cache   available
Mem:          192103      182257        7849         544        1997        7714

To add to that, 50% of the RAM of this server is used by ZFS ARC cache (total RAM is 187Gio, so 93Gio for ARC cache), so the actual available memory should be much more than 8 Gio.

This issue is not only happening during backup of a template, but on some docker operation (creating a network for example), which is much more annoying.

I've solved the issue by limiting ARC RAM usage to 50Gio, which freed up about 40Gio :

 root@hyper.manioc:~# free -m
               total        used        free      shared  buff/cache   available
Mem:          192103      143701       46400         544        2001       46267
Swap:              0           0           0

After this, everything worked again (docker network, and backup), but I'm wondering why it needs that much RAM, and why I need to manually limit ARC memory usage. Did I misconfigure something wrong on my end ?

Thanks,