proxmox-backup-client killed /memory allocation of 27536946 bytes failed

[Rufus Ebonhawk]

Hi, we currently test proxmox-backup-client and experience that the backup of all VM with a memory <= 2GB fail due to a memory issue. Fiddled around with --entries-max 524288 and --chunk-size 2048 with no success. I have read that the size of chunk is crucial for performance, so we would not like to reduce it anyway.

Is > 2G memory needed for proxmox-backup-client backup?

We are on Rocky 8.

Thank you!

 

[Matthias.]

What's the error you get? What's your version?

proxmox-backup-client version

Anyways, it should work with less than 2 GB

 

[Rufus Ebonhawk]

proxmox-backup-client version
client version: 2.1.2

Client for Rocky 8 is from https://github.com/sg4r/proxmox-backup-client

Starting backup: vm/NAME/2022-03-04T12:12:34Z
Client name: NAME
Starting backup protocol: Fri Mar  4 13:12:34 2022
Using default encryption key..
Encryption key fingerprint: 57:99:b6:66:94:29:42:b1
Using default master key..
No previous manifest available.
Upload directory '/' to 'backup@pbs@IP-ADDRESS:8007:vm-backup' as NAME.pxar.didx
skipping mount point: "boot"
skipping mount point: "dev"
skipping mount point: "mnt/share"
skipping mount point: "proc"
skipping mount point: "run"
skipping mount point: "srv"
skipping mount point: "sys"
skipping mount point: "var/lib/nfs/rpc_pipefs"
bin/pbs-backup.sh: line 12: 155735 Killed                  PBS_PASSWORD_FILE="NAME" /usr/local/sbin/proxmox-backup-client backup NAME.pxar:/ --backup-id NAME --repository backup@pbs@IP-ADDRESS:vm-backup --backup-type vm

This is really strange, it is using almost all CPU and needs all RAM.

 

[Rufus Ebonhawk]

dmesg -T:

[Fri Mar  4 13:24:04 2022] [ 155799]     0 155799   648872   468354  4890624        0             0 proxmox-backup-
[Fri Mar  4 13:24:04 2022] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1173401116.slice/session-2916.scope,task=proxmox-backup-,pid=155799,uid=0
[Fri Mar  4 13:24:04 2022] Out of memory: Killed process 155799 (proxmox-backup-) total-vm:2595488kB, anon-rss:1873416kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:4776kB oom_score_adj:0
[Fri Mar  4 13:24:04 2022] oom_reaper: reaped process 155799 (proxmox-backup-), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

 

[Rufus Ebonhawk]

Hi, just come from another thread which solved this one!

https://forum.proxmox.com/threads/huge-number-in-size-column.105965/

We just added --exclude /var/log/lastlog as a parameter and - you know what - now the backup runs smoothly! RAM usage remains <= 1GiB and the backup is much faster! Really nice!

Is there a way for PBS to handle this situation with sparse-files?

 

[dcsapak]

great that you found a workaround, but what probably would also fix it (without excluding the lastlog) would be this commit: https://git.proxmox.com/?p=proxmox-...ff;h=d91a0f9fc90aecabc4f359d968f716a14562ce78
which tuned the allocator parameter a bit so that the general memory usage does not get so high.. though this is only in 2.1.5 and your client is 2.1.2

Is there a way for PBS to handle this situation with sparse-files?

pbs already handles sparse files (by backing up zeroes and restoring sparse) but i think the allocator did not play well with our memory usage pattern...

try upgrading the client to 2.1.5, and see if it is better

 

[Rufus Ebonhawk]

We tried to update out EL8 client trying to adapt this script https://github.com/sg4r/proxmox-backup-client without luck. We cCanged commit ids but obviously we did not do it right. We do not have the time for now to dive deeper into it.

Do you have an idea when official client for EL8 is coming?

 

[dcsapak]

Do you have an idea when official client for EL8 is coming?

no AFAIK there is no timeline on that ...

you could try to contact the original maintainer of that script and ask if he can update it.
alternatively you could tune the allocator by using an environment variable:

MALLOC_MMAP_THRESHOLD_=131072 proxmox-backup-client ...

this is basically the same as we set internally in the newer versions.

 

[Rufus Ebonhawk]

Thanks. I already contacted the maintainer. It seems he is not active currently.

I understand that there is a need for a timeline. What I do not understand, is that a client for EL8 is not a top priority. We have only Rocky8 in production. We use Debian only for our Proxmox Server... From our perspective, the missing client for EL is against the requirements of the market ...

 

[Rufus Ebonhawk]

MALLOC_MMAP_THRESHOLD_=131072 proxmox-backup-client ...

I tried that on one machine (without --exclude /var/log/lastlog). Very long-running job, high cpu consumption, more RAM needed than usual.

Sparse file /var/log/lastlog is the problem...