Memory leak in 4.0.14

trm96

New Member
Aug 28, 2024
4
0
1
I upgraded from 3.4.x to 4.0.14 and have found a huge memory leak somewhere in PBS. I have my instance running as a VM. The node it's running on its the only thing on it. Over night my memory usage went from ~1GB to over 4GB. I run backups every day and also run prune, GC and verify once a day. See htop below

1757072940851.png
 
Hello trm96 and welcome to our forum! Are you using ZFS?

Since I see a lot of proxmox-backup-proxy processes, could you please post the generated file after executing top -o %MEM -b -n 1 -w512 > $(hostname)-$(date -Is)-top.txt

Also, could you please check whether you find something relevant in the journal? In other words, please check the output of journalctl --since="1 day ago" (or, when in doubt, feel free to write the output to a file and post it here).

We have had some similar reports in a different thread, but that thread is only relevant in case you cannot explain the high memory usage (that is, the memory usage is high even if no processes seem to use that much memory, even if accumulated together). In case you are using ZFS, please check this post, otherwise this other post might still be relevant for you. If you would like to try an older kernel version, you can pin the kernel either temporarily or permanently.
 
@l.leahu-vladucu

After reviewing the links you provided this has not fixed the issue. Also what I have found is if I reboot the PBS VM the memory does not decrease on the node the VM is hosted on. Only if I shut down the VM does the RAM go back to normal usage.
 
@l.leahu-vladucu

After reviewing the links you provided this has not fixed the issue. Also what I have found is if I reboot the PBS VM the memory does not decrease on the node the VM is hosted on. Only if I shut down the VM does the RAM go back to normal usage.
Did you ever figure this out or did they fix it in an update after this? I was having a similar memory leak issue after reinstalling Proxmox Backup Server with version 4. I didn’t have this issue with version 3. Started doing this immediately after migrating to a fresh version 4 VM. Nothing has changed between the VM I have using version 3 and the new VM using version 4 but I had to increase RAM from the previous 4GB to 16GB just to get teh VM to stop crashing/freezing with OOM errors and not realizing it until a month later when I go to run updates and realize the stupid PBS VM only stayed running for 1 week or less before it went OOM again. Increasing RAM to 16GB seems to have temporarily fixed the issue but this is only a work around that isn’t viable long term because it is nearly using the entire 16B for some reason when nothing has changed from it’s config or backup load between the old VM w/ version 3 and this new VM I migrated everything too with version 4.

I need to lower the RAM back down to a sensible amount, 4GB is the recommend for a small PBS server, and the most I feel comfortable using is 8GB, which is definitely overkill for the little backup load I have on it. Seems I am going to have to migrate everything back to version 3 again if they have not fixed this in updates after the one mentioned in your post. Not sure if there is any other good free alternatives to Proxmox Backup Server either but may have to start looking into that or spending the insane money on RAM and NVMe drives for my Truenas Server to do the deduplication itself instead, just so I don’t have PBS taking up 16GB to backup like 200GB give or take of used VM/CT data between 10-15 VMs/CTs.
 
Did you ever figure this out or did they fix it in an update after this? I was having a similar memory leak issue after reinstalling Proxmox Backup Server with version 4. I didn’t have this issue with version 3. Started doing this immediately after migrating to a fresh version 4 VM. Nothing has changed between the VM I have using version 3 and the new VM using version 4 but I had to increase RAM from the previous 4GB to 16GB just to get teh VM to stop crashing/freezing with OOM errors and not realizing it until a month later when I go to run updates and realize the stupid PBS VM only stayed running for 1 week or less before it went OOM again. Increasing RAM to 16GB seems to have temporarily fixed the issue but this is only a work around that isn’t viable long term because it is nearly using the entire 16B for some reason when nothing has changed from it’s config or backup load between the old VM w/ version 3 and this new VM I migrated everything too with version 4.

I need to lower the RAM back down to a sensible amount, 4GB is the recommend for a small PBS server, and the most I feel comfortable using is 8GB, which is definitely overkill for the little backup load I have on it. Seems I am going to have to migrate everything back to version 3 again if they have not fixed this in updates after the one mentioned in your post. Not sure if there is any other good free alternatives to Proxmox Backup Server either but may have to start looking into that or spending the insane money on RAM and NVMe drives for my Truenas Server to do the deduplication itself instead, just so I don’t have PBS taking up 16GB to backup like 200GB give or take of used VM/CT data between 10-15 VMs/CTs.
No, its till an issue. It has not affected performance so I have come to the realization that it uses 80 to 95 precent of RAM on the VM
What i have noticed is that when rebooted the RAM usage is normal (600MB) but when a backup is preformed on one of my VMs/LXCs then they RAM goes back to being above 5GB. This behavior is the same no matter how much RAM i have provisioned to the PBS VM