Large backup folder 7200 files

[dazza76]

We have 300 vms on a few hosts keeping 12 backups. we are finding we are hitting some sort of limit with this number of backups on an nfs share
previously it used to just time out , but you could click it again an it would load , now it chews out all the ram in the host till it dies.
when i say uses all the ram I mean it uses 100+ gb of ram.

any ideas , except separating all the backups in to different backup locations or proxmox-backup server

 

[Chris]

now it chews out all the ram in the host till it dies.

Hi,
which process is consuming all the memory? How are the files stored on the NFS share? All in a single folder?

 

[dazza76]

Hello Chris ,
Yes single nfs share for backups. We have worked out that the ganesha-nfs process on the backup location needed a good kick and it seems a bit happier now can still be slow.
Using the standard backup process you dont have a choice where to put the files if you have a suggested work around happy to test.
The process with the memory leak is the pvedaemon so guessing it gets in some kind of loop any just runs till it exhausts all the memory. around 30 min when it happens.
Cheers
D

 

[Chris]

Hello Chris ,
Yes single nfs share for backups. We have worked out that the ganesha-nfs process on the backup location needed a good kick and it seems a bit happier now can still be slow.
Using the standard backup process you dont have a choice where to put the files if you have a suggested work around happy to test.
The process with the memory leak is the pvedaemon so guessing it gets in some kind of loop any just runs till it exhausts all the memory. around 30 min when it happens.
Cheers
D

Yes, I see. No straight workaround here if you do not want to separate storage locations, AFAIK.
Could you provide your pveversion -v so I will try to reproduce and see where pvedaemon goes berserk.

 

[dazza76]

Sure the cluster was is using proxmox-ve: 7.2-1 (running kernel: 5.15.64-1-pve)

here is a screen shot of the memory usage


Example of the ps aux output
root 94929 1.2 10.4 21265700 13701000 ? R 00:36 3:39 pvedaemon worker
so in 36 seconds it consumed 10.4% of the available memory 13gb ram.

For replication information.
the NFS share was working and you could "ls" the lower folders with no issues, However when doing an "ls" on the dump folder it would appear to hang while it enumirated.

 

[Chris]

Sure the cluster was is using proxmox-ve: 7.2-1 (running kernel: 5.15.64-1-pve)

here is a screen shot of the memory usage
View attachment 48024

Example of the ps aux output
root 94929 1.2 10.4 21265700 13701000 ? R 00:36 3:39 pvedaemon worker
so in 36 seconds it consumed 10.4% of the available memory 13gb ram.

For replication information.
the NFS share was working and you could "ls" the lower folders with no issues, However when doing an "ls" on the dump folder it would appear to hang while it enumirated.

I am not able to reproduce the issue you are facing, having created over 7200 "fake" backups on my NFS share. I face no issues, neither with the current PVE version, nor with the outdated PVE 7.2-1. All the backups I created include logs and metadata files, so the actual number of files is 3x the number of backups. Loading in the WebUI can take up to a few seconds, but no memory spikes or anything unusual to see in my case.

A first suggestion here would be to upgrade your system to the latest version. Also, if I understand you correctly, the NFS storage "hangs" when navigating from the command line. So please check if the "ganesha-nfs" is up for the task of handling the workload, I am not familiar with it.

Also, you could consider splitting your backup store into several individual shares (e.g. one per VM pool), in order to reduce the number of files per folder.

Also, please check the journal for errors around the time pvedaemon acts up.