High number of backup job stuck server

[petruzzo]

HI
I use a PBS server for the backup of 4 clusters made by 6 nodes (don't ask for an even number), All worked fine before I added the 4th cluster, but when I did the server became very slow in the backups and the response on the Web interface is very slow. Yesterday night all the PVE that performed backup on the PBS had an issue with pvestatd until I restarted the PBS.
The storage is made by a Raidz2 on HDD with cache and log on ssd.
Is possible that we reach the maximum number of simultaneous backups on the server?

Spoiler: Pveversion

proxmox-backup: 3.2.0 (running kernel: 6.8.8-3-pve)
proxmox-backup-server: 3.2.7-1 (running version: 3.2.7)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.8-3
proxmox-kernel-6.8.8-3-pve-signed: 6.8.8-3
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
proxmox-kernel-6.5.13-5-pve-signed: 6.5.13-5
proxmox-kernel-6.5: 6.5.13-5
proxmox-kernel-6.5.13-3-pve-signed: 6.5.13-3
proxmox-kernel-6.5.13-1-pve-signed: 6.5.13-1
proxmox-kernel-6.5.11-6-pve-signed: 6.5.11-6
ifupdown2: 3.2.0-1+pmx9
libjs-extjs: 7.0.0-4
proxmox-backup-docs: 3.2.7-1
proxmox-backup-client: 3.2.7-1
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.3
pve-xtermjs: 5.3.0-3
smartmontools: 7.3-pve1
zfsutils-linux: 2.2.4-pve1

 

[Chris]

HI
I use a PBS server for the backup of 4 clusters made by 6 nodes (don't ask for an even number), All worked fine before I added the 4th cluster, but when I did the server became very slow in the backups and the response on the Web interface is very slow. Yesterday night all the PVE that performed backup on the PBS had an issue with pvestatd until I restarted the PBS.
The storage is made by a Raidz2 on HDD with cache and log on ssd.
Is possible that we reach the maximum number of simultaneous backups on the server?

Spoiler: Pveversion

proxmox-backup: 3.2.0 (running kernel: 6.8.8-3-pve)
proxmox-backup-server: 3.2.7-1 (running version: 3.2.7)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.8-3
proxmox-kernel-6.8.8-3-pve-signed: 6.8.8-3
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
proxmox-kernel-6.5.13-5-pve-signed: 6.5.13-5
proxmox-kernel-6.5: 6.5.13-5
proxmox-kernel-6.5.13-3-pve-signed: 6.5.13-3
proxmox-kernel-6.5.13-1-pve-signed: 6.5.13-1
proxmox-kernel-6.5.11-6-pve-signed: 6.5.11-6
ifupdown2: 3.2.0-1+pmx9
libjs-extjs: 7.0.0-4
proxmox-backup-docs: 3.2.7-1
proxmox-backup-client: 3.2.7-1
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.3
pve-xtermjs: 5.3.0-3
smartmontools: 7.3-pve1
zfsutils-linux: 2.2.4-pve1

Hi,
please monitor your PBS hardware usage (CPU, memory, IO). That should tell you a bit more about where a possible bottle neck is. Do all the backups for all the clusters run at the same time? Maybe you should spread the schedule for the backup jobs to be run at different times, so not to overload the server?

 

[petruzzo]

Hi Chris
the jobs are set one for a single VM because we have different retention policies for VMs. The CPU is a 32-core Xeon and the load average in the worst case was under 80%.
In the WE I move the backup, of the last added cluster, to another PBS, only the backup that has the schedule on the time when the problem appears and we don't have any other issue.
Today I moved back the backup job on the original PBS, after making a sync of the backup data, To understand if the problem is gone or not.