Retrieving backup list from PBS runs into timeout

[AraToken]

Hello,

i am running into a minor inconvenience with one of our larger proxmox clusters and the related backup servers.

About the environment: i am working with a rather big PVE cluster, including more than 5 nodes with a lot of VMs on each host. Those VMs back up on currently 2 PBS-Servers regularly, resulting in one PBS handling around a hundred VMs up to 400 backups in total per backup server.

As expected, this is quite a lot but the backups are handled quite fine. That is except the little problem i want to ask about here:
The PBS seems to time out around every two or three weeks, meaning i get an error 500 message (error listing snapshots - 500 read timeout) as soon as i try to list my backup storage over the PVE. Restarting the proxmox-backup and backup-proxy services seems to fix this, as they build up quite a high load on the system over time.

So one of my first thoughts was to simply create a cronjob that restarts those services from time to time automatically but i'm kinda worried of the consequences if, one day for some reason the services fail to restart and therefore my backups not working anymore.

My question is: are there maybe additional ways to resolve this timeout-issue?
Thank you in advance and best regards

 

[fabian]

the load should not increase over time, but scale with the number and type of currently running tasks on the PBS side.. a restart is dangerous as it will stop running tasks (unless you actually mean a reload, which gracefully hands over control to the new instance while allowing the old instance to continue processing some types of tasks).

when you encounter the issue, it would be interesting to check the running tasks and used resources before taking any action.

 

[AraToken]

Hi fabian and thank you for the quick reply!
I actually meant restart because i noticed that graceful reloads did not help much last time.
I will try to clarify next time what tasks are running in detail. Right now i assume that most of it results from the verify jobs that are scheduled during daytime, every few weeks and that the amout of verifications is simply too much so that the server basically never really finishes it.
I dont think that there is a lot of finetuning-possibilitites for that except further hardware upgrades or an additional PBS?

Thank you and best regards!

 

[mac.linux.free]

Hello,

i am running into a minor inconvenience with one of our larger proxmox clusters and the related backup servers.

About the environment: i am working with a rather big PVE cluster, including more than 5 nodes with a lot of VMs on each host. Those VMs back up on currently 2 PBS-Servers regularly, resulting in one PBS handling around a hundred VMs up to 400 backups in total per backup server.

As expected, this is quite a lot but the backups are handled quite fine. That is except the little problem i want to ask about here:
The PBS seems to time out around every two or three weeks, meaning i get an error 500 message (error listing snapshots - 500 read timeout) as soon as i try to list my backup storage over the PVE. Restarting the proxmox-backup and backup-proxy services seems to fix this, as they build up quite a high load on the system over time.

So one of my first thoughts was to simply create a cronjob that restarts those services from time to time automatically but i'm kinda worried of the consequences if, one day for some reason the services fail to restart and therefore my backups not working anymore.

My question is: are there maybe additional ways to resolve this timeout-issue?
Thank you in advance and best regards

I have the same issue. After reloading the Page or changing the storage on the left, it is working mostly.

 

[fabian]

if the verification load is at fault you could tweak the verification settings:
- less frequent verifications (since each chunk will at most be verified once in a verification tasks)
- not re-verifying already verified snapshots as often

 

[Waschbüsch]

if the verification load is at fault you could tweak the verification settings:
- less frequent verifications (since each chunk will at most be verified once in a verification tasks)
- not re-verifying already verified snapshots as often

Fabian, is it possible that some settings changed? E.g. the length of the timeout? I ask because I have the same issue after upgrading the pve nodes to the latest packages a few days ago. It used to be more like 20 or so seconds before I got the timeout when trying to view the backups for a VM. Now, it seems to only take 5 seconds.
Is it possible to set the timeout somewhere?
I have no problem viewing the contents of a backup store on the pbs server and no tasks are running on it.

 

[Waschbüsch]

PS:
There is no timeout issue when using proxmox-backup-client on one of the pve nodes to list all snapshots. It takes much longer than 5 seconds, though.
But as soon as I try to use pvesm to list backups, I run into the timeout (after roughly 5 seconds).

I ended up getting the right name of the snapshot and use qmrestore manually on the command line. Currently, the UI is useless to me, unfortunately.

 

[fabian]

yes, there was a bug that shortened the timeout (fixed in git since a while - in case you want to manually patch your system). fixed packages should be available in the repos (at least pvetest) soonish!

 

[Waschbüsch]

yes, there was a bug that shortened the timeout (fixed in git since a while - in case you want to manually patch your system). fixed packages should be available in the repos (at least pvetest) soonish!

Thanks! That is good to know!

 

[b.miller]

yes, there was a bug that shortened the timeout (fixed in git since a while - in case you want to manually patch your system). fixed packages should be available in the repos (at least pvetest) soonish!

Hi @fabian - is this the correct line to retrieve and apply this patch?

curl https://git.proxmox.com/git/pve-storage.git | git apply

Also will applying this require any service restarts?

 

[fabian]

no it's not. please just update to a fixed version