Retrieving backup list from PBS runs into timeout

AraToken

Member
Jul 8, 2021
41
5
13
28
Hello,

i am running into a minor inconvenience with one of our larger proxmox clusters and the related backup servers.

About the environment: i am working with a rather big PVE cluster, including more than 5 nodes with a lot of VMs on each host. Those VMs back up on currently 2 PBS-Servers regularly, resulting in one PBS handling around a hundred VMs up to 400 backups in total per backup server.

As expected, this is quite a lot but the backups are handled quite fine. That is except the little problem i want to ask about here:
The PBS seems to time out around every two or three weeks, meaning i get an error 500 message (error listing snapshots - 500 read timeout) as soon as i try to list my backup storage over the PVE. Restarting the proxmox-backup and backup-proxy services seems to fix this, as they build up quite a high load on the system over time.

So one of my first thoughts was to simply create a cronjob that restarts those services from time to time automatically but i'm kinda worried of the consequences if, one day for some reason the services fail to restart and therefore my backups not working anymore.

My question is: are there maybe additional ways to resolve this timeout-issue?
Thank you in advance and best regards
 
the load should not increase over time, but scale with the number and type of currently running tasks on the PBS side.. a restart is dangerous as it will stop running tasks (unless you actually mean a reload, which gracefully hands over control to the new instance while allowing the old instance to continue processing some types of tasks).

when you encounter the issue, it would be interesting to check the running tasks and used resources before taking any action.
 
Hi fabian and thank you for the quick reply!
I actually meant restart because i noticed that graceful reloads did not help much last time.
I will try to clarify next time what tasks are running in detail. Right now i assume that most of it results from the verify jobs that are scheduled during daytime, every few weeks and that the amout of verifications is simply too much so that the server basically never really finishes it.
I dont think that there is a lot of finetuning-possibilitites for that except further hardware upgrades or an additional PBS?

Thank you and best regards!
 
Hello,

i am running into a minor inconvenience with one of our larger proxmox clusters and the related backup servers.

About the environment: i am working with a rather big PVE cluster, including more than 5 nodes with a lot of VMs on each host. Those VMs back up on currently 2 PBS-Servers regularly, resulting in one PBS handling around a hundred VMs up to 400 backups in total per backup server.

As expected, this is quite a lot but the backups are handled quite fine. That is except the little problem i want to ask about here:
The PBS seems to time out around every two or three weeks, meaning i get an error 500 message (error listing snapshots - 500 read timeout) as soon as i try to list my backup storage over the PVE. Restarting the proxmox-backup and backup-proxy services seems to fix this, as they build up quite a high load on the system over time.

So one of my first thoughts was to simply create a cronjob that restarts those services from time to time automatically but i'm kinda worried of the consequences if, one day for some reason the services fail to restart and therefore my backups not working anymore.

My question is: are there maybe additional ways to resolve this timeout-issue?
Thank you in advance and best regards
I have the same issue. After reloading the Page or changing the storage on the left, it is working mostly.
 
if the verification load is at fault you could tweak the verification settings:
- less frequent verifications (since each chunk will at most be verified once in a verification tasks)
- not re-verifying already verified snapshots as often
 
if the verification load is at fault you could tweak the verification settings:
- less frequent verifications (since each chunk will at most be verified once in a verification tasks)
- not re-verifying already verified snapshots as often
Fabian, is it possible that some settings changed? E.g. the length of the timeout? I ask because I have the same issue after upgrading the pve nodes to the latest packages a few days ago. It used to be more like 20 or so seconds before I got the timeout when trying to view the backups for a VM. Now, it seems to only take 5 seconds.
Is it possible to set the timeout somewhere?
I have no problem viewing the contents of a backup store on the pbs server and no tasks are running on it.
 
PS:
There is no timeout issue when using proxmox-backup-client on one of the pve nodes to list all snapshots. It takes much longer than 5 seconds, though.
But as soon as I try to use pvesm to list backups, I run into the timeout (after roughly 5 seconds).

I ended up getting the right name of the snapshot and use qmrestore manually on the command line. Currently, the UI is useless to me, unfortunately.
 
yes, there was a bug that shortened the timeout (fixed in git since a while - in case you want to manually patch your system). fixed packages should be available in the repos (at least pvetest) soonish!
 
yes, there was a bug that shortened the timeout (fixed in git since a while - in case you want to manually patch your system). fixed packages should be available in the repos (at least pvetest) soonish!

Hi @fabian - is this the correct line to retrieve and apply this patch?

curl https://git.proxmox.com/git/pve-storage.git | git apply

Also will applying this require any service restarts?
 
no it's not. please just update to a fixed version ;)
 
  • Like
Reactions: b.miller

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!