PBS storage slow load and timeout issues

harmonyp

Member
Nov 26, 2020
124
3
18
44
As you can see from this gif https://gyazo.com/3f5c7ff162709258e8f335aa0bc1d728 PBS sometimes takes 30+ seconds to load and even timeouts. How can I diagnose the issue? The latency between the servers is low.

It feels like once it loads other backups of other virtual machines load faster as if it's caching.
 
Last edited:

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
7,175
1,312
164
this can happen if the PBS takes too long to respond (probably because of high load or slow storage).
 

harmonyp

Member
Nov 26, 2020
124
3
18
44
this can happen if the PBS takes too long to respond (probably because of high load or slow storage).
Just tested this happens when the load is quite low too.

Is there some sort of cache that gets updated on requests? I have found once one server loads the backup list they all do fast afterwards even if not in the same cluster.
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
7,175
1,312
164
not on the PVE side - but of course the page cache (or ARC if you use ZFS) on the PBS side can help keeping stuff cached in-memory..
 

harmonyp

Member
Nov 26, 2020
124
3
18
44
not on the PVE side - but of course the page cache (or ARC if you use ZFS) on the PBS side can help keeping stuff cached in-memory..
I am not sure if we are on the same page. Every day when I load the PBS for the first time that day it loads extremely slow https://gyazo.com/3784ccd0416d5d300da6f69f0029e181 leaving it a few minutes once I come back it starts loading fast as if the layout of the PBS data gets cached (like a sitemap) for all virtual machines https://gyazo.com/af9572b0499a397fe8365e47e5304cdf
 
Last edited:

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
7,175
1,312
164
yes. the request goes like this

GUI -> PVE -> pbs-client -> PBS

first time with uncached data on the PBS side it takes long
second time with cached data on the PBS side it's fast

do you do backups at night? those probably evict the metadata for the listing, and the next time you list all backups you need to fill the cache again and it's slow.

there is no caching of this on the PVE side or in the GUI. you can check for yourself in the browser dev console - the requests for 'api2/json/nodes/$NODE/storage/$STORAGE/content?content=backup' are made everytime you open that panel.
 

harmonyp

Member
Nov 26, 2020
124
3
18
44
Ok thanks makes sense. Backups are daily yes.

How can I update this metadata then? I would like to do it at certain time daily with a cronjob on the PBS server.
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
7,175
1,312
164
basically anything that does a snapshot listing should be sufficient to warm the cache (proxmox-backup-client snapshot list ... or pvesm list PBSSTORAGE)
 

harmonyp

Member
Nov 26, 2020
124
3
18
44
basically anything that does a snapshot listing should be sufficient to warm the cache (proxmox-backup-client snapshot list ... or pvesm list PBSSTORAGE)
Any ideas why this cron doesn't work?

Code:
30 * * * * /bin/echo | pvesm list PBS
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
7,175
1,312
164
pvesm is in /usr/sbin, which is not in the default $PATH when executed via cron. the echo is also not needed, simply /usr/sbin/pvesm list PBS should work
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!