PBS storage slow load and timeout issues

[harmonyp]

As you can see from this gif https://gyazo.com/3f5c7ff162709258e8f335aa0bc1d728 PBS sometimes takes 30+ seconds to load and even timeouts. How can I diagnose the issue? The latency between the servers is low.

It feels like once it loads other backups of other virtual machines load faster as if it's caching.

 

[harmonyp]

No one had this issue?

 

[fabian]

this can happen if the PBS takes too long to respond (probably because of high load or slow storage).

 

[harmonyp]

this can happen if the PBS takes too long to respond (probably because of high load or slow storage).

Just tested this happens when the load is quite low too.

Is there some sort of cache that gets updated on requests? I have found once one server loads the backup list they all do fast afterwards even if not in the same cluster.

 

[fabian]

not on the PVE side - but of course the page cache (or ARC if you use ZFS) on the PBS side can help keeping stuff cached in-memory..

 

[harmonyp]

not on the PVE side - but of course the page cache (or ARC if you use ZFS) on the PBS side can help keeping stuff cached in-memory..

I am not sure if we are on the same page. Every day when I load the PBS for the first time that day it loads extremely slow https://gyazo.com/3784ccd0416d5d300da6f69f0029e181 leaving it a few minutes once I come back it starts loading fast as if the layout of the PBS data gets cached (like a sitemap) for all virtual machines https://gyazo.com/af9572b0499a397fe8365e47e5304cdf

 

[fabian]

yes. the request goes like this

GUI -> PVE -> pbs-client -> PBS

first time with uncached data on the PBS side it takes long
second time with cached data on the PBS side it's fast

do you do backups at night? those probably evict the metadata for the listing, and the next time you list all backups you need to fill the cache again and it's slow.

there is no caching of this on the PVE side or in the GUI. you can check for yourself in the browser dev console - the requests for 'api2/json/nodes/$NODE/storage/$STORAGE/content?content=backup' are made everytime you open that panel.

 

[harmonyp]

Ok thanks makes sense. Backups are daily yes.

How can I update this metadata then? I would like to do it at certain time daily with a cronjob on the PBS server.

 

[fabian]

basically anything that does a snapshot listing should be sufficient to warm the cache (proxmox-backup-client snapshot list ... or pvesm list PBSSTORAGE)

 

[harmonyp]

basically anything that does a snapshot listing should be sufficient to warm the cache (proxmox-backup-client snapshot list ... or pvesm list PBSSTORAGE)

Any ideas why this cron doesn't work?

30 * * * * /bin/echo | pvesm list PBS

 

[fabian]

pvesm is in /usr/sbin, which is not in the default $PATH when executed via cron. the echo is also not needed, simply /usr/sbin/pvesm list PBS should work

 

[Harliff]

Looks like this workaround works!

# crontab -l | grep -v '^#'
@hourly /usr/sbin/pvesm list my-pbs-server > /dev/null