PBS HTTP request timeout issue

[megabit]

Hello,

Recently, I started getting the following error when checking backups locally using proxmox-backup-client: Error: http request timed out

root@pbs:~# proxmox-backup-client snapshot list --repository 'root@pam!token@127.0.0.1:datastore1' -ns my-ns1 | grep vm | wc -l 
Error: http request timed out
root@pbs:~# proxmox-backup-client snapshot list --repository 'root@pam!token@127.0.0.1:datastore1' -ns my-ns1 | grep vm | wc -l 
7327

The error occurs on the first run of each day. Subsequent runs complete successfully without any issues. As I understand it, the first run scans the metadata and caches it.

There are several thousand VM backups (--backup-type vm) in this namespace. The datastore itself consists of 8 HDDs configured as RAIDZ2. I cannot split the backups across multiple namespaces, as all of these backups need to be stored in a single namespace.

PBS version:

root@pbs:~# proxmox-backup-client version
client version: 4.2.2
root@pbs:~# proxmox-backup-manager version
proxmox-backup-server 4.2.7-1 running version: 4.2.2

Is there a way to increase the HTTP request timeout? Alternatively, is there another way to prevent the request from timing out?

Thank you.

 

[UdoB]

The datastore itself consists of 8 HDDs configured as RAIDZ2.

You get the IOPS of a single rotating disk. This is the worst case scenario. By far.

At the beginning everything works, and you start putting actual backups on it. Then the number of chunks increases. Massively. Then you reach a point where (for example) listing the content in Web-UI does timeout. Surprisingly (?) the second click does work. The reason is simple: for the second click the most data was already loaded into Ram.

How do I know? Been there, done that! For me it was my first try to use PBS, version 1.0. I learned that at least you need a fast metadata and PBS needs IOPS to run sufficiently. Since then I always build pools with multiple vdevs with mirrored disks plus an NVME/SSD mirror as a "Special Device". Search for it here in the forum, it has been discussed multiple times.

And to make it clear: "only rotating rust" is not recommended for a reason --> it leads to problems like yours.

For reference: https://pbs.proxmox.com/docs/installation.html#recommended-server-system-requirements

 

[UdoB]

Alternatively, is there another way to prevent the request from timing out?

1. add more Ram to increase the ARC and/or experiment with "cache=meta-data-only"
2. add some good NVMe/SSDs as a mentioned "Special Device"; note that only new data will land there - but there is a helping command for this
3. experiment with a Cache Device - which will only help a second read attempt to succeed

All of the above may help.

The solution would be to re-install everything with a better suited topology and/or devices.