PBS HTTP request timeout issue

[megabit]

Hello,

Recently, I started getting the following error when checking backups locally using proxmox-backup-client: Error: http request timed out

root@pbs:~# proxmox-backup-client snapshot list --repository 'root@pam!token@127.0.0.1:datastore1' -ns my-ns1 | grep vm | wc -l 
Error: http request timed out
root@pbs:~# proxmox-backup-client snapshot list --repository 'root@pam!token@127.0.0.1:datastore1' -ns my-ns1 | grep vm | wc -l 
7327

The error occurs on the first run of each day. Subsequent runs complete successfully without any issues. As I understand it, the first run scans the metadata and caches it.

There are several thousand VM backups (--backup-type vm) in this namespace. The datastore itself consists of 8 HDDs configured as RAIDZ2. I cannot split the backups across multiple namespaces, as all of these backups need to be stored in a single namespace.

PBS version:

root@pbs:~# proxmox-backup-client version
client version: 4.2.2
root@pbs:~# proxmox-backup-manager version
proxmox-backup-server 4.2.7-1 running version: 4.2.2

Is there a way to increase the HTTP request timeout? Alternatively, is there another way to prevent the request from timing out?

Thank you.

 

[UdoB]

The datastore itself consists of 8 HDDs configured as RAIDZ2.

You get the IOPS of a single rotating disk. This is the worst case scenario. By far.

At the beginning everything works, and you start putting actual backups on it. Then the number of chunks increases. Massively. Then you reach a point where (for example) listing the content in Web-UI does timeout. Surprisingly (?) the second click does work. The reason is simple: for the second click the most data was already loaded into Ram.

How do I know? Been there, done that! For me it was my first try to use PBS, version 1.0. I learned that at least you need a fast metadata and PBS needs IOPS to run sufficiently. Since then I always build pools with multiple vdevs with mirrored disks plus an NVME/SSD mirror as a "Special Device". Search for it here in the forum, it has been discussed multiple times.

And to make it clear: "only rotating rust" is not recommended for a reason --> it leads to problems like yours.

For reference: https://pbs.proxmox.com/docs/installation.html#recommended-server-system-requirements

 

[UdoB]

Alternatively, is there another way to prevent the request from timing out?

1. add more Ram to increase the ARC and/or experiment with "cache=meta-data-only"
2. add some good NVMe/SSDs as a mentioned "Special Device"; note that only new data will land there - but there is a helping command for this
3. experiment with a Cache Device - which will only help a second read attempt to succeed

All of the above may help.

The solution would be to re-install everything with a better suited topology and/or devices.

 

[megabit]

UdoB, thanks for your reply and suggestions!

1. add more Ram to increase the ARC and/or experiment with "cache=meta-data-only"

I tried setting primarycache=metadata, but after running garbage collection, everything starts over from scratch.

2. add some good NVMe/SSDs as a mentioned "Special Device"; note that only new data will land there - but there is a helping command for this

Maybe this approach works, but I don't have the option to add SSD/NVMe drives.

3. experiment with a Cache Device - which will only help a second read attempt to succeed

Yes, unfortunately, that's the only way it works right now. One request to fetch the data into the cache, and a second one to return the result.

That's why one of the options I wanted to explore was finding a way to increase the request timeout.

 

[news]

please show the zfs dataset config.
in the root console type and exchange the placeholder:
zfs get all <zfs-pool>/<zfs-dataset>

 

[news]

please call too:

zpool status <zfs-pool>
zpool list -v <zfs-pool>

 

[megabit]

Here is the information

zfs get all <zfs-pool>/<zfs-dataset>

# zfs get all datastore1
NAME        PROPERTY                VALUE                      SOURCE
datastore1  type                    filesystem                 -
datastore1  creation                Tue Jun 30 10:26 2026      -
datastore1  used                    28.3T                      -
datastore1  available               12.9T                      -
datastore1  referenced              28.3T                      -
datastore1  compressratio           1.04x                      -
datastore1  mounted                 yes                        -
datastore1  quota                   none                       default
datastore1  reservation             none                       default
datastore1  recordsize              128K                       default
datastore1  mountpoint              /mnt/datastore/datastore1  local
datastore1  sharenfs                off                        default
datastore1  checksum                on                         default
datastore1  compression             on                         local
datastore1  atime                   on                         default
datastore1  devices                 on                         default
datastore1  exec                    on                         default
datastore1  setuid                  on                         default
datastore1  readonly                off                        default
datastore1  zoned                   off                        default
datastore1  snapdir                 hidden                     default
datastore1  aclmode                 discard                    default
datastore1  aclinherit              restricted                 default
datastore1  createtxg               1                          -
datastore1  canmount                on                         default
datastore1  xattr                   sa                         default
datastore1  copies                  1                          default
datastore1  version                 5                          -
datastore1  utf8only                off                        -
datastore1  normalization           none                       -
datastore1  casesensitivity         sensitive                  -
datastore1  vscan                   off                        default
datastore1  nbmand                  off                        default
datastore1  sharesmb                off                        default
datastore1  refquota                none                       default
datastore1  refreservation          none                       default
datastore1  guid                    9788989650938046039        -
datastore1  primarycache            all                        local
datastore1  secondarycache          all                        default
datastore1  usedbysnapshots         0B                         -
datastore1  usedbydataset           28.3T                      -
datastore1  usedbychildren          1.82G                      -
datastore1  usedbyrefreservation    0B                         -
datastore1  logbias                 latency                    default
datastore1  objsetid                54                         -
datastore1  dedup                   off                        default
datastore1  mlslabel                none                       default
datastore1  sync                    standard                   default
datastore1  dnodesize               legacy                     default
datastore1  refcompressratio        1.04x                      -
datastore1  written                 28.3T                      -
datastore1  logicalused             29.2T                      -
datastore1  logicalreferenced       29.1T                      -
datastore1  volmode                 default                    default
datastore1  filesystem_limit        none                       default
datastore1  snapshot_limit          none                       default
datastore1  filesystem_count        none                       default
datastore1  snapshot_count          none                       default
datastore1  snapdev                 hidden                     default
datastore1  acltype                 off                        default
datastore1  context                 none                       default
datastore1  fscontext               none                       default
datastore1  defcontext              none                       default
datastore1  rootcontext             none                       default
datastore1  relatime                on                         local
datastore1  redundant_metadata      all                        default
datastore1  overlay                 on                         default
datastore1  encryption              off                        default
datastore1  keylocation             none                       default
datastore1  keyformat               none                       default
datastore1  pbkdf2iters             0                          default
datastore1  special_small_blocks    0                          default
datastore1  prefetch                all                        default
datastore1  direct                  standard                   default
datastore1  longname                off                        default
datastore1  defaultuserquota        0                          -
datastore1  defaultgroupquota       0                          -
datastore1  defaultprojectquota     0                          -
datastore1  defaultuserobjquota     0                          -
datastore1  defaultgroupobjquota    0                          -
datastore1  defaultprojectobjquota  0                          -

zpool status <zfs-pool>

# zpool status datastore1
  pool: datastore1
 state: ONLINE
  scan: scrub repaired 0B in 15:31:21 with 0 errors on Sun Oct 11 15:55:21 2026
config:

    NAME                                        STATE     READ WRITE CKSUM
    datastore1                                  ONLINE       0     0     0
      raidz2-0                                  ONLINE       0     0     0
        scsi-1ATA_ST8000NM0055-1RM112_ZA1ANWFG  ONLINE       0     0     0
        scsi-1ATA_ST8000NM0055-1RM112_ZA1ANKRA  ONLINE       0     0     0
        scsi-1ATA_ST8000NM0055-1RM112_ZA1AN5VC  ONLINE       0     0     0
        scsi-1ATA_ST8000NM0055-1RM112_ZA1ANWAV  ONLINE       0     0     0
        scsi-1ATA_ST8000NM0055-1RM112_ZA1ANJLF  ONLINE       0     0     0
        scsi-1ATA_ST8000NM0055-1RM112_ZA1AMVEL  ONLINE       0     0     0
        scsi-1ATA_ST8000NM0055-1RM112_ZA1ANFMN  ONLINE       0     0     0
        scsi-1ATA_ST8000NM0055-1RM112_ZA1ANJR7  ONLINE       0     0     0

errors: No known data errors

zpool list -v <zfs-pool>

# zpool list -v datastore1
NAME                                         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
datastore1                                  58.2T  39.8T  18.4T        -         -    19%    68%  1.00x    ONLINE  -
  raidz2-0                                  58.2T  39.8T  18.4T        -         -    19%  68.4%      -    ONLINE        -
    scsi-1ATA_ST8000NM0055-1RM112_ZA1ANWFG  7.28T      -      -        -         -      -      -      -    ONLINE        -
    scsi-1ATA_ST8000NM0055-1RM112_ZA1ANKRA  7.28T      -      -        -         -      -      -      -    ONLINE        -
    scsi-1ATA_ST8000NM0055-1RM112_ZA1AN5VC  7.28T      -      -        -         -      -      -      -    ONLINE        -
    scsi-1ATA_ST8000NM0055-1RM112_ZA1ANWAV  7.28T      -      -        -         -      -      -      -    ONLINE        -
    scsi-1ATA_ST8000NM0055-1RM112_ZA1ANJLF  7.28T      -      -        -         -      -      -      -    ONLINE        -
    scsi-1ATA_ST8000NM0055-1RM112_ZA1AMVEL  7.28T      -      -        -         -      -      -      -    ONLINE        -
    scsi-1ATA_ST8000NM0055-1RM112_ZA1ANFMN  7.28T      -      -        -         -      -      -      -    ONLINE        -
    scsi-1ATA_ST8000NM0055-1RM112_ZA1ANJR7  7.28T      -      -        -         -      -      -      -    ONLINE        -

 

[news]

Thanks,
you use 68.38% of your usable space of aprox. max 80% of 39.8 TByte.
This will be 46.56 TByte of your RaidZ2 HDD Raid.

So you have left: 46.56 TByte - 39.8 TByte = 6.76 TByte

Please create and set a zfs pool quota of 20% of your space:

zfs create datastore1/quota
zfs set refreservation=11.6T datastore1/quota

These are the rules for zfs datasets with only hdds.

 

[UdoB]

Maybe this approach works, but I don't have the option to add SSD/NVMe drives.

Because there are no free slots for these devices? Or because the system is not reachable physically - e.g. in a remote datacenter?

Anyway, to implement said "Special Device" I want to mention two more options:

First: is an unused PCIe slot available? There are cards with both NVMe and SATA options. You need at least two of the same kind.

And, second: for my Homelab I could possibly imagine to prepare three USB-to-NVMe devices, ideally from three different manufacturers. And yes, I mean a three-fold, single mirror. I say this because I have been there, done that with two devices. On my systems this construct was not stable on the long run. But for my Homelab it was good enough. For an it-has-to-be-reliable commercial setting this approach is probably definitely a no-go!

My recommended capacity would be 2TB (and adjusting "special_small_blocks"), while 1TB is fine and 250GB is the absolute minimum.

Good luck!

 

[megabit]

Please create and set a zfs pool quota of 20% of your space:

I think I see where you're going with this. However, in this case, datastore space utilization does not exceed 80% during the issue. Setting a quota won't help here.

Because there are no free slots for these devices? Or because the system is not reachable physically - e.g. in a remote datacenter?

1. Yes. 2. Yes. There is no possibility to modify the server.

 

[news]

what about proxmox bs deduplication ration? You can find that value on the GUI.
did you run e.g. all 4 hours a proxmox bs pune and a garbage collection job?
this is necessary for delete old proxmox bs chunks.
The max chunks is 4 MByte!
on my system with zfs compression on the chunk size is a little over 2 MByte.

 

[xybo]

For HDD disks you should set direct=disabled