[SOLVED] S3 backup hangs proxmox-backup-proxy

E

et1000

Member
Jan 11, 2024
16
1
8
France
Hello,

I'm trying to back up some VMs and CT to a S3 bucket.
PBS is a VM running on PVE. My cache is a dedicated ext4 disk, mounted on /var/cache/pbs-s3/, which is hosted on a ZFS datastore.

Regularly, the backup job hangs in a middle of a backup, and proxmox-backup-proxy stops responding, as does the web UI. I can still ping the VM, open a shell, etc.; it's not overloaded. The backup job gets stuck without any error log, just INFO: 11% (82.2 GiB of 747.0 GiB) in 7m 14s, read: 80.5 MiB/s, write: 77.6 MiB/s

The command service proxmox-backup-proxy shows this:
Code: 
root@pbs01-lab:~# service proxmox-backup-proxy status
● proxmox-backup-proxy.service - Proxmox Backup API Proxy Server
     Loaded: loaded (/usr/lib/systemd/system/proxmox-backup-proxy.service; enabled; preset: enabled)
     Active: active (running) since Tue 2026-08-04 17:06:46 CEST; 23min ago
 Invocation: c294629ed5014ee89315d5f6286d26f2
   Main PID: 3767 (proxmox-backup-)
      Tasks: 43 (limit: 1415)
     Memory: 1G (peak: 1.2G)
        CPU: 19min 26.042s
     CGroup: /system.slice/proxmox-backup-proxy.service
             └─3767 /usr/lib/x86_64-linux-gnu/proxmox-backup/proxmox-backup-proxy

Aug 04 17:18:14 pbs01-lab proxmox-backup-proxy[3767]: error during snapshot file listing: 'unable to load blob '"/var/cache/pbs-s3/vm/109/2026-08-04T15:08:24Z/index.json.blob"' - No such file or directory (os error 2)'
Aug 04 17:18:14 pbs01-lab proxmox-backup-proxy[3767]: error during snapshot file listing: 'unable to load blob '"/var/cache/pbs-s3/vm/109/2026-08-04T14:45:21Z/index.json.blob"' - No such file or directory (os error 2)'
Aug 04 17:20:51 pbs01-lab proxmox-backup-proxy[3767]: error during snapshot file listing: 'unable to load blob '"/var/cache/pbs-s3/vm/101/2026-08-04T11:50:19Z/index.json.blob"' - No such file or directory (os error 2)'
Aug 04 17:20:51 pbs01-lab proxmox-backup-proxy[3767]: error during snapshot file listing: 'unable to load blob '"/var/cache/pbs-s3/vm/109/2026-08-04T14:28:59Z/index.json.blob"' - No such file or directory (os error 2)'
Aug 04 17:20:51 pbs01-lab proxmox-backup-proxy[3767]: error during snapshot file listing: 'unable to load blob '"/var/cache/pbs-s3/vm/109/2026-08-04T15:08:24Z/index.json.blob"' - No such file or directory (os error 2)'
Aug 04 17:20:51 pbs01-lab proxmox-backup-proxy[3767]: error during snapshot file listing: 'unable to load blob '"/var/cache/pbs-s3/vm/109/2026-08-04T14:45:21Z/index.json.blob"' - No such file or directory (os error 2)'
Aug 04 17:27:05 pbs01-lab proxmox-backup-proxy[3767]: error during snapshot file listing: 'unable to load blob '"/var/cache/pbs-s3/vm/101/2026-08-04T11:50:19Z/index.json.blob"' - No such file or directory (os error 2)'
Aug 04 17:27:05 pbs01-lab proxmox-backup-proxy[3767]: error during snapshot file listing: 'unable to load blob '"/var/cache/pbs-s3/vm/109/2026-08-04T14:28:59Z/index.json.blob"' - No such file or directory (os error 2)'
Aug 04 17:27:05 pbs01-lab proxmox-backup-proxy[3767]: error during snapshot file listing: 'unable to load blob '"/var/cache/pbs-s3/vm/109/2026-08-04T15:08:24Z/index.json.blob"' - No such file or directory (os error 2)'
Aug 04 17:27:05 pbs01-lab proxmox-backup-proxy[3767]: error during snapshot file listing: 'unable to load blob '"/var/cache/pbs-s3/vm/109/2026-08-04T14:45:21Z/index.json.blob"' - No such file or directory (os error 2)'

I've tried adding barried=0 to /etc/fstab for my cache disk and writeback (unsafe) for the cache management of that disk. I also tried limiting the bandwidth at 150 MiB/s.

When I restart proxmox-backup-proxy, the GUI immediatly comes back, the current VM job is dropped and the next VM is processed normally.

What is happening?
Emile
 
Last edited:
Please post the output of proxmox-backup-manager version --verbose and make sure you are running an up to date version. There have been some bugfixes with respect to hanging proxy [0] packaged in proxmox-backup-server version 4.2.2-1.

Also, the errors you are seeing are most likely due to unfinished backup snapshots which did not get cleaned up since you restart the proxy? There was a patch to silence these benign errors [1], which makes me think you might be running an outdated version without the bugfix [0].

[0] https://git.proxmox.com/?p=proxmox-backup.git;a=commit;h=23400016322c7a6981f111558e8d22666e32ee8c
[1] https://git.proxmox.com/?p=proxmox-backup.git;a=commit;h=f25b75287e8c28b568760589c2dd47eab43eeae9
 
Hello Chris, here's the output:
Code: 
proxmox-backup                     4.2.0         running kernel: 7.0.14-8-pve
proxmox-backup-server              4.2.4-1       running version: 4.2.4
proxmox-kernel-helper              9.2.0
proxmox-kernel-7.0.14-8-pve-signed 7.0.14-8
proxmox-kernel-7.0                 7.0.14-8
proxmox-kernel-7.0.0-3-pve-signed  7.0.0-3
ifupdown2                          3.3.0-1+pmx12
libjs-extjs                        7.0.0-5
proxmox-backup-docs                4.2.4-1
proxmox-backup-client              4.2.4-1
proxmox-enterprise-support-keyring 1.1
proxmox-mail-forward               1.0.3
proxmox-mini-journalreader         1.7
proxmox-offline-mirror-helper      0.7.4
proxmox-widget-toolkit             5.2.6
pve-xtermjs                        6.0.0-2
smartmontools                      7.5-pve2
zfsutils-linux                     2.4.3-pve1

I saw some connection errors in the logs so my guess is that the virtual network device was faulty. I changed it for VirtIO and I installed qemu-guest-agent. Currently the backup seems fine, I'll let you know if I get some errors.
 
You must log in or register to reply here.
Top Bottom
Back