Hello,
I'm currently trying the new S3 feature from PBS 4.
I have three cluster getting backed up by the same PBS instance, into 2 different DS, one local, one S3 for a new cluster.
PBS freeze during backup, the Web UI become unresponsive, the backup job fail, i need to restart proxmox-backup-proxy to fix it.
The instance is running in a Proxmox VE VM:
I have three datastore:
DATASTORE_D S3
DATASTORE_R S3
pbs-local (local DS that I wanted to migrate to DATASTORE_R)
S3 target are OVHCloud bucket located in their WAW region.
When the backup job is running, PBS just straight up freeze (Web UI included), with nothing in the log:
I'm kinda lost because I can't find an explanation for the issue in the logs. Any idea ?
Thanks.
Best,
Maiko
I'm currently trying the new S3 feature from PBS 4.
I have three cluster getting backed up by the same PBS instance, into 2 different DS, one local, one S3 for a new cluster.
PBS freeze during backup, the Web UI become unresponsive, the backup job fail, i need to restart proxmox-backup-proxy to fix it.
The instance is running in a Proxmox VE VM:
Code:
root@proxmox-2:~# cat /etc/pve/qemu-server/1002.conf
agent: 1,freeze-fs-on-backup=0,fstrim_cloned_disks=1
balloon: 4096
boot: order=scsi0
cores: 4
cpu: host,flags=+aes
cpuunits: 7500
hotplug: 0
machine: q35
memory: 8192
name: pbs
net0: virtio=02:00:00:66:1c:ff,bridge=vmbr0
net1: virtio=BC:24:11:7A:35:09,bridge=pbs
numa: 0
onboot: 1
ostype: l26
scsi0: rpool:vm-1002-disk-0,discard=on,iothread=1,size=32G,ssd=1
scsi1: rpool:vm-1002-disk-1,discard=on,iothread=1,size=256G,ssd=1
scsi2: rpool:vm-1002-disk-2,backup=0,discard=on,iothread=1,size=1256G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=3fe8e153-c525-4b85-a5c6-11d0a9b117d1
sockets: 1
startup: order=3,up=5,down=120
tablet: 0
tags: perso
vcpus: 4
vmgenid: fc406f36-79a4-4cfb-8786-cf1d82167d47I have three datastore:
DATASTORE_D S3
DATASTORE_R S3
pbs-local (local DS that I wanted to migrate to DATASTORE_R)
S3 target are OVHCloud bucket located in their WAW region.
When the backup job is running, PBS just straight up freeze (Web UI included), with nothing in the log:
Code:
root@pbs:~# date
Sat Sep 6 09:17:12 AM CEST 2025
root@pbs:~# time curl https://localhost:8007
^C
real 0m21.952s
user 0m0.022s
sys 0m0.004s
root@pbs:~# systemctl status proxmox-backup
● proxmox-backup.service - Proxmox Backup API Server
Loaded: loaded (/usr/lib/systemd/system/proxmox-backup.service; enabled; preset: enabled)
Active: active (running) since Fri 2025-09-05 13:15:40 CEST; 20h ago
Invocation: c7deca2e28cf41579ff93bdb70ad9161
Main PID: 668 (proxmox-backup-)
Tasks: 8 (limit: 9443)
Memory: 27.3M (peak: 435.8M, swap: 9M, swap peak: 10.4M)
CPU: 1h 43min 6.655s
CGroup: /system.slice/proxmox-backup.service
└─668 /usr/lib/x86_64-linux-gnu/proxmox-backup/proxmox-backup-api
Sep 05 16:34:22 pbs proxmox-backup-api[668]: authentication failure; rhost=[::ffff:xx.xx.xx.xx]:53440 user=root@pam msg=invalid tfa response
Sep 05 17:05:59 pbs proxmox-backup-api[668]: notified via target `mail-to-root`
Sep 05 21:21:23 pbs proxmox-backup-api[668]: notified via target `mail-to-root`
Sep 05 21:22:08 pbs proxmox-backup-api[668]: notified via target `mail-to-root`
Sep 06 00:00:18 pbs proxmox-backup-api[668]: re-opening access-log file
Sep 06 00:00:25 pbs proxmox-backup-api[668]: re-opening auth-log file
Sep 06 00:13:41 pbs proxmox-backup-api[668]: notified via target `mail-to-root`
Sep 06 09:11:03 pbs proxmox-backup-api[668]: pam_unix(proxmox-backup-auth:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=::ffff:xx.xx.xx.xx user=root
Sep 06 09:11:05 pbs proxmox-backup-api[668]: authentication failure; rhost=[::ffff:xx.xx.xx.xx]:57130 user=root@pam msg=authentication error - AUTH_ERR (7)
Sep 06 09:11:09 pbs proxmox-backup-api[668]: authentication failure; rhost=[::ffff:xx.xx.xx.xx]:57131 user=root@pam msg=invalid tfa response
root@pbs:~# systemctl status proxmox-backup-proxy
● proxmox-backup-proxy.service - Proxmox Backup API Proxy Server
Loaded: loaded (/usr/lib/systemd/system/proxmox-backup-proxy.service; enabled; preset: enabled)
Active: active (running) since Fri 2025-09-05 13:15:40 CEST; 20h ago
Invocation: bc190e81b4394f71b40165a583c9334a
Main PID: 723 (proxmox-backup-)
Tasks: 7 (limit: 9443)
Memory: 1G (peak: 3G, swap: 26M, swap peak: 52.4M)
CPU: 3h 58min 7.362s
CGroup: /system.slice/proxmox-backup-proxy.service
└─723 /usr/lib/x86_64-linux-gnu/proxmox-backup/proxmox-backup-proxy
Sep 06 04:45:50 pbs proxmox-backup-proxy[723]: rrd journal successfully committed (41 files in 0.014 seconds)
Sep 06 05:15:50 pbs proxmox-backup-proxy[723]: rrd journal successfully committed (41 files in 0.015 seconds)
Sep 06 05:45:50 pbs proxmox-backup-proxy[723]: rrd journal successfully committed (41 files in 0.041 seconds)
Sep 06 06:15:50 pbs proxmox-backup-proxy[723]: rrd journal successfully committed (41 files in 0.046 seconds)
Sep 06 06:45:50 pbs proxmox-backup-proxy[723]: rrd journal successfully committed (41 files in 0.039 seconds)
Sep 06 07:15:51 pbs proxmox-backup-proxy[723]: rrd journal successfully committed (41 files in 0.127 seconds)
Sep 06 07:45:51 pbs proxmox-backup-proxy[723]: rrd journal successfully committed (41 files in 0.015 seconds)
Sep 06 08:15:51 pbs proxmox-backup-proxy[723]: rrd journal successfully committed (41 files in 0.063 seconds)
Sep 06 08:45:51 pbs proxmox-backup-proxy[723]: rrd journal successfully committed (41 files in 0.057 seconds)
Sep 06 09:11:48 pbs proxmox-backup-proxy[723]: error during snapshot file listing: 'unable to load blob '"/mnt/datastore/s3-cache/DATASTORE_D/ns/NAMESPACE_D/vm/303/2025-09-05T04:13:51Z/index.json.blob"' - No such fil>
root@pbs:~# systemctl restart proxmox-backup-proxy
root@pbs:~# systemctl status proxmox-backup-proxy
● proxmox-backup-proxy.service - Proxmox Backup API Proxy Server
Loaded: loaded (/usr/lib/systemd/system/proxmox-backup-proxy.service; enabled; preset: enabled)
Active: active (running) since Sat 2025-09-06 09:19:32 CEST; 9s ago
Invocation: 18c15ae0a6fd49d091562bc58880eff8
Main PID: 20945 (proxmox-backup-)
Tasks: 9 (limit: 9443)
Memory: 24.4M (peak: 26.7M)
CPU: 452ms
CGroup: /system.slice/proxmox-backup-proxy.service
└─20945 /usr/lib/x86_64-linux-gnu/proxmox-backup/proxmox-backup-proxy
Sep 06 09:19:31 pbs systemd[1]: Starting proxmox-backup-proxy.service - Proxmox Backup API Proxy Server...
Sep 06 09:19:32 pbs proxmox-backup-proxy[20945]: applied rrd journal (6596 entries in 0.016 seconds)
Sep 06 09:19:32 pbs proxmox-backup-proxy[20945]: catching shutdown signal
Sep 06 09:19:32 pbs proxmox-backup-proxy[20945]: catching reload signal
Sep 06 09:19:32 pbs systemd[1]: Started proxmox-backup-proxy.service - Proxmox Backup API Proxy Server.
Sep 06 09:19:32 pbs proxmox-backup-proxy[20945]: rrd journal successfully committed (41 files in 0.113 seconds)
Sep 06 09:19:32 pbs proxmox-backup-proxy[20945]: Using datastore cache with capacity 8096 for store DATASTORE_D
root@pbs:~#
root@pbs:~# curl https://localhost:8007
curl: (60) SSL: no alternative certificate subject name matches target hostname 'localhost'
Code:
root@pbs:~# proxmox-backup-manager versions --verbose
proxmox-backup 4.0.0 running kernel: 6.14.11-1-pve
proxmox-backup-server 4.0.14-1 running version: 4.0.14
proxmox-kernel-helper 9.0.4
pve-kernel-5.11 7.0-10
proxmox-kernel-6.14.11-1-pve-signed 6.14.11-1
proxmox-kernel-6.14 6.14.11-1
proxmox-kernel-6.14.8-2-pve-signed 6.14.8-2
proxmox-kernel-6.8.12-13-pve-signed 6.8.12-13
proxmox-kernel-6.8 6.8.12-13
pve-kernel-5.13.19-2-pve 5.13.19-4
pve-kernel-5.11.22-7-pve 5.11.22-12
pve-kernel-5.11.22-1-pve 5.11.22-2
ifupdown2 3.3.0-1+pmx10
libjs-extjs 7.0.0-5
proxmox-backup-docs 4.0.14-1
proxmox-backup-client 4.0.14-1
proxmox-mail-forward 1.0.2
proxmox-mini-journalreader 1.6
proxmox-offline-mirror-helper 0.7.1
proxmox-widget-toolkit 5.0.5
pve-xtermjs 5.5.0-2
smartmontools 7.4-pve1
zfsutils-linux 2.3.4-pve1
root@pbs:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
s3-cache 10.7G 235G 96K /mnt/datastore/s3-cache
s3-cache/D 1.50G 127G 1.50G /mnt/datastore/s3-cache/DATASTORE_D
s3-cache/R 9.18G 119G 9.18G /mnt/datastore/s3-cache/DATASTORE_R
root@pbs:~# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
s3-cache 254G 10.7G 243G - - 25% 4% 1.00x ONLINE -
root@pbs:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 3.9G 0 3.9G 0% /dev
tmpfs 795M 1.5M 793M 1% /run
/dev/mapper/pbs-root 24G 8.2G 14G 38% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1.0M 0 1.0M 0% /run/credentials/systemd-journald.service
tmpfs 3.9G 0 3.9G 0% /tmp
/dev/sdc1 1.3T 296G 961G 24% /mnt/datastore/pbs-local
tmpfs 1.0M 0 1.0M 0% /run/credentials/getty@tty1.service
s3-cache 236G 128K 236G 1% /mnt/datastore/s3-cache
s3-cache/D 128G 1.5G 127G 2% /mnt/datastore/s3-cache/DATASTORE_D
s3-cache/R 128G 9.2G 119G 8% /mnt/datastore/s3-cache/DATASTORE_R
tmpfs 795M 8.0K 795M 1% /run/user/0
root@pbs:~# proxmox-backup-manager s3 endpoint list --output-format json-pretty
[
{
"access-key": "XXXXX",
"endpoint": "{{bucket}}.s3.{{region}}.io.cloud.ovh.net",
"id": "OVHCloud",
"provider-quirks": [
"skip-if-none-match-header"
],
"region": "waw"
},
{
"access-key": "YYYYY",
"endpoint": "{{bucket}}.s3.{{region}}.io.cloud.ovh.net",
"id": "OVHCloud-D",
"provider-quirks": [
"skip-if-none-match-header"
],
"region": "waw"
}
]
root@pbs:~# proxmox-backup-manager datastore list --output-format json-pretty
[
{
"backend": "type=s3,client=OVHCloud-D,bucket=proxmoxbackupserver-d",
"gc-schedule": "11:00",
"name": "digityser",
"path": "/mnt/datastore/s3-cache/DATASTORE_D"
},
{
"backend": "bucket=pbs-r,client=OVHCloud,type=s3",
"comment": "",
"gc-schedule": "daily",
"name": "DATASTORE_R",
"notification-mode": "notification-system",
"path": "/mnt/datastore/s3-cache/DATASTORE_R"
},
{
"gc-schedule": "daily",
"name": "pbs-local",
"notification-mode": "legacy-sendmail",
"notify": "gc=error,prune=error,sync=error,verify=error",
"path": "/mnt/datastore/pbs-local"
}
]I'm kinda lost because I can't find an explanation for the issue in the logs. Any idea ?
Thanks.
Best,
Maiko
