PBS unavailable

A

andmattia

Member
Feb 13, 2024
37
3
8
I add some S3 storage and randomly the proxmox-backup-proxy.service stop and broke backup process. Server it's still live and work but process dump without error.

In PV node I see unreachable datastore but if I restare the service all works fine.

Any idea how to investigate the cause?
 
Last edited:
andmattia said:
Maybe. Any idea if I can mitigate this? This fix will be release in next version/patch?
Click to expand...
No, there is no mitigation and yes, this will be part of the next version bump. I cannot give an ETA however.
 
After some test my idea is that all issues is relate do incremental
NFO: include disk 'scsi0' 'local:110/vm-110-disk-1.raw' 127G
INFO: include disk 'scsi1' 'local:110/vm-110-disk-3.raw' 60G
INFO: include disk 'efidisk0' 'local:110/vm-110-disk-0.qcow2' 528K
INFO: include disk 'tpmstate0' 'local:110/vm-110-disk-2.raw' 4M
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/110/2025-08-27T21:22:34Z'
INFO: attaching TPM drive to QEMU for backup
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '73360cb0-d767-4ca6-8468-18a4536b8ecd'
INFO: resuming VM again
INFO: efidisk0: dirty-bitmap status: OK (drive clean)
INFO: scsi0: dirty-bitmap status: OK (500.0 MiB of 127.0 GiB dirty)
INFO: scsi1: dirty-bitmap status: OK (172.0 MiB of 60.0 GiB dirty)
INFO: tpmstate0-backup: dirty-bitmap status: created new
INFO: using fast incremental mode (dirty-bitmap), 672.0 MiB dirty of 187.0 GiB total
INFO: 100% (672.0 MiB of 672.0 MiB) in 3s, read: 224.0 MiB/s, write: 218.7 MiB/s
ERROR: backup close image failed: command error: stream closed because of a broken pipe
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 110 failed - backup close image failed: command error: stream closed because of a broken pipe
INFO: Failed at 2025-08-27 23:22:53
INFO: Backup job finished with errors
INFO: notified via target `support`
INFO: skipping disabled target 'mail-to-root'
TASK ERROR: job errors

if is a fresh backup usually no issue
 
Chris said:
proxmox-backup-server version 4.0.15-1 is available in the pbs-test repository. You might want to test if the included bugfixes solve your issue, thanks. To activate the test repo, please see https://pbs.proxmox.com/docs/installation.html#proxmox-backup-test-repository
Click to expand...
I too am having this issue, and attempted using `4.0.15-1`; however, the issue persists. I have the same behavior as before: backups stop at 100%, and only a restart of `proxmox-backup-proxy.service` restores access to the UI, in turn cancelling all backup jobs.
 
Unfortunately the provided bufix seems to be incomplete or the issue unrelated, as also reported by others https://bugzilla.proxmox.com/show_bug.cgi?id=6750.

Please share the VM config for the VM at which the backup job fails, qm config <VMID> --current
 
Here are the logs and config:

Code: 
Header
Proxmox
Virtual Environment 9.0.10
Datacenter
Logs
()
INFO: starting new backup job: vzdump 101 102 107 108 100 400 109 105 201 9000 9001 9002 9003 --notes-template '{{guestname}}' --storage xxx --mode snapshot --fleecing 0 --all 0 --node pve-4 --notification-mode notification-system
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2025-09-18 08:08:08
INFO: status = running
INFO: include disk 'scsi0' 'pool-1:vm-100-disk-1' 32G
INFO: include disk 'efidisk0' 'pool-1:vm-100-disk-0' 528K
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/100/2025-09-18T13:08:08Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '418975ef-71c3-4348-b661-2e0debbd0b12'
INFO: resuming VM again
INFO: efidisk0: dirty-bitmap status: OK (drive clean)
INFO: scsi0: dirty-bitmap status: OK (1.7 GiB of 32.0 GiB dirty)
INFO: using fast incremental mode (dirty-bitmap), 1.7 GiB dirty of 32.0 GiB total
INFO:  47% (816.0 MiB of 1.7 GiB) in 3s, read: 272.0 MiB/s, write: 269.3 MiB/s
INFO:  56% (976.0 MiB of 1.7 GiB) in 6s, read: 53.3 MiB/s, write: 53.3 MiB/s
INFO:  79% (1.3 GiB of 1.7 GiB) in 9s, read: 129.3 MiB/s, write: 126.7 MiB/s
INFO: 100% (1.7 GiB of 1.7 GiB) in 12s, read: 120.0 MiB/s, write: 120.0 MiB/s
INFO: backup was done incrementally, reused 30.33 GiB (94%)
INFO: transferred 1.68 GiB in 20 seconds (86.2 MiB/s)
INFO: adding notes to backup
INFO: Finished Backup of VM 100 (00:00:22)
INFO: Backup finished at 2025-09-18 08:08:30
INFO: Starting Backup of VM 102 (qemu)
INFO: Backup started at 2025-09-18 08:08:31
INFO: status = running
INFO: include disk 'scsi0' 'pool-1:vm-102-disk-0' 64G
INFO: include disk 'scsi1' 'pool-1:vm-102-disk-1' 32G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/102/2025-09-18T13:08:31Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'd7050a82-404f-430a-8813-77023a5925c2'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: OK (164.0 MiB of 64.0 GiB dirty)
INFO: scsi1: dirty-bitmap status: OK (40.0 MiB of 32.0 GiB dirty)
INFO: using fast incremental mode (dirty-bitmap), 204.0 MiB dirty of 96.0 GiB total
INFO: 100% (204.0 MiB of 204.0 MiB) in 3s, read: 68.0 MiB/s, write: 64.0 MiB/s
ERROR: backup close image failed: command error: stream closed because of a broken pipe
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 102 failed - backup close image failed: command error: stream closed because of a broken pipe
INFO: Failed at 2025-09-18 08:11:24
INFO: Starting Backup of VM 108 (qemu)
INFO: Backup started at 2025-09-18 08:11:24
INFO: status = running
INFO: include disk 'scsi0' 'pool-1:vm-108-disk-1' 32G
INFO: include disk 'efidisk0' 'pool-1:vm-108-disk-0' 528K
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/108/2025-09-18T13:11:24Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '20252268-cf99-497b-9a44-faa16f2f4f45'
INFO: resuming VM again
INFO: efidisk0: dirty-bitmap status: OK (drive clean)
INFO: scsi0: dirty-bitmap status: OK (120.0 MiB of 32.0 GiB dirty)
INFO: using fast incremental mode (dirty-bitmap), 120.0 MiB dirty of 32.0 GiB total
INFO: 100% (120.0 MiB of 120.0 MiB) in 3s, read: 40.0 MiB/s, write: 40.0 MiB/s
INFO: Waiting for server to finish backup validation...
INFO: backup was done incrementally, reused 31.88 GiB (99%)
INFO: transferred 120.00 MiB in 4 seconds (30.0 MiB/s)
INFO: adding notes to backup
INFO: Finished Backup of VM 108 (00:00:09)
INFO: Backup finished at 2025-09-18 08:11:33
INFO: Starting Backup of VM 201 (qemu)
INFO: Backup started at 2025-09-18 08:11:36
INFO: status = running
INFO: VM Name: unifi-controller-4
INFO: include disk 'scsi0' 'pool-1:vm-201-disk-0' 32G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/201/2025-09-18T13:11:36Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '7b838838-c356-4a3d-ad7f-4f1f2b8c53e6'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: OK (632.0 MiB of 32.0 GiB dirty)
INFO: using fast incremental mode (dirty-bitmap), 632.0 MiB dirty of 32.0 GiB total
INFO: 100% (632.0 MiB of 632.0 MiB) in 3s, read: 210.7 MiB/s, write: 202.7 MiB/s
INFO: backup was done incrementally, reused 31.41 GiB (98%)
INFO: transferred 632.00 MiB in 9 seconds (70.2 MiB/s)
INFO: adding notes to backup
INFO: Finished Backup of VM 201 (00:00:12)
INFO: Backup finished at 2025-09-18 08:11:48
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors

Code: 
agent: 1
boot: order=scsi0;net0
cores: 2
machine: pc,viommu=virtio
memory: 2048
meta: creation-qemu=7.1.0,ctime=1672956432
name: 
net0: virtio=XXXXXX,bridge=Infra,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: pool-1:vm-102-disk-0,discard=on,iothread=1,size=64G
scsi1: pool-1:vm-102-disk-1,discard=on,iothread=1,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=833f056f-03ca-411b-b335-51b6f4996c43
sockets: 1
vga: qxl
vmgenid: 526f51d2-5667-488f-9cf9-2d23a726bffa
 
I was able to get through a few complete backups with 4.0.16 in the test repo. Thank you for the fix. Hope it works out for everyone else.
 
You must log in or register to reply here.
Top Bottom