hello, around every four backup runs we encounter the following problem: the backup of a single proxmox ve with a single vm to pbs seems to fail:
while this wouldn't bother us too much the problem is that the vm that is backuped has a broken file system afterwards so somehow the failed backup damages some bits. what happens is:
- vm is stopped
- vm is immediately started again
- data is copied
- copying fails for some reason
- shortly after the whole vm freezes and writes no more logs
- rebooting the vm takes you into initram where one has to repair the file system, luckily that has worked so far.
i'm now testing to shutdown the vm via vzdump hook script at the beginning of the backup and restart it at the end so that the vm is powered down during the backup process. however, so far the error has not shown again.
Code:
INFO: starting new backup job: vzdump --quiet 1 --mode stop --fleecing 0 --storage synology --all 1 --notification-mode notification-system --notes-template '{{guestname}}'
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2026-05-09 23:00:02
INFO: status = running
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: tomedo
INFO: include disk 'scsi0' 'local-zfs:vm-101-disk-0' 1000G
INFO: stopping virtual guest
INFO: creating Proxmox Backup Server archive 'vm/101/2026-05-09T21:00:02Z'
INFO: enabling encryption
INFO: starting kvm to execute backup task
INFO: starting backup via QMP command
INFO: started backup task 'fa1c7773-603f-4455-b25a-b3cb58281128'
INFO: resuming VM again after 95 seconds
INFO: scsi0: dirty-bitmap status: created new
INFO: 0% (1.9 GiB of 1000.0 GiB) in 3s, read: 640.0 MiB/s, write: 380.0 MiB/s
INFO: 1% (11.1 GiB of 1000.0 GiB) in 6s, read: 3.1 GiB/s, write: 189.3 MiB/s
INFO: 2% (20.7 GiB of 1000.0 GiB) in 10s, read: 2.4 GiB/s, write: 93.0 MiB/s
INFO: 3% (30.9 GiB of 1000.0 GiB) in 21s, read: 953.1 MiB/s, write: 59.6 MiB/s
INFO: 4% (40.6 GiB of 1000.0 GiB) in 33s, read: 827.3 MiB/s, write: 25.7 MiB/s
INFO: 5% (50.1 GiB of 1000.0 GiB) in 46s, read: 749.2 MiB/s, write: 15.4 MiB/s
INFO: 6% (62.4 GiB of 1000.0 GiB) in 59s, read: 970.2 MiB/s, write: 10.2 MiB/s
INFO: 8% (80.7 GiB of 1000.0 GiB) in 1m 2s, read: 6.1 GiB/s, write: 14.7 MiB/s
INFO: 9% (99.5 GiB of 1000.0 GiB) in 1m 5s, read: 6.3 GiB/s, write: 13.3 MiB/s
INFO: 11% (118.6 GiB of 1000.0 GiB) in 1m 8s, read: 6.4 GiB/s, write: 2.7 MiB/s
INFO: 13% (137.4 GiB of 1000.0 GiB) in 1m 11s, read: 6.3 GiB/s, write: 2.7 MiB/s
INFO: 15% (155.8 GiB of 1000.0 GiB) in 1m 14s, read: 6.1 GiB/s, write: 0 B/s
INFO: 16% (169.3 GiB of 1000.0 GiB) in 1m 17s, read: 4.5 GiB/s, write: 61.3 MiB/s
INFO: 18% (188.6 GiB of 1000.0 GiB) in 1m 20s, read: 6.4 GiB/s, write: 0 B/s
INFO: 19% (195.0 GiB of 1000.0 GiB) in 1m 23s, read: 2.1 GiB/s, write: 96.0 MiB/s
INFO: 20% (200.9 GiB of 1000.0 GiB) in 1m 30s, read: 860.0 MiB/s, write: 98.9 MiB/s
INFO: 21% (210.1 GiB of 1000.0 GiB) in 1m 45s, read: 625.3 MiB/s, write: 108.8 MiB/s
INFO: 21% (210.1 GiB of 1000.0 GiB) in 17m 33s, read: 0 B/s, write: 0 B/s
ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: timed out
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 101 failed - backup write data failed: command error: write_data upload error: pipelined request failed: timed out
INFO: Failed at 2026-05-09 23:19:10
INFO: Backup job finished with errors
INFO: notified via target `strato`
INFO: notified via target `mail-to-root`
TASK ERROR: job errorswhile this wouldn't bother us too much the problem is that the vm that is backuped has a broken file system afterwards so somehow the failed backup damages some bits. what happens is:
- vm is stopped
- vm is immediately started again
- data is copied
- copying fails for some reason
- shortly after the whole vm freezes and writes no more logs
Code:
Mai 09 23:02:56 tomedo com.teamviewer.TeamViewer[3265]: QSystemTrayIcon::setVisible: No Icon set
-- Boot 06c56b41a531498395caddcf71344ddd --
Mai 10 15:20:29 tomedo kernel: shpchp 0000:05:01.0: pci_hp_register failed with error -16- rebooting the vm takes you into initram where one has to repair the file system, luckily that has worked so far.
i'm now testing to shutdown the vm via vzdump hook script at the beginning of the backup and restart it at the end so that the vm is powered down during the backup process. however, so far the error has not shown again.
