Proxmox backups not working after PBS server restart

[xPakrikx]

Every time after restart of PBS server, backups just stop working a we need to reboot all PVE nodes.

INFO: starting new backup job: vzdump 131 --notes-template '{{guestname}}' --storage backup01-cl01-thiss --remove 0 --mode snapshot --node pve008
INFO: Starting Backup of VM 131 (qemu)
INFO: Backup started at 2022-10-14 19:37:12
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: lp-th-dc-dns01
INFO: include disk 'scsi0' 'storage01-vol004:vm-131-disk-0' 16G
INFO: creating Proxmox Backup Server archive 'vm/131/2022-10-14T17:37:12Z'
INFO: starting kvm to execute backup task
INFO: started backup task '49ccbb02-c15f-4803-81af-b38c3b43f377'
INFO: scsi0: dirty-bitmap status: created new
INFO:   4% (692.0 MiB of 16.0 GiB) in 3s, read: 230.7 MiB/s, write: 0 B/s
INFO:   5% (912.0 MiB of 16.0 GiB) in 6s, read: 73.3 MiB/s, write: 0 B/s

Backup job is just stuck.

After some time backup job failed with error:

ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: timed out

Restore from same backup server works just fine but vzdump nope. Currently only fix for this is reboot every node. Any idea where can be a problem ?

For backup we are using Dell R530 (PERC H730 raid6 6x10TB single ext4 partition)

 

[xPakrikx]

Also is there any documentation about PBS architecture or i need to go on source code level ?

 

[xPakrikx]

Any idea how to restart pbs connector on PVE ? I tried reload network interfaces and remove an add PBS storage in PVE but nothing helps. Only reboot

 

[xPakrikx]

Update: backups starts working after many restarts od PBS server. So now i am lost

 

[dcsapak]

you could post the syslog/journal from the time it's not working, maybe there is a hint there what's going wrong