[SOLVED] Error message during backup: backup write data failed: command error: protocol canceled

[rada]

Hi everyone! We have been using Proxmox Backup Server during the last month without issues. It's a wonderful complement to Proxmox Virtual Environment.

This week several backups failed and we would like to further troubleshoot the issue.

In one of our PVE installations, the backup process starts and several VMs and CTs are properly backuped. However, we receive this error on one of them:

709: 2021-02-12 00:00:57 INFO: Starting Backup of VM 709 (qemu)
709: 2021-02-12 00:00:57 INFO: status = running
709: 2021-02-12 00:00:57 INFO: VM Name: [VM name removed]
709: 2021-02-12 00:00:57 INFO: include disk 'scsi0' 'local-lvm:vm-709-disk-0' 500G
709: 2021-02-12 00:00:57 INFO: backup mode: snapshot
709: 2021-02-12 00:00:57 INFO: ionice priority: 7
709: 2021-02-12 00:00:57 INFO: creating Proxmox Backup Server archive 'vm/709/2021-02-12T03:00:57Z'
709: 2021-02-12 00:00:58 INFO: started backup task '68dbcac5-4267-4496-a748-6d461ebcfc45'
709: 2021-02-12 00:00:58 INFO: resuming VM again
709: 2021-02-12 00:00:58 INFO: scsi0: dirty-bitmap status: OK (16.4 GiB of 500.0 GiB dirty)
709: 2021-02-12 00:00:58 INFO: using fast incremental mode (dirty-bitmap), 16.4 GiB dirty of 500.0 GiB total
709: 2021-02-12 00:01:01 INFO:   1% (284.0 MiB of 16.4 GiB) in  3s, read: 94.7 MiB/s, write: 93.3 MiB/s
709: 2021-02-12 00:01:04 INFO:   3% (564.0 MiB of 16.4 GiB) in  6s, read: 93.3 MiB/s, write: 81.3 MiB/s
709: 2021-02-12 00:01:07 INFO:   5% (872.0 MiB of 16.4 GiB) in  9s, read: 102.7 MiB/s, write: 58.7 MiB/s
709: 2021-02-12 00:01:10 INFO:   6% (1.1 GiB of 16.4 GiB) in 12s, read: 93.3 MiB/s, write: 81.3 MiB/s
709: 2021-02-12 00:01:13 INFO:   8% (1.4 GiB of 16.4 GiB) in 15s, read: 97.3 MiB/s, write: 94.7 MiB/s
709: 2021-02-12 00:01:16 INFO:   9% (1.5 GiB of 16.4 GiB) in 18s, read: 25.3 MiB/s, write: 25.3 MiB/s
709: 2021-02-12 00:09:15 INFO:   9% (1.5 GiB of 16.4 GiB) in  8m 17s, read: 0 B/s, write: 0 B/s
709: 2021-02-12 00:09:15 ERROR: backup write data failed: command error: protocol canceled
709: 2021-02-12 00:09:15 INFO: aborting backup job
709: 2021-02-12 00:09:15 ERROR: Backup of VM 709 failed - backup write data failed: command error: protocol canceled

[EDIT]

Under the datastore in PBS, we see the "loading" icon under the backup size:



Would you give us further assistance to troubleshoot this issue?

Thank you so much for your kind support and assistance!

 

[dcsapak]

this could mean that the http2 connection is dropped somewhere.... can you post the logs from the server also?
is there any device between the client and server that could influence the http2 connection?

 

[rada]

Dear @dcsapak, thank you so much for pointing us in the right direction. Although we checked the backup logs, when we checked the syslog we noticed a lot of kernel panic messages.

We had to get physical access to the server and we noticed the PSU was failing. After solving the issue, we could start again the backup job and ended OK.

Thank you so much for your patience and support.