Backup failure - Invalid exchange (os error 52)

[aDZ5OfZ8l8GU8lSlLhBD]

Hi,

I have the following error backing up a Windows 2019 server VM - other VMs back up ok.

This is the command that is running:
INFO: starting new backup job: vzdump 103 --remove 0 --mode snapshot --node prox --storage PBS

PVE logs:
ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: Invalid exchange (os error 52)

PBS logs:
2021-06-22T10:58:41+01:00: POST /fixed_chunk: 400 Bad Request: Invalid exchange (os error 52)

There is nothing in dmesg on either host but on PBS it shows:

Jun 22 10:58:41 pbs zed: eid=37 class=data pool='Storage' priority=0 err=52 flags=0x808881 bookmark=54:53323:0:1 Jun 22 10:58:42 pbs zed: eid=38 class=checksum pool='Storage' vdev=sdb1 algorithm=fletcher4 size=4096 offset=17397821440 priority=0 err=52 flags=0x180880 delay=16ms bookmark=54:53323:0:1

Things I have tried:
1. removing all disks, backing up config only (works)
2. backing up individual disks on the server (there are two, fails with same error every time)
3. adding a third (VIRTIO SCSI) disk, using windows backup to backup to that, then running backup / vzdump on that disk alone (fails)

Does anyone have any ideas? I can't find any debug / verbosity switches within the vzdump command, I don't know where to go next.

Thanks

PS, package versions PVE:

proxmox-ve: 6.4-1 (running kernel: 5.11.21-1-pve)
pve-manager: 6.4-8 (running version: 6.4-8/185e14db)
pve-kernel-5.11: 7.0-2~bpo10
pve-kernel-5.4: 6.4-3
pve-kernel-helper: 6.4-3
pve-kernel-5.11.21-1-pve: 5.11.21-1~bpo10
pve-kernel-5.4.119-1-pve: 5.4.119-1
pve-kernel-5.4.114-1-pve: 5.4.114-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: not correctly installed
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.4-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-3
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-3
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.10-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-6
pve-cluster: 6.4-1
pve-container: 3.3-5
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.2-4
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1

Package versions PBS:

proxmox-backup: 1.0-4 (running kernel: 5.4.119-1-pve)
proxmox-backup-server: 1.1.10-1 (running version: 1.1.10)
pve-kernel-5.4: 6.4-3
pve-kernel-helper: 6.4-3
pve-kernel-5.4.119-1-pve: 5.4.119-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
ifupdown2: 3.0.0-1+pve3
libjs-extjs: 6.0.1-10
proxmox-backup-docs: 1.1.10-1
proxmox-backup-client: 1.1.10-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-6
pve-xtermjs: 4.7.0-3
smartmontools: 7.2-pve2
zfsutils-linux: 2.0.4-pve1

 

[dcsapak]

can you post your journal/syslog/dmesg of pbs?
is your storage ok? (e.g. for zfs, zpool status would be interesting)