[SOLVED] restore failed: detected chunk with wrong digest.

[jasonsansone]

I am attempting to restore a backup from PBS to a node. I have run a complete verification pass on all backups. Everything passes. However, the restore immediately errors out with "restore failed: detected chunk with wrong digest.". What is going on?

new volume ID is 'CephRBD:vm-112-disk-0'
/dev/rbd6
new volume ID is 'CephRBD:vm-112-disk-1'
/dev/rbd7
restore proxmox backup image: /usr/bin/pbs-restore --repository root@pam@192.168.2.222:Home vm/112/2021-05-17T05:31:54Z drive-efidisk0.img.fidx /dev/rbd/CephRBD/vm-112-disk-0 --verbose --format raw --skip-zero
connecting to repository 'root@pam@192.168.2.222:Home'
open block backend for target '/dev/rbd/CephRBD/vm-112-disk-0'
starting to restore snapshot 'vm/112/2021-05-17T05:31:54Z'
download and verify backup index
progress 100% (read 131072 bytes, zeroes = 0% (0 bytes), duration 0 sec)
restore image complete (bytes=131072, duration=0.00s, speed=60.41MB/s)
restore proxmox backup image: /usr/bin/pbs-restore --repository root@pam@192.168.2.222:Home vm/112/2021-05-17T05:31:54Z drive-virtio0.img.fidx /dev/rbd/CephRBD/vm-112-disk-1 --verbose --format raw --skip-zero
connecting to repository 'root@pam@192.168.2.222:Home'
open block backend for target '/dev/rbd/CephRBD/vm-112-disk-1'
starting to restore snapshot 'vm/112/2021-05-17T05:31:54Z'
download and verify backup index
restore failed: detected chunk with wrong digest.

Node Packages:

proxmox-ve: 6.4-1 (running kernel: 5.11.17-1-pve)
pve-manager: 6.4-6 (running version: 6.4-6/be2fa32c)
pve-kernel-5.11: 7.0-1~bpo10
pve-kernel-5.4: 6.4-2
pve-kernel-helper: 6.4-2
pve-kernel-5.11.17-1-pve: 5.11.17-1~bpo10
pve-kernel-5.11.7-1-pve: 5.11.7-1~bpo10
pve-kernel-5.4.114-1-pve: 5.4.114-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph: 15.2.11-pve1
ceph-fuse: 15.2.11-pve1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: not correctly installed
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.4-1
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-3
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-2
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.6-2
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-4
pve-cluster: 6.4-1
pve-container: 3.3-5
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-3
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1

PBS Packages:

proxmox-backup: 1.0-4 (running kernel: 5.11.17-1-pve)
proxmox-backup-server: 1.1.5-1 (running version: 1.1.5)
pve-kernel-5.11: 7.0-0+3~bpo10
pve-kernel-5.4: 6.4-1
pve-kernel-helper: 6.4-1
pve-kernel-5.11.7-1-pve: 5.11.7-1~bpo10
pve-kernel-5.4.106-1-pve: 5.4.106-1
ifupdown2: 3.0.0-1+pve3
libjs-extjs: 6.0.1-10
proxmox-backup-docs: 1.1.5-1
proxmox-backup-client: 1.1.5-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-2
pve-xtermjs: 4.7.0-3
smartmontools: 7.2-pve2
zfsutils-linux: 2.0.4-pve1

 

[dcsapak]

can you also post the 'read' task from the pbs side?

 

[jasonsansone]

The samething happens when taking a backup. Here are the logs.

PVE Log:

INFO: starting new backup job: vzdump 112 --storage HomePBS --remove 0 --node viper --mode snapshot INFO: Starting Backup of VM 112 (qemu) INFO: Backup started at 2021-05-27 08:15:05 INFO: status = running INFO: VM Name: catalina INFO: include disk 'efidisk0' 'CephRBD:vm-112-disk-0' 128K INFO: backup mode: snapshot INFO: ionice priority: 7 INFO: creating Proxmox Backup Server archive 'vm/112/2021-05-27T13:15:05Z' INFO: started backup task 'a5ab761b-fdc1-46ae-9b80-ba67a8913871' INFO: resuming VM again INFO: efidisk0: dirty-bitmap status: existing bitmap was invalid and has been cleared INFO: virtio0: dirty-bitmap status: existing bitmap was invalid and has been cleared INFO:   0% (312.0 MiB of 500.0 GiB) in 3s, read: 104.0 MiB/s, write: 50.7 MiB/s INFO:   1% (5.7 GiB of 500.0 GiB) in 35s, read: 173.8 MiB/s, write: 18.6 MiB/s INFO:   1% (7.4 GiB of 500.0 GiB) in 37s, read: 864.0 MiB/s, write: 18.0 MiB/s ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: detected chunk with wrong digest. INFO: aborting backup job INFO: resuming VM again ERROR: Backup of VM 112 failed - backup write data failed: command error: write_data upload error: pipelined request failed: detected chunk with wrong digest. INFO: Failed at 2021-05-27 08:15:43 INFO: Backup job finished with errors TASK ERROR: job errors

PBS Log:

2021-05-27T08:15:05-05:00: starting new backup on datastore 'Home': "vm/112/2021-05-27T13:15:05Z" 2021-05-27T08:15:05-05:00: download 'index.json.blob' from previous backup. 2021-05-27T08:15:05-05:00: register chunks in 'drive-efidisk0.img.fidx' from previous backup. 2021-05-27T08:15:05-05:00: download 'drive-efidisk0.img.fidx' from previous backup. 2021-05-27T08:15:05-05:00: created new fixed index 1 ("vm/112/2021-05-27T13:15:05Z/drive-efidisk0.img.fidx") 2021-05-27T08:15:05-05:00: register chunks in 'drive-virtio0.img.fidx' from previous backup. 2021-05-27T08:15:05-05:00: download 'drive-virtio0.img.fidx' from previous backup. 2021-05-27T08:15:06-05:00: created new fixed index 2 ("vm/112/2021-05-27T13:15:05Z/drive-virtio0.img.fidx") 2021-05-27T08:15:06-05:00: add blob "/media/storage/home/vm/112/2021-05-27T13:15:05Z/qemu-server.conf.blob" (563 bytes, comp: 563) 2021-05-27T08:15:42-05:00: POST /fixed_chunk: 400 Bad Request: detected chunk with wrong digest. 2021-05-27T08:15:42-05:00: POST /fixed_chunk: 400 Bad Request: detected chunk with wrong digest. 2021-05-27T08:15:42-05:00: POST /fixed_chunk: 400 Bad Request: detected chunk with wrong digest. 2021-05-27T08:15:42-05:00: POST /fixed_chunk: 400 Bad Request: detected chunk with wrong digest. 2021-05-27T08:15:42-05:00: backup failed: connection error: Transport endpoint is not connected (os error 107) 2021-05-27T08:15:42-05:00: removing failed backup 2021-05-27T08:15:42-05:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: protocol error: stream no longer needed 2021-05-27T08:15:42-05:00: TASK ERROR: connection error: Transport endpoint is not connected (os error 107)

 

[jasonsansone]

Also encountering problems when syncing between remote and destination PBS.

Remote:

May 28 09:04:00 pbs proxmox-backup-proxy[194]: TASK ERROR: connection error: Connection reset by peer (os error 104)

Destination:

2021-05-28T09:04:00-05:00: sync group vm/115 failed - detected chunk with wrong digest.

 

[dcsapak]

i am still waiting on the task log of the 'Read' task from the pbs where you tried to restore the vm...

 

[dcsapak]

also i would check your memory, constant chunk digest errors indicate that something calculates a wrong checksum (client or server)

 

[jasonsansone]

also i would check your memory, constant chunk digest errors indicate that something calculates a wrong checksum (client or server)

Thanks for tips. I will have to do another restore attempt and access logs from both ends for you. Regarding memory: I have problems regardless of the cluster I am restoring to. All nodes have 192GB ECC DDR4. I wasn’t aware of a good way to test ECC, but I am up for any recommendations.

 

[entilza]

Thanks for tips. I will have to do another restore attempt and access logs from both ends for you. Regarding memory: I have problems regardless of the cluster I am restoring to. All nodes have 192GB ECC DDR4. I wasn’t aware of a good way to test ECC, but I am up for any recommendations.

The boot media has a mem test I believe, when you install proxmox.

Or use memtest86 or memtest.org

 

[jasonsansone]

The problem is unrelated to PBS directly. This thread can be closed. I will post separately in a depeserate plea for help. Thank you.