[SOLVED] restore failed: detected chunk with wrong digest.

jasonsansone

Active Member
May 17, 2021
155
38
33
Oklahoma City, OK
www.sansonehowell.com
I am attempting to restore a backup from PBS to a node. I have run a complete verification pass on all backups. Everything passes. However, the restore immediately errors out with "restore failed: detected chunk with wrong digest.". What is going on?

Screen Shot 2021-05-17 at 11.55.52 PM.png

Code:
new volume ID is 'CephRBD:vm-112-disk-0'
/dev/rbd6
new volume ID is 'CephRBD:vm-112-disk-1'
/dev/rbd7
restore proxmox backup image: /usr/bin/pbs-restore --repository root@pam@192.168.2.222:Home vm/112/2021-05-17T05:31:54Z drive-efidisk0.img.fidx /dev/rbd/CephRBD/vm-112-disk-0 --verbose --format raw --skip-zero
connecting to repository 'root@pam@192.168.2.222:Home'
open block backend for target '/dev/rbd/CephRBD/vm-112-disk-0'
starting to restore snapshot 'vm/112/2021-05-17T05:31:54Z'
download and verify backup index
progress 100% (read 131072 bytes, zeroes = 0% (0 bytes), duration 0 sec)
restore image complete (bytes=131072, duration=0.00s, speed=60.41MB/s)
restore proxmox backup image: /usr/bin/pbs-restore --repository root@pam@192.168.2.222:Home vm/112/2021-05-17T05:31:54Z drive-virtio0.img.fidx /dev/rbd/CephRBD/vm-112-disk-1 --verbose --format raw --skip-zero
connecting to repository 'root@pam@192.168.2.222:Home'
open block backend for target '/dev/rbd/CephRBD/vm-112-disk-1'
starting to restore snapshot 'vm/112/2021-05-17T05:31:54Z'
download and verify backup index
restore failed: detected chunk with wrong digest.

Node Packages:
Code:
proxmox-ve: 6.4-1 (running kernel: 5.11.17-1-pve)
pve-manager: 6.4-6 (running version: 6.4-6/be2fa32c)
pve-kernel-5.11: 7.0-1~bpo10
pve-kernel-5.4: 6.4-2
pve-kernel-helper: 6.4-2
pve-kernel-5.11.17-1-pve: 5.11.17-1~bpo10
pve-kernel-5.11.7-1-pve: 5.11.7-1~bpo10
pve-kernel-5.4.114-1-pve: 5.4.114-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph: 15.2.11-pve1
ceph-fuse: 15.2.11-pve1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: not correctly installed
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.4-1
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-3
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-2
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.6-2
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-4
pve-cluster: 6.4-1
pve-container: 3.3-5
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-3
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1

PBS Packages:
Code:
proxmox-backup: 1.0-4 (running kernel: 5.11.17-1-pve)
proxmox-backup-server: 1.1.5-1 (running version: 1.1.5)
pve-kernel-5.11: 7.0-0+3~bpo10
pve-kernel-5.4: 6.4-1
pve-kernel-helper: 6.4-1
pve-kernel-5.11.7-1-pve: 5.11.7-1~bpo10
pve-kernel-5.4.106-1-pve: 5.4.106-1
ifupdown2: 3.0.0-1+pve3
libjs-extjs: 6.0.1-10
proxmox-backup-docs: 1.1.5-1
proxmox-backup-client: 1.1.5-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-2
pve-xtermjs: 4.7.0-3
smartmontools: 7.2-pve2
zfsutils-linux: 2.0.4-pve1
 
can you also post the 'read' task from the pbs side?
 
The samething happens when taking a backup. Here are the logs.

PVE Log:
Code:
INFO: starting new backup job: vzdump 112 --storage HomePBS --remove 0 --node viper --mode snapshot INFO: Starting Backup of VM 112 (qemu) INFO: Backup started at 2021-05-27 08:15:05 INFO: status = running INFO: VM Name: catalina INFO: include disk 'efidisk0' 'CephRBD:vm-112-disk-0' 128K INFO: backup mode: snapshot INFO: ionice priority: 7 INFO: creating Proxmox Backup Server archive 'vm/112/2021-05-27T13:15:05Z' INFO: started backup task 'a5ab761b-fdc1-46ae-9b80-ba67a8913871' INFO: resuming VM again INFO: efidisk0: dirty-bitmap status: existing bitmap was invalid and has been cleared INFO: virtio0: dirty-bitmap status: existing bitmap was invalid and has been cleared INFO:   0% (312.0 MiB of 500.0 GiB) in 3s, read: 104.0 MiB/s, write: 50.7 MiB/s INFO:   1% (5.7 GiB of 500.0 GiB) in 35s, read: 173.8 MiB/s, write: 18.6 MiB/s INFO:   1% (7.4 GiB of 500.0 GiB) in 37s, read: 864.0 MiB/s, write: 18.0 MiB/s ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: detected chunk with wrong digest. INFO: aborting backup job INFO: resuming VM again ERROR: Backup of VM 112 failed - backup write data failed: command error: write_data upload error: pipelined request failed: detected chunk with wrong digest. INFO: Failed at 2021-05-27 08:15:43 INFO: Backup job finished with errors TASK ERROR: job errors

PBS Log:
Code:
2021-05-27T08:15:05-05:00: starting new backup on datastore 'Home': "vm/112/2021-05-27T13:15:05Z" 2021-05-27T08:15:05-05:00: download 'index.json.blob' from previous backup. 2021-05-27T08:15:05-05:00: register chunks in 'drive-efidisk0.img.fidx' from previous backup. 2021-05-27T08:15:05-05:00: download 'drive-efidisk0.img.fidx' from previous backup. 2021-05-27T08:15:05-05:00: created new fixed index 1 ("vm/112/2021-05-27T13:15:05Z/drive-efidisk0.img.fidx") 2021-05-27T08:15:05-05:00: register chunks in 'drive-virtio0.img.fidx' from previous backup. 2021-05-27T08:15:05-05:00: download 'drive-virtio0.img.fidx' from previous backup. 2021-05-27T08:15:06-05:00: created new fixed index 2 ("vm/112/2021-05-27T13:15:05Z/drive-virtio0.img.fidx") 2021-05-27T08:15:06-05:00: add blob "/media/storage/home/vm/112/2021-05-27T13:15:05Z/qemu-server.conf.blob" (563 bytes, comp: 563) 2021-05-27T08:15:42-05:00: POST /fixed_chunk: 400 Bad Request: detected chunk with wrong digest. 2021-05-27T08:15:42-05:00: POST /fixed_chunk: 400 Bad Request: detected chunk with wrong digest. 2021-05-27T08:15:42-05:00: POST /fixed_chunk: 400 Bad Request: detected chunk with wrong digest. 2021-05-27T08:15:42-05:00: POST /fixed_chunk: 400 Bad Request: detected chunk with wrong digest. 2021-05-27T08:15:42-05:00: backup failed: connection error: Transport endpoint is not connected (os error 107) 2021-05-27T08:15:42-05:00: removing failed backup 2021-05-27T08:15:42-05:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: protocol error: stream no longer needed 2021-05-27T08:15:42-05:00: TASK ERROR: connection error: Transport endpoint is not connected (os error 107)
 
Last edited:
Also encountering problems when syncing between remote and destination PBS.

Remote:
Code:
May 28 09:04:00 pbs proxmox-backup-proxy[194]: TASK ERROR: connection error: Connection reset by peer (os error 104)

Destination:
Code:
2021-05-28T09:04:00-05:00: sync group vm/115 failed - detected chunk with wrong digest.
 
i am still waiting on the task log of the 'Read' task from the pbs where you tried to restore the vm...
 
also i would check your memory, constant chunk digest errors indicate that something calculates a wrong checksum (client or server)
 
  • Like
Reactions: jasonsansone
also i would check your memory, constant chunk digest errors indicate that something calculates a wrong checksum (client or server)
Thanks for tips. I will have to do another restore attempt and access logs from both ends for you. Regarding memory: I have problems regardless of the cluster I am restoring to. All nodes have 192GB ECC DDR4. I wasn’t aware of a good way to test ECC, but I am up for any recommendations.
 
Thanks for tips. I will have to do another restore attempt and access logs from both ends for you. Regarding memory: I have problems regardless of the cluster I am restoring to. All nodes have 192GB ECC DDR4. I wasn’t aware of a good way to test ECC, but I am up for any recommendations.
The boot media has a mem test I believe, when you install proxmox.

Or use memtest86 or memtest.org
 
  • Like
Reactions: jasonsansone

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!