[SOLVED] restore failed: detected chunk with wrong digest. / TASK ERROR: connection error: unexpected EOF

[max1337]

Hi everyone,
I had to recreate a zfs pool after it went corrupt and my somewhat naive me thought that I would be able to restore all the VMs from our proxmox Backup Server 4.2.4 in no time =)
I was able to restore most of the VMs even though not all could be restored from the most recent backups (because of the issue I am going to describe here). However one of the VMs has four different backups I tried to use for restore but none of the backups worked. All of them crashed during the restore process at various percentages with the following error message on the PVE side (9.2.6)

[...]
progress 32% (read 34359738368 bytes, zeroes = 13% (4588568576 bytes), duration 116 sec)
restore failed: detected chunk with wrong digest.
temporary volume 'zfs_vms:vm-123-disk-1' successfully removed
temporary volume 'zfs_vms:vm-123-disk-0' successfully removed
error before or during data restore, some or all disks were not completely restored. VM 123 state is NOT cleaned up.
TASK ERROR: command '/usr/bin/pbs-restore --repository root@pam@10.241.1.112:zfs vm/123/2026-06-06T23:05:21Z drive-scsi0.img.fidx /dev/zvol/zfs/VMs/vm-123-disk-1 --verbose --format raw --skip-zero' failed: exit code 255

on the PBS side I see

Aug 06 15:25:02 pbs proxmox-backup-proxy[1173]: TASK ERROR: connection error: unexpected EOF

From my undestanding, this indicates a backup file that is corrupted. But I ran the verification on all backups for said machine and they came back with no errors.
Any idea what I can do to restore the machine?

Thanks a lot
Max

 

[j.theisen]

Hi @max1337

thanks for posting on the forum!

From the past "detected chunk with wrong digest" errors especially if encountered at different parts of the backup / restore, mostly indicate a problem with the RAM of the PBS.
Can you check that with a live medium f.ex.?

Best regards
Jonas

EDIT: Add references:

J

Thread 'Backup fails - detected chunk with wrong digest'

Sep 15, 2023

Hello,

i have following setup:

Proxmox Server with some VMs (64GB ECC UDIMM RAM, 8 Core CPU, ZFS Storage Pool) (own dedicated server in datacenter)
Proxmox Backup Server (64GB RAM, ZFS Storage Pool) (rented dedicated server from hetzner)

I have connected the Proxmox Backup Server as storage on the Proxmox Server and created a Backup schedule, each day at 3 am.
The first week it went fine and everything was backuped. But recently it started to fail backupping the vms.
This is my Proxmox Server Backup-Task Log:

INFO: starting new backup job: vzdump --storage de_backup...

• jstrauss
• backup detected chunk with wrong digest error hetzner
• Replies: 2
• Forum: Proxmox Backup: Installation and configuration

P

Thread 'Backups schlagen häufig wegen falschem Digest fehl'

May 29, 2024

Hallo,

ich habe bei unserem PBS das Problem, dass die Backups häufig, aber nicht immer mit folgendem Fehler fehlschlagen.

ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: detected chunk with wrong digest.

Ich weiß nicht genau, wie ich so etwas debuggen kann. Falls ihr hier mehr Hinweise zu habt, wäre euch sehr verbunden.

Das System ist eine Neuinstallation des PBS, das System ist unten aufgeführt.

Spoiler: Hardware

CPU: 8 x Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz (1 Socket)
Kernel Version: Linux 6.8.4-3-pve (2024-05-02T11:55Z)
RAM: 64GB ECC...

• Poehlmann
• detected chunk with wrong digest error failed
• Replies: 3
• Forum: Proxmox Backup Server (Deutsch/German)

 

[max1337]

Hi Jonas,
thank you very much for pointing me in that direction. I don't know why I did not find these threads when I was searching the forums.
I ran memtest86+ and it found issues right away. I will start to change memory modules until I found the broken one. Or do you think this could be an issue with the memory controller on the CPU side, too?

Cheers and have a nice weekend
Max

 

[max1337]

Hello everyone,
as a final update on this: memtest found errors with the RAM of the system. I started to unplug modules until I figured out which module was the culprit and replaced it. Ran memtest again with the system fully configured and was happy to see that the issue was solved. Hope this helps someone else at a later point.

Kind regards
Max