Backup fails - detected chunk with wrong digest

[jstrauss]

Hello,

i have following setup:

Proxmox Server with some VMs (64GB ECC UDIMM RAM, 8 Core CPU, ZFS Storage Pool) (own dedicated server in datacenter)
Proxmox Backup Server (64GB RAM, ZFS Storage Pool) (rented dedicated server from hetzner)

I have connected the Proxmox Backup Server as storage on the Proxmox Server and created a Backup schedule, each day at 3 am.
The first week it went fine and everything was backuped. But recently it started to fail backupping the vms.
This is my Proxmox Server Backup-Task Log:

INFO: starting new backup job: vzdump --storage de_backup --notes-template '{{guestname}}' --mode snapshot --prune-backups 'keep-last=10' --quiet 1 --mailnotification always --mailto office@jstrauss.at --all 1
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2023-09-15 02:00:02
INFO: status = running
INFO: VM Name: jstrauss
INFO: include disk 'scsi0' 'pool:vm-100-disk-0' 923852M
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/100/2023-09-15T00:00:02Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '27926b8d-9df5-492a-a7f7-6352dc1dce70'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: OK (396.7 GiB of 902.2 GiB dirty)
INFO: using fast incremental mode (dirty-bitmap), 396.7 GiB dirty of 902.2 GiB total
INFO:   0% (144.0 MiB of 396.7 GiB) in 3s, read: 48.0 MiB/s, write: 46.7 MiB/s
INFO:   1% (4.0 GiB of 396.7 GiB) in 1m 38s, read: 41.3 MiB/s, write: 38.7 MiB/s
INFO:   2% (7.9 GiB of 396.7 GiB) in 2m 48s, read: 58.2 MiB/s, write: 57.9 MiB/s
INFO:   3% (12.0 GiB of 396.7 GiB) in 4m 8s, read: 51.5 MiB/s, write: 51.3 MiB/s
INFO:   4% (15.9 GiB of 396.7 GiB) in 5m 19s, read: 56.3 MiB/s, write: 48.2 MiB/s
INFO:   5% (19.9 GiB of 396.7 GiB) in 7m 8s, read: 37.6 MiB/s, write: 37.6 MiB/s
INFO:   6% (23.9 GiB of 396.7 GiB) in 8m 31s, read: 49.0 MiB/s, write: 47.2 MiB/s
INFO:   7% (27.8 GiB of 396.7 GiB) in 9m 56s, read: 47.3 MiB/s, write: 42.4 MiB/s
INFO:   8% (31.8 GiB of 396.7 GiB) in 11m 1s, read: 62.8 MiB/s, write: 62.8 MiB/s
INFO:   9% (35.8 GiB of 396.7 GiB) in 11m 59s, read: 71.2 MiB/s, write: 70.9 MiB/s
INFO:  10% (39.7 GiB of 396.7 GiB) in 13m 9s, read: 57.3 MiB/s, write: 54.7 MiB/s
INFO:  11% (43.7 GiB of 396.7 GiB) in 14m 20s, read: 56.8 MiB/s, write: 55.8 MiB/s
INFO:  11% (44.3 GiB of 396.7 GiB) in 14m 30s, read: 65.2 MiB/s, write: 65.2 MiB/s
ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: detected chunk with wrong digest.
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 100 failed - backup write data failed: command error: write_data upload error: pipelined request failed: detected chunk with wrong digest.
INFO: Failed at 2023-09-15 02:14:32
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2023-09-15 02:14:32
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: test
INFO: include disk 'scsi0' 'pool:vm-101-disk-0' 2252M
INFO: creating Proxmox Backup Server archive 'vm/101/2023-09-15T00:14:32Z'
INFO: starting kvm to execute backup task
generating cloud-init ISO
INFO: started backup task '309eb494-bb60-49f5-94a3-3efaed635a7d'
INFO: scsi0: dirty-bitmap status: created new
INFO:  17% (396.0 MiB of 2.2 GiB) in 3s, read: 132.0 MiB/s, write: 0 B/s
INFO:  31% (712.0 MiB of 2.2 GiB) in 6s, read: 105.3 MiB/s, write: 0 B/s
INFO:  47% (1.0 GiB of 2.2 GiB) in 9s, read: 120.0 MiB/s, write: 0 B/s
INFO:  50% (1.1 GiB of 2.2 GiB) in 12s, read: 21.3 MiB/s, write: 0 B/s
INFO:  52% (1.1 GiB of 2.2 GiB) in 15s, read: 12.0 MiB/s, write: 0 B/s
INFO:  58% (1.3 GiB of 2.2 GiB) in 18s, read: 49.3 MiB/s, write: 0 B/s
INFO:  76% (1.7 GiB of 2.2 GiB) in 21s, read: 130.7 MiB/s, write: 0 B/s
INFO:  93% (2.1 GiB of 2.2 GiB) in 24s, read: 130.7 MiB/s, write: 0 B/s
INFO: 100% (2.2 GiB of 2.2 GiB) in 27s, read: 49.3 MiB/s, write: 0 B/s
INFO: backup is sparse: 132.00 MiB (5%) total zero data
INFO: backup was done incrementally, reused 2.20 GiB (100%)
INFO: transferred 2.20 GiB in 27 seconds (83.4 MiB/s)
INFO: stopping kvm after backup task
INFO: adding notes to backup
INFO: prune older backups with retention: keep-last=10
INFO: running 'proxmox-backup-client prune' for 'vm/101'
INFO: pruned 1 backup(s) not covered by keep-retention policy
INFO: Finished Backup of VM 101 (00:00:34)
INFO: Backup finished at 2023-09-15 02:15:06
INFO: Backup job finished with errors
TASK ERROR: job errors

And on the Proxmox Backup Server following is in the log:

2023-09-15T02:00:02+02:00: starting new backup on datastore 'poolBackup': "vm/100/2023-09-15T00:00:02Z"
2023-09-15T02:00:02+02:00: download 'index.json.blob' from previous backup.
2023-09-15T02:00:02+02:00: register chunks in 'drive-scsi0.img.fidx' from previous backup.
2023-09-15T02:00:02+02:00: download 'drive-scsi0.img.fidx' from previous backup.
2023-09-15T02:00:02+02:00: created new fixed index 1 ("vm/100/2023-09-15T00:00:02Z/drive-scsi0.img.fidx")
2023-09-15T02:00:02+02:00: add blob "/mnt/datastore/poolBackup/vm/100/2023-09-15T00:00:02Z/qemu-server.conf.blob" (470 bytes, comp: 470)
2023-09-15T02:14:30+02:00: POST /fixed_chunk: 400 Bad Request: detected chunk with wrong digest.
2023-09-15T02:14:31+02:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: stream error received: stream no longer needed
2023-09-15T02:14:32+02:00: backup ended and finish failed: backup ended but finished flag is not set.
2023-09-15T02:14:32+02:00: removing unfinished backup
2023-09-15T02:14:32+02:00: TASK ERROR: backup ended but finished flag is not set.

I searched at the web but the only thing I found was maybe the RAM has an error, but why does it maybe 15 times work and then suddenly not again.
Thanks for help.

 

[dcsapak]

I searched at the web but the only thing I found was maybe the RAM has an error, but why does it maybe 15 times work and then suddenly not again.

thats exactly the type of behaviour wrong RAM would show, since the different attempts will use different parts of the memory (since there is also other stuff going on.

i'd suggest doing a memory test