Backups randomly fail

[pengelstein]

Hi everyone,

We have some backups that are sometimes failing, with some rather large VMs. In the Proxmox VE logs, we see the following logs:

831: 2024-04-12 04:32:07 ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: detected chunk with wrong digest.
831: 2024-04-12 04:32:07 INFO: aborting backup job
831: 2024-04-12 04:32:07 INFO: resuming VM again
831: 2024-04-12 04:32:07 ERROR: Backup of VM 831 failed - backup write data failed: command error: write_data upload error: pipelined request failed: detected chunk with wrong digest.

In the backup server, we have the following corresponding logs:

2024-04-12T04:30:10+02:00: add blob "/mnt/datastore/PBS_DS/vm/831/2024-04-12T02:30:08Z/qemu-server.conf.blob" (438 bytes, comp: 438)
2024-04-12T04:32:04+02:00: POST /fixed_chunk: 400 Bad Request: detected chunk with wrong digest.
2024-04-12T04:32:04+02:00: POST /fixed_chunk: 400 Bad Request: detected chunk with wrong digest.
2024-04-12T04:32:06+02:00: backup ended and finish failed: backup ended but finished flag is not set.
2024-04-12T04:32:06+02:00: removing unfinished backup
2024-04-12T04:32:06+02:00: TASK ERROR: backup ended but finished flag is not set.
2024-04-12T04:32:06+02:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.

I don't really see what can I do to diagnose the issue. I saw on some other threads that it might be a memory or disk issue, but the ram in the server is quite new and we don't see wearout on the disks.

Thanks!

 

[ggoller]

Hi!
please do a memtest anyway, so that we can rule out a ram issue!
If a 'memtest86' entry is not already present in the grub menu, add it like this: https://forum.proxmox.com/threads/memtest86.114820/#post-496486