[SOLVED] Backups failing

M

muzicman0

New Member
Oct 16, 2024
13
1
3
I have now tried 3 PC's, multiple RAM configurations, etc. I just don't know why my backups keep failing? PBS is installed on a m.2 SSD. I have 2 HDDs for backups, one WD Red Plus 10 TB that is new, and one older Seagate 5 TB. Both drives have the same types of errors. Happy to provide any logging required, but I am new to Proxmox, so I may need instructions on how to get them. I do know my way around Linux as a casual user though.

Any help is VERY much appreciated.

Here is an example of a backup that failed:

Code: 
INFO: starting new backup job: vzdump 100 --storage VM_Backups --mailto xxxx@yyyy.com --node hw-vhost03-srv --remove 0 --mode snapshot --notes-template '{{guestname}}' --notification-mode auto
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2024-11-03 16:01:42
INFO: status = running
INFO: VM Name: HW-UPS-MONITOR-VM
INFO: include disk 'sata0' 'data_6tb_2:vm-100-disk-3' 75G
INFO: include disk 'tpmstate0' 'data_6tb_2:vm-100-disk-5' 4M
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/100/2024-11-04T00:01:42Z'
INFO: attaching TPM drive to QEMU for backup
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'c9d0846c-097c-450d-bea6-500022738afa'
INFO: resuming VM again
INFO: sata0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: tpmstate0-backup: dirty-bitmap status: created new
INFO:   0% (748.0 MiB of 75.0 GiB) in 3s, read: 249.3 MiB/s, write: 245.3 MiB/s
INFO:   1% (972.0 MiB of 75.0 GiB) in 6s, read: 74.7 MiB/s, write: 74.7 MiB/s
INFO:   2% (1.5 GiB of 75.0 GiB) in 11s, read: 119.2 MiB/s, write: 119.2 MiB/s
INFO:   3% (2.3 GiB of 75.0 GiB) in 18s, read: 118.3 MiB/s, write: 117.1 MiB/s
INFO:   4% (3.1 GiB of 75.0 GiB) in 25s, read: 108.0 MiB/s, write: 108.0 MiB/s
INFO:   5% (3.8 GiB of 75.0 GiB) in 31s, read: 115.3 MiB/s, write: 115.3 MiB/s
INFO:   6% (4.5 GiB of 75.0 GiB) in 42s, read: 71.6 MiB/s, write: 71.6 MiB/s
INFO:   7% (5.3 GiB of 75.0 GiB) in 51s, read: 89.8 MiB/s, write: 89.8 MiB/s
INFO:   8% (6.1 GiB of 75.0 GiB) in 1m 7s, read: 47.8 MiB/s, write: 47.8 MiB/s
INFO:   9% (6.8 GiB of 75.0 GiB) in 1m 14s, read: 109.7 MiB/s, write: 109.7 MiB/s
INFO:  10% (7.5 GiB of 75.0 GiB) in 1m 28s, read: 51.1 MiB/s, write: 51.1 MiB/s
INFO:  11% (8.4 GiB of 75.0 GiB) in 1m 39s, read: 80.0 MiB/s, write: 80.0 MiB/s
INFO:  12% (9.1 GiB of 75.0 GiB) in 1m 45s, read: 124.0 MiB/s, write: 124.0 MiB/s
INFO:  13% (9.8 GiB of 75.0 GiB) in 1m 57s, read: 57.7 MiB/s, write: 57.7 MiB/s
INFO:  14% (10.6 GiB of 75.0 GiB) in 2m 9s, read: 69.0 MiB/s, write: 69.0 MiB/s
INFO:  15% (11.3 GiB of 75.0 GiB) in 2m 24s, read: 51.7 MiB/s, write: 51.7 MiB/s
INFO:  16% (12.1 GiB of 75.0 GiB) in 2m 30s, read: 134.0 MiB/s, write: 120.0 MiB/s
INFO:  17% (12.8 GiB of 75.0 GiB) in 2m 44s, read: 49.7 MiB/s, write: 49.7 MiB/s
INFO:  18% (13.5 GiB of 75.0 GiB) in 2m 52s, read: 90.5 MiB/s, write: 89.5 MiB/s
INFO:  19% (14.4 GiB of 75.0 GiB) in 3m 5s, read: 69.2 MiB/s, write: 69.2 MiB/s
INFO:  20% (15.0 GiB of 75.0 GiB) in 3m 11s, read: 108.0 MiB/s, write: 108.0 MiB/s
INFO:  21% (15.9 GiB of 75.0 GiB) in 3m 26s, read: 57.1 MiB/s, write: 57.1 MiB/s
INFO:  22% (16.5 GiB of 75.0 GiB) in 3m 36s, read: 68.4 MiB/s, write: 68.4 MiB/s
INFO:  23% (17.3 GiB of 75.0 GiB) in 3m 44s, read: 104.5 MiB/s, write: 104.5 MiB/s
INFO:  24% (18.1 GiB of 75.0 GiB) in 4m, read: 45.5 MiB/s, write: 45.5 MiB/s
INFO:  25% (18.8 GiB of 75.0 GiB) in 4m 6s, read: 126.0 MiB/s, write: 126.0 MiB/s
INFO:  26% (19.6 GiB of 75.0 GiB) in 4m 21s, read: 53.3 MiB/s, write: 53.3 MiB/s
INFO:  27% (20.3 GiB of 75.0 GiB) in 4m 27s, read: 126.0 MiB/s, write: 126.0 MiB/s
INFO:  28% (21.1 GiB of 75.0 GiB) in 4m 45s, read: 46.2 MiB/s, write: 46.2 MiB/s
INFO:  29% (21.8 GiB of 75.0 GiB) in 4m 51s, read: 117.3 MiB/s, write: 117.3 MiB/s
INFO:  30% (22.6 GiB of 75.0 GiB) in 5m 6s, read: 55.5 MiB/s, write: 55.5 MiB/s
INFO:  31% (23.3 GiB of 75.0 GiB) in 5m 17s, read: 62.9 MiB/s, write: 62.9 MiB/s
INFO:  32% (24.1 GiB of 75.0 GiB) in 5m 28s, read: 77.5 MiB/s, write: 77.5 MiB/s
INFO:  33% (24.8 GiB of 75.0 GiB) in 5m 33s, read: 130.4 MiB/s, write: 130.4 MiB/s
INFO:  34% (25.6 GiB of 75.0 GiB) in 5m 53s, read: 42.6 MiB/s, write: 42.6 MiB/s
INFO:  35% (26.4 GiB of 75.0 GiB) in 6m 7s, read: 58.9 MiB/s, write: 58.9 MiB/s
INFO:  36% (27.1 GiB of 75.0 GiB) in 6m 13s, read: 124.7 MiB/s, write: 124.7 MiB/s
INFO:  37% (27.9 GiB of 75.0 GiB) in 6m 32s, read: 38.7 MiB/s, write: 38.7 MiB/s
INFO:  38% (28.6 GiB of 75.0 GiB) in 6m 38s, read: 122.7 MiB/s, write: 122.7 MiB/s
INFO:  39% (29.3 GiB of 75.0 GiB) in 7m 7s, read: 25.5 MiB/s, write: 25.5 MiB/s
INFO:  40% (30.0 GiB of 75.0 GiB) in 7m 16s, read: 82.7 MiB/s, write: 62.2 MiB/s
INFO:  40% (30.6 GiB of 75.0 GiB) in 7m 51s, read: 15.5 MiB/s, write: 15.5 MiB/s
ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: detected chunk with wrong digest.
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 100 failed - backup write data failed: command error: write_data upload error: pipelined request failed: detected chunk with wrong digest.
INFO: Failed at 2024-11-03 16:09:42
INFO: Backup job finished with errors
TASK ERROR: job errors

Here is the log from the PBE side:


Code: 
2024-11-03T16:01:47-08:00: starting new backup on datastore 'VM_Backups' from ::ffff:10.1.10.223: "vm/100/2024-11-04T00:01:42Z"
2024-11-03T16:01:47-08:00: GET /previous: 400 Bad Request: no valid previous backup
2024-11-03T16:01:47-08:00: created new fixed index 1 ("vm/100/2024-11-04T00:01:42Z/drive-sata0.img.fidx")
2024-11-03T16:01:47-08:00: created new fixed index 2 ("vm/100/2024-11-04T00:01:42Z/drive-tpmstate0-backup.img.fidx")
2024-11-03T16:01:47-08:00: add blob "/media/VM_Backups/vm/100/2024-11-04T00:01:42Z/qemu-server.conf.blob" (440 bytes, comp: 440)
2024-11-03T16:09:08-08:00: POST /fixed_chunk: 400 Bad Request: detected chunk with wrong digest.
2024-11-03T16:09:41-08:00: backup ended and finish failed: backup ended but finished flag is not set.
2024-11-03T16:09:41-08:00: removing unfinished backup
2024-11-03T16:09:41-08:00: TASK ERROR: backup ended but finished flag is not set.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:41-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:42-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:43-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:43-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:43-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:43-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:43-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:43-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:43-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:43-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:43-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:43-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
2024-11-03T16:09:43-08:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.

EDIT: Just a quick clarification. I also set up a CIFS backup location and it works fine.
 
Last edited:
that sounds like broken memory either on the client or server side..
 
  • Like
Reactions: muzicman0
Looks like it was memory on the server side. Never occurred to me that it could be on that side since I wasn't having problems with the VMs, and I was able to back up over CIFS. It was a cheap set of RAM modules that I bought, that are brand new. As soon as I pulled those, backups seem to work now. I will test my large 4.5 TB backup and see if it also succeeds.

UPDATE: Willing to call it solved at this point. Never made it past 1% on the large backup. I am now at 10%. I ordered a new set of (better) RAM so I can get the server back up to 64 GB. Thanks for the help!
 
Last edited:
  • Like
Reactions: fabian
since the digest is calculated at the client side and re-verified at the server side (with additional checksums on the transport layer that should prevent the data being corrupted in flight on the wire), a digest mismatch is almost always memory on either end :)
 
  • Like
Reactions: Johannes S
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back