Proxmox Backup Issue

JRNetwork

New Member
Mar 26, 2023
9
3
3
Hi Guys,
I had an issue with a backup of a VM, it wouldnt find the backups it had and complained that it couldnt do an incremental backup. I removed the backups within proxmox backup server and then the backups started again. This happened to multiple VMs no idea why.

All the backups started functioning again excluding VM 107. This VM is a web server and I need to ensure it gets backed up correctly, problem is everytime it backs up to 22% I get the following error:

ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: Overwriting existing (encrypted) chunk '27b54105e22dafe5ce4e0fd9936af313a33fa3b2806837537f4434bd29db52f5' on store 'JRCluster' is not allowed!

I've attempted garbage collection (after 24 hours and 5 minutes) How can I clear these chunks, the VM does not show within the list of backups now as I fully removed the backup from PBS.

I'm using PBS: 2.4-2. Any pointers would be amazing as I'm unsure apart from recreating the datastore on how to fix this issue.

Code:
INFO: Starting Backup of VM 107 (qemu)
INFO: Backup started at 2023-07-17 12:00:25
INFO: status = running
INFO: VM Name: Windu
INFO: include disk 'scsi0' 'local:107/vm-107-disk-0.qcow2' 150G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/107/2023-07-17T11:00:25Z'
INFO: enabling encryption
INFO: started backup task '1e3f87f6-ca8f-4987-9bc3-25cae8723d21'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO:   0% (132.0 MiB of 150.0 GiB) in 3s, read: 44.0 MiB/s, write: 44.0 MiB/s
INFO:   1% (1.6 GiB of 150.0 GiB) in 13s, read: 149.2 MiB/s, write: 80.8 MiB/s
INFO:   2% (3.2 GiB of 150.0 GiB) in 22s, read: 178.7 MiB/s, write: 178.7 MiB/s
INFO:   3% (4.6 GiB of 150.0 GiB) in 30s, read: 181.0 MiB/s, write: 181.0 MiB/s
INFO:   4% (6.0 GiB of 150.0 GiB) in 39s, read: 165.8 MiB/s, write: 165.8 MiB/s
INFO:   5% (8.9 GiB of 150.0 GiB) in 52s, read: 225.2 MiB/s, write: 109.8 MiB/s
INFO:   6% (9.8 GiB of 150.0 GiB) in 55s, read: 325.3 MiB/s, write: 72.0 MiB/s
INFO:   7% (10.5 GiB of 150.0 GiB) in 1m 11s, read: 43.0 MiB/s, write: 43.0 MiB/s
INFO:   8% (12.1 GiB of 150.0 GiB) in 1m 51s, read: 39.8 MiB/s, write: 39.8 MiB/s
INFO:   9% (13.5 GiB of 150.0 GiB) in 2m 20s, read: 51.7 MiB/s, write: 51.7 MiB/s
INFO:  10% (15.1 GiB of 150.0 GiB) in 2m 38s, read: 87.8 MiB/s, write: 87.8 MiB/s
INFO:  11% (16.5 GiB of 150.0 GiB) in 3m 5s, read: 54.2 MiB/s, write: 54.2 MiB/s
INFO:  12% (18.0 GiB of 150.0 GiB) in 3m 43s, read: 40.4 MiB/s, write: 40.4 MiB/s
INFO:  13% (19.5 GiB of 150.0 GiB) in 4m 23s, read: 38.6 MiB/s, write: 38.6 MiB/s
INFO:  14% (21.0 GiB of 150.0 GiB) in 5m, read: 41.4 MiB/s, write: 41.4 MiB/s
INFO:  15% (22.5 GiB of 150.0 GiB) in 5m 29s, read: 53.2 MiB/s, write: 53.2 MiB/s
INFO:  16% (24.0 GiB of 150.0 GiB) in 6m 1s, read: 48.1 MiB/s, write: 48.1 MiB/s
INFO:  17% (25.5 GiB of 150.0 GiB) in 6m 51s, read: 30.6 MiB/s, write: 30.6 MiB/s
INFO:  18% (27.0 GiB of 150.0 GiB) in 7m 13s, read: 70.7 MiB/s, write: 70.7 MiB/s
INFO:  19% (28.5 GiB of 150.0 GiB) in 7m 58s, read: 34.0 MiB/s, write: 34.0 MiB/s
INFO:  20% (30.0 GiB of 150.0 GiB) in 8m 46s, read: 31.8 MiB/s, write: 31.8 MiB/s
INFO:  21% (31.5 GiB of 150.0 GiB) in 9m 31s, read: 33.7 MiB/s, write: 33.7 MiB/s
INFO:  22% (33.0 GiB of 150.0 GiB) in 10m 19s, read: 32.1 MiB/s, write: 32.1 MiB/s
INFO:  22% (33.4 GiB of 150.0 GiB) in 10m 33s, read: 30.6 MiB/s, write: 30.6 MiB/s
ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: Overwriting existing (encrypted) chunk '27b54105e22dafe5ce4e0fd9936af313a33fa3b2806837537f4434bd29db52f5' on store 'JRCluster' is not allowed!
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 107 failed - backup write data failed: command error: write_data upload error: pipelined request failed: Overwriting existing (encrypted) chunk '27b54105e22dafe5ce4e0fd9936af313a33fa3b2806837537f4434bd29db52f5' on store 'JRCluster' is not allowed!
INFO: Failed at 2023-07-17 12:10:58
INFO: Backup job finished with errors
TASK ERROR: job errors
 
ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: Overwriting existing (encrypted) chunk '27b54105e22dafe5ce4e0fd9936af313a33fa3b2806837537f4434bd29db52f5' on store 'JRCluster' is not allowed!
this error basically means that the chunk on disk is different from the one that is tried to be uploaded, but their hash is the same. so either you just found a sha256 collision (*very* unlikely) or one of the chunks is corrupt
(my guess is the one already existing on disk?)

you can test the on disk chunk with the 'proxmox-backup-debug' tool namely with

Code:
proxmox-backup-debug inspect chunk /path/to/chunk

it will tell you if the CRC checksum is correct
if it is, you can also use the '--keyfile' and '--decode' options to try to decode the contentt and see if that works

if that all checks out, i guess the client sends some wrong data/chunk or miscalculates the checksum
 
  • Like
Reactions: JRNetwork
Heya,
I ran it and got this:
1689666632284.png

Would i be better to delete this chunk in the file system and try again ? Will this affect any other backups ?
 
Would i be better to delete this chunk in the file system and try again ? Will this affect any other backups ?
yes, as it seems the file is corrupted on disk...
if you want to be on the safe side you can also move the file away should you need it later
 
One other question on this, just to understand. Since I deleted all backups to do with this VM, why did garbage collection not clear up this chunk ?
 
One other question on this, just to understand. Since I deleted all backups to do with this VM, why did garbage collection not clear up this chunk ?
it is possible that the chunk is referenced by a backup of another vm/host
 
  • Like
Reactions: JRNetwork
Or you didn't wait the minimum of 24 hours and 5 minutes between deleting it and doing the GC. In that case the GC will just ignore these chunks.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!