Encrypted backup of only one VM fails, but non-encrypted works fine

[curruscanis]

I have an odd situation with a Proxmox Backup Server backing up VM's from a 3 node cluster, cluster is all at current PVE version 9.2.3, and the PBS system is on a physical host also current version of PBS 4.2.1.

I have setup a backup storage on the cluster using the PBS server and if I have the backup storage set as non-encrypted the backup of the two VM's on the cluster backup fine. VM 101 is a Windows 2022 server, vm 103 with a Windows 11 OS. However, if I change the storage backup to encrypted and use a proper encryption key the back of the VM-103 fails at 9% every time. I have deleted all backups / run Garbage Collection / waited longer than 24 hours and 5 minutes, and many other tasks without success. The encrypted backups of the VM-101 work just fine, only the VM-103 does not work encrypted.

Another Cluster also uses the PBS and backs up many VM's all to an encrypted name space just fine. So it is just this one VM that seems to have issues.

I have tried cloning the VM and running backups tests, exactly the same issues occur, and fail at exacly 9% just like the original, but it also only fails if encrypted.

The VM-103 is a UEFI boot with a EFI DISK, TPM disk, and boot Disk, storage is on a CEPH cluster accross the three cluster nodes.

The backup directories all have proper permissions...

Below are the backup logs - first the failed encrypted backup log, second the successful non-encrypted backup:

------------------------------------------------

vzdump 103 --mode snapshot --notification-mode notification-system --storage YSC-DT-PBS --remove 0 --node MACC-PVE-3 --notes-template '{{guestname}}'

103: 2026-06-07 13:30:42 INFO: Starting Backup of VM 103 (qemu)
103: 2026-06-07 13:30:42 INFO: status = running
103: 2026-06-07 13:30:42 INFO: VM Name: v-FAX
103: 2026-06-07 13:30:42 INFO: include disk 'scsi0' 'CEPH-VM:vm-103-disk-1' 100G
103: 2026-06-07 13:30:42 INFO: include disk 'efidisk0' 'CEPH-VM:vm-103-disk-0' 528K
103: 2026-06-07 13:30:42 INFO: include disk 'tpmstate0' 'CEPH-VM:vm-103-disk-2' 4M
103: 2026-06-07 13:30:42 INFO: backup mode: snapshot
103: 2026-06-07 13:30:42 INFO: ionice priority: 7
103: 2026-06-07 13:30:42 INFO: creating Proxmox Backup Server archive 'vm/103/2026-06-07T19:30:42Z'
103: 2026-06-07 13:30:42 INFO: enabling encryption
103: 2026-06-07 13:30:42 INFO: attaching TPM drive to QEMU for backup
103: 2026-06-07 13:30:42 INFO: issuing guest-agent 'fs-freeze' command
103: 2026-06-07 13:30:46 INFO: starting backup via QMP command
103: 2026-06-07 13:30:47 INFO: issuing guest-agent 'fs-thaw' command
103: 2026-06-07 13:30:47 INFO: started backup task 'f63a02d6-ac3a-455d-8a61-95d47a356664'
103: 2026-06-07 13:30:47 INFO: resuming VM again
103: 2026-06-07 13:30:47 INFO: efidisk0: dirty-bitmap status: existing bitmap was invalid and has been cleared
103: 2026-06-07 13:30:47 INFO: scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared
103: 2026-06-07 13:30:47 INFO: tpmstate0-backup: dirty-bitmap status: created new
103: 2026-06-07 13:30:50 INFO: 0% (352.0 MiB of 100.0 GiB) in 3s, read: 117.3 MiB/s, write: 62.7 MiB/s
103: 2026-06-07 13:31:22 INFO: 1% (1.0 GiB of 100.0 GiB) in 35s, read: 21.8 MiB/s, write: 21.8 MiB/s
103: 2026-06-07 13:32:10 INFO: 2% (2.1 GiB of 100.0 GiB) in 1m 23s, read: 21.9 MiB/s, write: 21.9 MiB/s
103: 2026-06-07 13:32:57 INFO: 3% (3.0 GiB of 100.0 GiB) in 2m 10s, read: 21.6 MiB/s, write: 21.6 MiB/s
103: 2026-06-07 13:33:44 INFO: 4% (4.0 GiB of 100.0 GiB) in 2m 57s, read: 21.0 MiB/s, write: 21.0 MiB/s
103: 2026-06-07 13:34:31 INFO: 5% (5.1 GiB of 100.0 GiB) in 3m 44s, read: 22.7 MiB/s, write: 22.7 MiB/s
103: 2026-06-07 13:35:16 INFO: 6% (6.0 GiB of 100.0 GiB) in 4m 29s, read: 22.0 MiB/s, write: 22.0 MiB/s
103: 2026-06-07 13:36:04 INFO: 7% (7.1 GiB of 100.0 GiB) in 5m 17s, read: 22.2 MiB/s, write: 22.2 MiB/s
103: 2026-06-07 13:36:50 INFO: 8% (8.1 GiB of 100.0 GiB) in 6m 3s, read: 22.1 MiB/s, write: 22.1 MiB/s
103: 2026-06-07 13:37:36 INFO: 9% (9.0 GiB of 100.0 GiB) in 6m 49s, read: 22.0 MiB/s, write: 22.0 MiB/s
103: 2026-06-07 13:37:48 INFO: 9% (9.2 GiB of 100.0 GiB) in 7m 1s, read: 17.3 MiB/s, write: 17.3 MiB/s
103: 2026-06-07 13:37:48 ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: Overwriting existing (encrypted) chunk '552bef595b29e12e35478064ddf764383b8f593dadc606df538dbed7bd0407df' on store 'YSC-DT-Backup' is not allowed!
103: 2026-06-07 13:37:48 INFO: aborting backup job
103: 2026-06-07 13:37:48 INFO: resuming VM again
103: 2026-06-07 13:37:48 ERROR: Backup of VM 103 failed - backup write data failed: command error: write_data upload error: pipelined request failed: Overwriting existing (encrypted) chunk '552bef595b29e12e35478064ddf764383b8f593dadc606df538dbed7bd0407df' on store 'YSC-DT-Backup' is not allowed!


The backup when not encrypted logs:

Logs​

vzdump 103 --notification-mode notification-system --mode snapshot --notes-template '{{guestname}}' --node MACC-PVE-3 --storage YSC-DT-PBS --remove 0

103: 2026-06-07 11:22:54 INFO: Starting Backup of VM 103 (qemu)
103: 2026-06-07 11:22:54 INFO: status = running
103: 2026-06-07 11:22:54 INFO: VM Name: v-FAX
103: 2026-06-07 11:22:54 INFO: include disk 'scsi0' 'CEPH-VM:vm-103-disk-1' 100G
103: 2026-06-07 11:22:54 INFO: include disk 'efidisk0' 'CEPH-VM:vm-103-disk-0' 528K
103: 2026-06-07 11:22:54 INFO: include disk 'tpmstate0' 'CEPH-VM:vm-103-disk-2' 4M
103: 2026-06-07 11:22:54 INFO: backup mode: snapshot
103: 2026-06-07 11:22:54 INFO: ionice priority: 7
103: 2026-06-07 11:22:54 INFO: creating Proxmox Backup Server archive 'vm/103/2026-06-07T17:22:54Z'
103: 2026-06-07 11:22:54 INFO: attaching TPM drive to QEMU for backup
103: 2026-06-07 11:22:54 INFO: issuing guest-agent 'fs-freeze' command
103: 2026-06-07 11:22:58 INFO: starting backup via QMP command
103: 2026-06-07 11:22:58 INFO: issuing guest-agent 'fs-thaw' command
103: 2026-06-07 11:22:58 INFO: started backup task 'ed5c4257-37c1-450f-82e9-b8524ddcc804'
103: 2026-06-07 11:22:58 INFO: resuming VM again
103: 2026-06-07 11:22:58 INFO: efidisk0: dirty-bitmap status: created new
103: 2026-06-07 11:22:58 INFO: scsi0: dirty-bitmap status: created new
103: 2026-06-07 11:22:58 INFO: tpmstate0-backup: dirty-bitmap status: created new
103: 2026-06-07 11:23:01 INFO: 0% (400.0 MiB of 100.0 GiB) in 3s, read: 133.3 MiB/s, write: 78.7 MiB/s
103: 2026-06-07 11:23:35 INFO: 1% (1.1 GiB of 100.0 GiB) in 37s, read: 20.5 MiB/s, write: 20.5 MiB/s
103: 2026-06-07 11:24:24 INFO: 2% (2.1 GiB of 100.0 GiB) in 1m 26s, read: 21.0 MiB/s, write: 21.0 MiB/s
103: 2026-06-07 11:25:11 INFO: 3% (3.0 GiB of 100.0 GiB) in 2m 13s, read: 21.1 MiB/s, write: 21.1 MiB/s
103: 2026-06-07 11:25:58 INFO: 4% (4.0 GiB of 100.0 GiB) in 3m, read: 21.3 MiB/s, write: 21.3 MiB/s
103: 2026-06-07 11:26:46 INFO: 5% (5.0 GiB of 100.0 GiB) in 3m 48s, read: 21.4 MiB/s, write: 21.4 MiB/s
103: 2026-06-07 11:27:35 INFO: 6% (6.0 GiB of 100.0 GiB) in 4m 37s, read: 21.3 MiB/s, write: 21.3 MiB/s
103: 2026-06-07 11:28:25 INFO: 7% (7.1 GiB of 100.0 GiB) in 5m 27s, read: 20.6 MiB/s, write: 20.6 MiB/s
103: 2026-06-07 11:29:14 INFO: 8% (8.1 GiB of 100.0 GiB) in 6m 16s, read: 20.9 MiB/s, write: 20.9 MiB/s
103: 2026-06-07 11:30:03 INFO: 9% (9.1 GiB of 100.0 GiB) in 7m 5s, read: 20.9 MiB/s, write: 20.9 MiB/s
103: 2026-06-07 11:30:51 INFO: 10% (10.0 GiB of 100.0 GiB) in 7m 53s, read: 20.8 MiB/s, write: 20.8 MiB/s
103: 2026-06-07 11:31:41 INFO: 11% (11.0 GiB of 100.0 GiB) in 8m 43s, read: 20.4 MiB/s, write: 20.4 MiB/s
103: 2026-06-07 11:32:30 INFO: 12% (12.0 GiB of 100.0 GiB) in 9m 32s, read: 21.1 MiB/s, write: 21.1 MiB/s
103: 2026-06-07 11:33:20 INFO: 13% (13.0 GiB of 100.0 GiB) in 10m 22s, read: 20.0 MiB/s, write: 20.0 MiB/s
103: 2026-06-07 11:34:34 INFO: 14% (14.0 GiB of 100.0 GiB) in 11m 36s, read: 14.3 MiB/s, write: 14.3 MiB/s
103: 2026-06-07 11:36:39 INFO: 15% (15.0 GiB of 100.0 GiB) in 13m 41s, read: 8.0 MiB/s, write: 8.0 MiB/s
103: 2026-06-07 11:39:01 INFO: 16% (16.0 GiB of 100.0 GiB) in 16m 3s, read: 7.2 MiB/s, write: 7.2 MiB/s
103: 2026-06-07 11:41:04 INFO: 17% (17.0 GiB of 100.0 GiB) in 18m 6s, read: 8.3 MiB/s, write: 8.3 MiB/s
103: 2026-06-07 11:43:30 INFO: 18% (18.0 GiB of 100.0 GiB) in 20m 32s, read: 7.0 MiB/s, write: 7.0 MiB/s
103: 2026-06-07 11:45:38 INFO: 19% (19.0 GiB of 100.0 GiB) in 22m 40s, read: 8.0 MiB/s, write: 8.0 MiB/s
103: 2026-06-07 11:47:42 INFO: 20% (20.0 GiB of 100.0 GiB) in 24m 44s, read: 8.2 MiB/s, write: 8.2 MiB/s
103: 2026-06-07 11:49:48 INFO: 21% (21.0 GiB of 100.0 GiB) in 26m 50s, read: 8.2 MiB/s, write: 8.2 MiB/s
103: 2026-06-07 11:51:49 INFO: 22% (22.0 GiB of 100.0 GiB) in 28m 51s, read: 8.5 MiB/s, write: 8.5 MiB/s
103: 2026-06-07 11:53:45 INFO: 23% (23.0 GiB of 100.0 GiB) in 30m 47s, read: 8.8 MiB/s, write: 8.8 MiB/s
103: 2026-06-07 11:58:00 INFO: 24% (24.0 GiB of 100.0 GiB) in 35m 2s, read: 4.0 MiB/s, write: 4.0 MiB/s
103: 2026-06-07 11:59:54 INFO: 25% (25.0 GiB of 100.0 GiB) in 36m 56s, read: 9.0 MiB/s, write: 9.0 MiB/s
103: 2026-06-07 12:01:35 INFO: 26% (26.0 GiB of 100.0 GiB) in 38m 37s, read: 10.1 MiB/s, write: 10.1 MiB/s
103: 2026-06-07 12:03:54 INFO: 27% (27.0 GiB of 100.0 GiB) in 40m 56s, read: 7.3 MiB/s, write: 7.3 MiB/s
103: 2026-06-07 12:05:58 INFO: 28% (28.0 GiB of 100.0 GiB) in 43m, read: 8.3 MiB/s, write: 8.3 MiB/s
103: 2026-06-07 12:08:00 INFO: 29% (29.0 GiB of 100.0 GiB) in 45m 2s, read: 8.4 MiB/s, write: 8.4 MiB/s
103: 2026-06-07 12:10:06 INFO: 30% (30.0 GiB of 100.0 GiB) in 47m 8s, read: 8.2 MiB/s, write: 8.2 MiB/s
103: 2026-06-07 12:12:12 INFO: 31% (31.0 GiB of 100.0 GiB) in 49m 14s, read: 8.1 MiB/s, write: 8.1 MiB/s
103: 2026-06-07 12:14:07 INFO: 32% (32.0 GiB of 100.0 GiB) in 51m 9s, read: 8.9 MiB/s, write: 8.9 MiB/s
103: 2026-06-07 12:15:49 INFO: 33% (33.0 GiB of 100.0 GiB) in 52m 51s, read: 10.0 MiB/s, write: 10.0 MiB/s
103: 2026-06-07 12:19:16 INFO: 34% (34.0 GiB of 100.0 GiB) in 56m 18s, read: 4.9 MiB/s, write: 4.9 MiB/s
103: 2026-06-07 12:21:27 INFO: 35% (35.0 GiB of 100.0 GiB) in 58m 29s, read: 7.9 MiB/s, write: 7.9 MiB/s
103: 2026-06-07 12:23:36 INFO: 36% (36.0 GiB of 100.0 GiB) in 1h 38s, read: 7.9 MiB/s, write: 7.9 MiB/s
103: 2026-06-07 12:25:19 INFO: 37% (37.0 GiB of 100.0 GiB) in 1h 2m 21s, read: 10.0 MiB/s, write: 10.0 MiB/s
103: 2026-06-07 12:25:55 INFO: 38% (38.0 GiB of 100.0 GiB) in 1h 2m 57s, read: 28.8 MiB/s, write: 28.6 MiB/s
103: 2026-06-07 12:27:05 INFO: 39% (39.0 GiB of 100.0 GiB) in 1h 4m 7s, read: 14.4 MiB/s, write: 12.6 MiB/s
103: 2026-06-07 12:28:47 INFO: 44% (44.4 GiB of 100.0 GiB) in 1h 5m 49s, read: 54.5 MiB/s, write: 5.7 MiB/s
103: 2026-06-07 12:28:50 INFO: 68% (68.1 GiB of 100.0 GiB) in 1h 5m 52s, read: 7.9 GiB/s, write: 0 B/s
103: 2026-06-07 12:28:53 INFO: 93% (93.9 GiB of 100.0 GiB) in 1h 5m 55s, read: 8.6 GiB/s, write: 0 B/s
103: 2026-06-07 12:28:56 INFO: 99% (99.4 GiB of 100.0 GiB) in 1h 5m 58s, read: 1.8 GiB/s, write: 26.7 MiB/s
103: 2026-06-07 12:29:26 INFO: 100% (100.0 GiB of 100.0 GiB) in 1h 6m 28s, read: 21.9 MiB/s, write: 20.6 MiB/s
103: 2026-06-07 13:18:48 INFO: backup is sparse: 60.04 GiB (60%) total zero data
103: 2026-06-07 13:18:48 INFO: backup was done incrementally, reused 60.04 GiB (60%)
103: 2026-06-07 13:18:48 INFO: transferred 100.00 GiB in 6950 seconds (14.7 MiB/s)
103: 2026-06-07 13:18:48 INFO: adding notes to backup
103: 2026-06-07 13:18:49 INFO: Finished Backup of VM 103 (01:55:55)

 

[UdoB]

103: 2026-06-07 13:37:48 ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: Overwriting existing (encrypted) chunk '552bef595b29e12e35478064ddf764383b8f593dadc606df538dbed7bd0407df' on store 'YSC-DT-Backup' is not allowed!

"not allowed" sounds like an access-right problem.

Examine that chunk on the destination. You may move it away (this means to create a backup of it and then eliminate the problematic chunk) and repeat your attempt to backup #103

Re-verify that the user "backup" on the PBS has full access rights on that storage.

To elaborate: it should look similar like mine:

root@pbs:/pbspool/pbs0/.chunks/552b# ls -Al | head
total 27352
-rw-r--r-- 1 backup backup  692125 Jun  7 02:39 552b0b6bfff424221370cf2db7ec80fe446b200eefe83592eb7aac5eb2e8b56b
-rw-r--r-- 1 backup backup 1075692 Jun  3 04:19 552b2434a490d4654c6cb684b6145fd089df1bd08e66127eb027fe0e903729b7

If you had tampered with that area in the past as "root" (or any other user) for any reason, then the owner may be "root" and not "backup"...

 

[Robert Obkircher]

"not allowed" sounds like an access-right problem.

No, this error message indicates that another encrypted chunk with the same hash but a different size already exists. That should not happen which is why the operation is not allowed.

It could be a corrupted file from a previous attempt, but I'm not sure.

 

[Chris]

Please post the output of proxmox-backup-debug inspect chunk <datastore-base-path>/.chunks/552bef595b29e12e35478064ddf764383b8f593dadc606df538dbed7bd0407df. This should tell more about potential chunk corruption.

Also note, you need to run garbage collection after the 24h 5min for chunks to be cleaned up. Or reduce the datastore tuning option gc-atime-cutoff as described in https://pbs.proxmox.com/docs/storage.html#tuning

 

[curruscanis]

"not allowed" sounds like an access-right problem.

Examine that chunk on the destination. You may move it away (this means to create a backup of it and then eliminate the problematic chunk) and repeat your attempt to backup #103

Re-verify that the user "backup" on the PBS has full access rights on that storage.

To elaborate: it should look similar like mine:

root@pbs:/pbspool/pbs0/.chunks/552b# ls -Al | head
total 27352
-rw-r--r-- 1 backup backup  692125 Jun  7 02:39 552b0b6bfff424221370cf2db7ec80fe446b200eefe83592eb7aac5eb2e8b56b
-rw-r--r-- 1 backup backup 1075692 Jun  3 04:19 552b2434a490d4654c6cb684b6145fd089df1bd08e66127eb027fe0e903729b7

If you had tampered with that area in the past as "root" (or any other user) for any reason, then the owner may be "root" and not "backup"...

All my directories seem to be correct:


root@yscdtpbs:/mnt/dtsynlun2/.chunks/55be# ls -Al | head
total 132204
-rw-r--r-- 1 backup backup 3012125 May 23 00:12 55be068759f6752b5b385c0fe2664b08fedc5c5dca8a50da953160becc175818
-rw-r--r-- 1 backup backup 4182402 May 19 12:35 55be0a5bf7d3f86857afcb8ea4824a3510c4bee24cd6a91d37918927fb0ab21d
-rw-r--r-- 1 backup backup 4194348 May 19 13:36 55be0f80a1f95f4f86a52defaa9178ea47c4f6da769068bd2bd84c613d697572
-rw-r--r-- 1 backup backup 3701289 May 23 16:00 55be10794a3d551bcf46e20c227fe8d44af709bc5fec0afe1e2a913f822254bb
-rw-r--r-- 1 backup backup 4187558 May 19 23:33 55be1a143560a92e7340044060dacc278c31a3009dfe128b3a5542cb5d633204
-rw-r--r-- 1 backup backup 1813610 Jun 4 04:19 55be1a50c3fa1be49d355e10ce3226f0014ec5017673d434345c2a34761355fb
-rw-r--r-- 1 backup backup 3400189 May 20 10:50 55be1b74bb28bc2b745d030755ed56233c0855ca40bcd664bbe3587d20ab7df7
-rw-r--r-- 1 backup backup 4194348 May 20 23:07 55be1ed519c57f667a598bd29922e9ccd3f49d4f8bbdab4e192c484a142cdeb6
-rw-r--r-- 1 backup backup 3079122 May 23 11:09 55be209b934b991c8eda8a7cdc368548a3ebd62cf4f3bdb7bc156521140d4de5

 

[curruscanis]

Please post the output of proxmox-backup-debug inspect chunk <datastore-base-path>/.chunks/552bef595b29e12e35478064ddf764383b8f593dadc606df538dbed7bd0407df. This should tell more about potential chunk corruption.

Also note, you need to run garbage collection after the 24h 5min for chunks to be cleaned up. Or reduce the datastore tuning option gc-atime-cutoff as described in https://pbs.proxmox.com/docs/storage.html#tuning

When I run that command I get the following error:

root@yscdtpbs:/mnt/dtsynlun2/.chunks# proxmox-backup-debug inspect chunk /mnt/dtsynlun2/.chunks/552bef595b29e12e35478064ddf764383b8f593dadc606df538dbed7bd0407df
Error: could not open chunk file - No such file or directory (os error 2)

 

[Chris]

When I run that command I get the following error:

root@yscdtpbs:/mnt/dtsynlun2/.chunks# proxmox-backup-debug inspect chunk /mnt/dtsynlun2/.chunks/552bef595b29e12e35478064ddf764383b8f593dadc606df538dbed7bd0407df
Error: could not open chunk file - No such file or directory (os error 2)

Sorry, forgot the chunk prefix folder, so the command should be proxmox-backup-debug inspect chunk /mnt/dtsynlun2/.chunks/552b/552bef595b29e12e35478064ddf764383b8f593dadc606df538dbed7bd0407df according to your output

 

[curruscanis]

Sorry, forgot the chunk prefix folder, so the command should be proxmox-backup-debug inspect chunk /mnt/dtsynlun2/.chunks/552b/552bef595b29e12e35478064ddf764383b8f593dadc606df538dbed7bd0407df according to your output

I get the following error with that path:

root@yscdtpbs:/mnt/dtsynlun2/.chunks/55be# proxmox-backup-debug inspect chunk /mnt/dtsynlun2/.chunks/552b/552bef595b29e12e35478064ddf764383b8f593dadc606df538dbed7bd0407df
Error: Data blob has wrong CRC checksum.

 

[Onslow]

@curruscanis
Have you used "Verify" function for this particular backup?
It probably will fail on this chunk, but for debugging or out of curiosity you may try.

If it fails, it will rename the chunk ....bad.0 and maybe the error will give way for new backup.

Or you may do what @UdoB has already suggested:

"You may move it away (this means to create a backup of it and then eliminate the problematic chunk) and repeat your attempt to backup #103".

Anyway, keep the erroneous chunk in other directory in case it is needed, for instance for analysis.

 

[curruscanis]

@curruscanis
Have you used "Verify" function for this particular backup?
It probably will fail on this chunk, but for debugging or out of curiosity you may try.

If it fails, it will rename the chunk ....bad.0 and maybe the error will give way for new backup.

Or you may do what @UdoB has already suggested:

"You may move it away (this means to create a backup of it and then eliminate the problematic chunk) and repeat your attempt to backup #103".

Anyway, keep the erroneous chunk in other directory in case it is needed, for instance for analysis.

I have not been able to verify the backup as the backups have been deleted via the gui... but yet the problem remains.

 

[curruscanis]

I am perfectly open to any method that will allow me to purge / delete / kill / remove / assasinate anything that is preventing me from backing up this vm encrypted if it does not affect the other VM's that are being backed up successfully.

Thank you all for your help!