My last weekly backup failed on two CTs
Original backup:
but CT 133 gave me new errors:
Total running time: 2s
Total size: 0 B
vzdump 133 --node FoxN100 --notes-template '{{guestname}}' --notification-mode notification-system --storage PBSBackup --remove 0 --mode snapshot
133: 2025-09-08 10:07:31 INFO: Starting Backup of VM 133 (qemu)
133: 2025-09-08 10:07:31 INFO: status = running
133: 2025-09-08 10:07:31 INFO: VM Name: haos
133: 2025-09-08 10:07:31 INFO: include disk 'scsi0' 'N100Pool:vm-133-disk-1' 32G
133: 2025-09-08 10:07:31 INFO: include disk 'efidisk0' 'N100Pool:vm-133-disk-0' 1M
133: 2025-09-08 10:07:31 INFO: backup mode: snapshot
133: 2025-09-08 10:07:31 INFO: ionice priority: 7
133: 2025-09-08 10:07:31 INFO: creating Proxmox Backup Server archive 'vm/133/2025-09-08T08:07:31Z'
133: 2025-09-08 10:07:31 INFO: issuing guest-agent 'fs-freeze' command
133: 2025-09-08 10:07:31 INFO: issuing guest-agent 'fs-thaw' command
133: 2025-09-08 10:07:32 INFO: started backup task '4c293a6a-e930-436e-adeb-8d02b599a207'
133: 2025-09-08 10:07:32 INFO: resuming VM again
133: 2025-09-08 10:07:32 INFO: efidisk0: dirty-bitmap status: existing bitmap was invalid and has been cleared
133: 2025-09-08 10:07:32 INFO: scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared
133: 2025-09-08 10:07:33 INFO: 2% (772.0 MiB of 32.0 GiB) in 1s, read: 772.0 MiB/s, write: 64.0 MiB/s
133: 2025-09-08 10:07:33 ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: inserting chunk on store 'PBSBackup' failed for 215414ebbe759334fc2acb8530a93542f1091c0f7bc11618346fee5bef2ff837 - mkstemp "/mnt/datastore/PBSBackup/.chunks/2154/215414ebbe759334fc2acb8530a93542f1091c0f7bc11618346fee5bef2ff837.tmp_XXXXXX" failed: ENOENT: No such file or directory
133: 2025-09-08 10:07:33 INFO: aborting backup job
133: 2025-09-08 10:07:33 INFO: resuming VM again
133: 2025-09-08 10:07:33 ERROR: Backup of VM 133 failed - backup write data failed: command error: write_data upload error: pipelined request failed: inserting chunk on store 'PBSBackup' failed for 215414ebbe759334fc2acb8530a93542f1091c0f7bc11618346fee5bef2ff837 - mkstemp "/mnt/datastore/PBSBackup/.chunks/2154/215414ebbe759334fc2acb8530a93542f1091c0f7bc11618346fee5bef2ff837.tmp_XXXXXX" failed: ENOENT: No such file or directory
I looked in the directory /mnt/datastore/PBSBackup/.chunks/2154/ and it is indeed missing. yett the .chunks folder is full other of chunks!
I did not change any permissions or users - the PBS works for other CTs
What can I do to resolve this issue short of recreating my datastore ???
Original backup:
| VMID | Name | Status | Time | Size | Filename |
|---|---|---|---|---|---|
| 106 | PDMFox | ok | 39s | 3.04 GiB | ct/106/2025-09-08T00:00:02Z |
| 108 | NetBird | ok | 12s | 925.135 MiB | ct/108/2025-09-08T00:00:42Z |
| 110 | ObserveFox | ok | 31s | 5.503 GiB | ct/110/2025-09-08T00:00:54Z |
| 111 | CheckMK | err | 6s | 0 B | |
| 114 | adguard | ok | 19s | 1.928 GiB | ct/114/2025-09-08T00:01:31Z |
| 117 | watchyourlan | ok | 10s | 898.452 MiB | ct/117/2025-09-08T00:01:50Z |
| 131 | peanut | ok | 36s | 2.189 GiB | ct/131/2025-09-08T00:02:01Z |
| 133 | haos | err | 2s | 0 B | |
| 136 | PulseTest | ok | 29s | 5.024 GiB | ct/136/2025-09-08T00:02:40Z |
After finding FS corruption (XFS) - I ran the xfs_repair which cleaned the filesystem from errors - rebooted PBS
CT 111 I was able to perform a manual backupbut CT 133 gave me new errors:
Details
| VMID | Name | Status | Time | Size | Filename |
|---|---|---|---|---|---|
| 133 | haos | err | 2s | 0 B |
Total running time: 2s
Total size: 0 B
Logs
vzdump 133 --node FoxN100 --notes-template '{{guestname}}' --notification-mode notification-system --storage PBSBackup --remove 0 --mode snapshot
133: 2025-09-08 10:07:31 INFO: Starting Backup of VM 133 (qemu)
133: 2025-09-08 10:07:31 INFO: status = running
133: 2025-09-08 10:07:31 INFO: VM Name: haos
133: 2025-09-08 10:07:31 INFO: include disk 'scsi0' 'N100Pool:vm-133-disk-1' 32G
133: 2025-09-08 10:07:31 INFO: include disk 'efidisk0' 'N100Pool:vm-133-disk-0' 1M
133: 2025-09-08 10:07:31 INFO: backup mode: snapshot
133: 2025-09-08 10:07:31 INFO: ionice priority: 7
133: 2025-09-08 10:07:31 INFO: creating Proxmox Backup Server archive 'vm/133/2025-09-08T08:07:31Z'
133: 2025-09-08 10:07:31 INFO: issuing guest-agent 'fs-freeze' command
133: 2025-09-08 10:07:31 INFO: issuing guest-agent 'fs-thaw' command
133: 2025-09-08 10:07:32 INFO: started backup task '4c293a6a-e930-436e-adeb-8d02b599a207'
133: 2025-09-08 10:07:32 INFO: resuming VM again
133: 2025-09-08 10:07:32 INFO: efidisk0: dirty-bitmap status: existing bitmap was invalid and has been cleared
133: 2025-09-08 10:07:32 INFO: scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared
133: 2025-09-08 10:07:33 INFO: 2% (772.0 MiB of 32.0 GiB) in 1s, read: 772.0 MiB/s, write: 64.0 MiB/s
133: 2025-09-08 10:07:33 ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: inserting chunk on store 'PBSBackup' failed for 215414ebbe759334fc2acb8530a93542f1091c0f7bc11618346fee5bef2ff837 - mkstemp "/mnt/datastore/PBSBackup/.chunks/2154/215414ebbe759334fc2acb8530a93542f1091c0f7bc11618346fee5bef2ff837.tmp_XXXXXX" failed: ENOENT: No such file or directory
133: 2025-09-08 10:07:33 INFO: aborting backup job
133: 2025-09-08 10:07:33 INFO: resuming VM again
133: 2025-09-08 10:07:33 ERROR: Backup of VM 133 failed - backup write data failed: command error: write_data upload error: pipelined request failed: inserting chunk on store 'PBSBackup' failed for 215414ebbe759334fc2acb8530a93542f1091c0f7bc11618346fee5bef2ff837 - mkstemp "/mnt/datastore/PBSBackup/.chunks/2154/215414ebbe759334fc2acb8530a93542f1091c0f7bc11618346fee5bef2ff837.tmp_XXXXXX" failed: ENOENT: No such file or directory
I looked in the directory /mnt/datastore/PBSBackup/.chunks/2154/ and it is indeed missing. yett the .chunks folder is full other of chunks!
I did not change any permissions or users - the PBS works for other CTs
What can I do to resolve this issue short of recreating my datastore ???