Since 9-30-2025 I have been having an issue getting 3 containers and a VM to back up to my Proxmox Backup Server. In total I have 15 containers and 2 VMs that all backup to PBS with these being the only issue.
I shutdown the containers and ran a fsck on them to make sure everything was fine which it was. Some of the containers are on data2 and data mount points which are different nvme formatted to ext4. Several other containers are on both of those mount points and have no issue backing up to PBS.
There is plenty of space on PBS (400GB which is enough to do several entire backups of each of these containers and the VM).
I was able to get CT115 to backup by deleting the failed backup on PBS, backing CT115 up onto my proxmox sever, then changing the destination to PBS and running the backup again. That method was not successful for the other 2 containers and 1 VM. They will backup locally.
Does anyone have any idea what is going on?
Versions-
FSCK-
PBS error for one of the backups.
PVE error for a CT.
I shutdown the containers and ran a fsck on them to make sure everything was fine which it was. Some of the containers are on data2 and data mount points which are different nvme formatted to ext4. Several other containers are on both of those mount points and have no issue backing up to PBS.
There is plenty of space on PBS (400GB which is enough to do several entire backups of each of these containers and the VM).
I was able to get CT115 to backup by deleting the failed backup on PBS, backing CT115 up onto my proxmox sever, then changing the destination to PBS and running the backup again. That method was not successful for the other 2 containers and 1 VM. They will backup locally.
Does anyone have any idea what is going on?
Versions-
Code:
root@pve:~# pveversion
pve-manager/8.3.2/3e76eec21c4a14a7 (running kernel: 6.8.12-5-pve)
root@pbs:~# uname -r
6.8.12-4-pve
root@pbs:~# proxmox-backup-manager version
proxmox-backup-server 3.4.7-1 running version: 3.4.7FSCK-
Code:
root@pve:~# pct fsck 114
fsck from util-linux 2.38.1
/mnt/data2/images/114/vm-114-disk-0.raw: clean, 28357/4194304 files, 3118175/16777216 blocks
root@pve:~# pct fsck 113
fsck from util-linux 2.38.1
/mnt/data2/images/113/vm-113-disk-0.raw: clean, 3247012/19660800 files, 40007371/78643200 blocks
root@pve:~# pct fsck 115
fsck from util-linux 2.38.1
/mnt/data/images/115/vm-115-disk-0.raw: clean, 579127/7864320 files, 16278288/31457280 blocksPBS error for one of the backups.
Code:
2025-09-16T03:08:56-04:00: successfully added chunk 5679000645adab60065e3fd8f668a55f966cd275160fc669215374de49f79505 to dynamic index 1 (offset 9153652, size 160374)
2025-09-16T03:08:56-04:00: successfully added chunk e1570d3848c4401f2cc1dec007e103d5a9af8c60508d43b054882b12d2a3dd05 to dynamic index 1 (offset 9314026, size 621438)
2025-09-16T03:08:56-04:00: successfully added chunk 9e4d0b34a2302abd2779323df6f5df5ca27bb6489392e98c2b855b3e1d92698c to dynamic index 1 (offset 9935464, size 285044)
2025-09-16T03:08:56-04:00: successfully added chunk b4ab98e6a7024666832bc9f8033dd9b49a5f419eee5b24b27e827a964236ae98 to dynamic index 1 (offset 10220508, size 184782)
2025-09-16T03:08:56-04:00: successfully added chunk 0ceeafb049aeac82cfdca6de9237c55925785cd06ef22bee80d339255419ceda to dynamic index 1 (offset 10405290, size 191977)
2025-09-16T03:08:56-04:00: successfully added chunk ee6068533449bc0c7bb2d31e8dbc0f6189d320fb4a5586d09a65871df041bfbe to dynamic index 1 (offset 10597267, size 854612)
2025-09-16T03:08:56-04:00: successfully added chunk e0503c7d5e861c036fb766cf0d7f439512687777e3e43eef7c772e2fcdebfcb2 to dynamic index 1 (offset 11451879, size 143678)
2025-09-16T03:08:56-04:00: successfully added chunk a7bb0d3747385a28e2c4e0b51915c1562563c9a981a1126e1ad17623de623bac to dynamic index 1 (offset 11595557, size 1752686)
2025-09-16T03:08:56-04:00: successfully added chunk 614c3980912b0bf96ff603159c868bc6b573e1386acfeb6b23763ad4b8ae9ec2 to dynamic index 1 (offset 13348243, size 142033)
2025-09-16T03:08:56-04:00: successfully added chunk 7c732639f64d6d21763352ec0daa8004e041c4d804471f67cb765133c2a7eddc to dynamic index 1 (offset 13490276, size 876877)
2025-09-16T03:08:56-04:00: successfully added chunk e734476ef7da5d9ec0322d5d7ff30e127f7d20472bbba8a9e2bf3e67cc3d1104 to dynamic index 1 (offset 14367153, size 1651126)
2025-09-16T03:08:56-04:00: successfully added chunk c6149e57bb15155f6b638dbfe90aaf2246a673470ab0e30d32bf525edfe72927 to dynamic index 1 (offset 16018279, size 342462)
2025-09-16T03:08:56-04:00: successfully added chunk 61e61ba7f644ef3c609972c84a1a142850ddd84c09d760e299a7fb86a749eef0 to dynamic index 1 (offset 16360741, size 73759)
2025-09-16T03:08:56-04:00: POST /dynamic_close
2025-09-16T03:08:56-04:00: Upload statistics for 'catalog.pcat1.didx'
2025-09-16T03:08:56-04:00: UUID: dd0964f5a0644de280cb8cb35c787f0b
2025-09-16T03:08:56-04:00: Checksum: 1e2388c772a4b27287eb0a1723eee5a2f61b1db0c2e4d522abe937b2964ecdb5
2025-09-16T03:08:56-04:00: Size: 16434500
2025-09-16T03:08:56-04:00: Chunk count: 34
2025-09-16T03:08:56-04:00: Upload size: 16434500 (100%)
2025-09-16T03:08:56-04:00: Duplicates: 0+9 (26%)
2025-09-16T03:08:56-04:00: Compression: 37%
2025-09-16T03:08:56-04:00: successfully closed dynamic index 1
2025-09-16T03:08:56-04:00: POST /blob
2025-09-16T03:08:56-04:00: add blob "/mnt/storage_nvme/ct/115/2025-09-16T07:00:06Z/index.json.blob" (535 bytes, comp: 535)
2025-09-16T03:08:56-04:00: POST /finish
2025-09-16T03:09:04-04:00: stat registered chunks failed - stat failed on 659c76493466bdd320ab97a4edafd4ca80cabed9d59263d0b398c4afac431642
Caused by:
Structure needs cleaning (os error 117)
2025-09-16T03:09:04-04:00: POST /finish: 400 Bad Request: stat known chunks failed - stat failed on 659c76493466bdd320ab97a4edafd4ca80cabed9d59263d0b398c4afac431642
Caused by:
Structure needs cleaning (os error 117)
2025-09-16T03:09:04-04:00: backup ended and finish failed: backup ended but finished flag is not set.
2025-09-16T03:09:04-04:00: removing unfinished backup
2025-09-16T03:09:04-04:00: removing backup snapshot "/mnt/storage_nvme/ct/115/2025-09-16T07:00:06Z"
2025-09-16T03:09:04-04:00: TASK ERROR: backup ended but finished flag is not set.PVE error for a CT.
Code:
()
INFO: starting new backup job: vzdump 113 --all 0 --storage pbs --notes-template '{{guestname}}' --fleecing 0 --mode stop --node pve --compress zstd
INFO: Starting Backup of VM 113 (lxc)
INFO: Backup started at 2025-11-02 10:52:30
INFO: status = running
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: ****
INFO: including mount point rootfs ('/') in backup
INFO: excluding bind mount point mp0 ('/mnt/plex/lvm') from backup (not a volume)
INFO: excluding bind mount point mp1 ('/mnt/Sabnzbd_Docker/incomplete') from backup (not a volume)
INFO: stopping virtual guest
INFO: creating Proxmox Backup Server archive 'ct/113/2025-11-02T15:52:30Z'
INFO: set max number of entries in memory for file-based backups to 1048576
INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=encrypt --keyfd=17 pct.conf:/var/tmp/vzdumptmp3028621_113/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 113 --backup-time 1762098750 --entries-max 1048576 --repository proxmox@pbs@192.168.1.113:proxmox
INFO: Starting backup: ct/113/2025-11-02T15:52:30Z
INFO: Client name: pve
INFO: Starting backup protocol: Sun Nov 2 10:52:43 2025
INFO: Using encryption key from file descriptor..
INFO: Downloading previous manifest (Tue Sep 9 02:30:01 2025)
INFO: Upload config file '/var/tmp/vzdumptmp3028621_113/etc/vzdump/pct.conf' to 'proxmox@pbs@192.168.1.113:8007:proxmox' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'proxmox@pbs@*****:proxmox' as root.pxar.didx
INFO: processed 22.564 GiB in 1m, uploaded 56.268 MiB
INFO: unclosed encoder dropped
INFO: closed encoder dropped with state
INFO: unfinished encoder state dropped
INFO: finished encoder state with errors
INFO: catalog upload error - channel closed
INFO: Error: inserting chunk on store 'proxmox' failed for 659b821fea4614f527039b8ea5f7e33a56c16a4676cddd75d5a8b48de4c8863d - mkstemp "/mnt/storage_nvme/.chunks/659b/659b821fea4614f527039b8ea5f7e33a56c16a4676cddd75d5a8b48de4c8863d.tmp_XXXXXX" failed: EUCLEAN: Structure needs cleaning
INFO: restarting vm
INFO: guest is online again after 131 seconds
ERROR: Backup of VM 113 failed - command '/usr/bin/proxmox-backup-client backup '--crypt-mode=encrypt' '--keyfd=17' pct.conf:/var/tmp/vzdumptmp3028621_113/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 113 --backup-time 1762098750 --entries-max 1048576 --repository proxmox@pbs@192.168.1.113:proxmox' failed: exit code 255
INFO: Failed at 2025-11-02 10:54:41
INFO: Backup job finished with errors
TASK ERROR: job errors
Last edited:
. I mean filesystem, not hardware.