Backups failed after server 4.1.8 and 4.1.10, client 4.1.9 updates

K

khe91

Active Member
Jul 24, 2020
22
1
43
VM 202
INFO: Error: upload failed: error at "var/cache/apparmor/29ff1d9f.0/usr.sbin.tcpdump-RXsIDo"
VM 206
INFO: Error: upload failed: error at "var/cache/apparmor/2693c843.0/nvidia_modprobe-l8oMqr"
VM 207
INFO: Error: upload failed: error at "var/cache/apparmor/7a03f264.0/nvidia_modprobe-QIS9FQ"
VM 300
INFO: Error: upload failed: error at "var/cache/apparmor/7a03f264.0/nvidia_modprobe-jI7R38"

Worked fine before the latest updates.
All Ubuntu 22.04 LTS vm's.
Backups to CIFS of the same vm's worked without errors, just to PBS server 4.1.8. gives the errors. Backups to PBS 4.1.6 work too.
 
Last edited:
Please share the full backup task logs and the vm configs, thanks.
 
Chris said:
Please share the full backup task logs and the vm configs, thanks.
Click to expand...

One example:

INFO: starting new backup job: vzdump 300 --mode stop --node pve2 --notes-template '{{guestname}}' --remove 0 --storage pbs100G --notification-mode notification-system
INFO: Starting Backup of VM 300 (lxc)
INFO: Backup started at 2026-04-25 16:29:38
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: dev
INFO: including mount point rootfs ('/') in backup
INFO: creating Proxmox Backup Server archive 'ct/300/2026-04-25T14:29:38Z'
/dev/rbd6
INFO: set max number of entries in memory for file-based backups to 1048576
INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp112210_300/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 300 --backup-time 1777127378 --change-detection-mode metadata --entries-max 1048576 --repository fritz@pam@192.xxx.yyy.54:pbs_zfs1 --ns pve2
INFO: Starting backup: [pve2]:ct/300/2026-04-25T14:29:38Z
INFO: Client name: pve2
INFO: Starting backup protocol: Sat Apr 25 16:29:38 2026
INFO: Downloading previous manifest (Thu Apr 23 20:00:57 2026)
INFO: Upload config file '/var/tmp/vzdumptmp112210_300/etc/vzdump/pct.conf' to 'fritz@pam@192.xxx.yyy.54:8007:pbs_zfs1' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'fritz@pam@192.xxx.yyy.54:8007:pbs_zfs1' as root.mpxar.didx
INFO: Using previous index as metadata reference for 'root.mpxar.didx'
INFO: unclosed encoder dropped
INFO: closed encoder dropped with state
INFO: unfinished encoder state dropped
INFO: root.ppxar: reused 2.184 GiB from previous snapshot for unchanged files (1263 chunks)
INFO: root.ppxar: had to backup 63.472 MiB of 2.596 GiB (compressed 21.534 MiB) in 8.61 s (average 7.373 MiB/s)
INFO: root.ppxar: backup was done incrementally, reused 2.534 GiB (97.6%)
INFO: Error: upload failed: error at "var/cache/apparmor/7a03f264.0/nvidia_modprobe-jI7R38"
ERROR: Backup of VM 300 failed - command '/usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp112210_300/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 300 --backup-time 1777127378 --change-detection-mode metadata --entries-max 1048576 --repository fritz@pam@192.xxx.yyy.54:pbs_zfs1 --ns pve2' failed: exit code 255
INFO: Failed at 2026-04-25 16:29:47
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors

arch: amd64
cores: 1
features: nesting=1
hostname: dev
memory: 2048
nameserver: 192.xxx.yyy.40
net0: name=eth0,bridge=vmbr0,gw=192.xxx.yyy.1,hwaddr=D2:D1:B1:00:CE:9A,ip=192.xxx.yyy.64/24,type=veth
ostype: ubuntu
rootfs: ceph:vm-300-disk-0,size=10G,mountoptions=discard
searchdomain: intranet.zzz.de
swap: 512
tags:
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net dev/net none bind,create=dir
 
Last edited:
Please try to get some debug output by running PBS_LOG=debug vzdump 300 --storage pbs100G --pbs-change-detection-mode=metadata | tail -n 300 > client-debug.log and attach the generated file here. Note that this will most likely contains some directory and filenames, so retract if required.
 
I'm seeing the same issue, on a single LXC (running Plex) out of many LXCs and VMs. Here's my log, hope it helps...
 

Attachments

Chris said:
Please try to get some debug output by running PBS_LOG=debug vzdump 300 --storage pbs100G --pbs-change-detection-mode=metadata | tail -n 300 > client-debug.log and attach the generated file here. Note that this will most likely contains some directory and filenames, so retract if required.
Click to expand...

Here comes the log:
 

Attachments

Same errors with PBS 4.1.10

VM 300
INFO: Error: upload failed: error at "var/cache/apparmor/7a03f264.0/nvidia_modprobe-jI7R38"

VM 202
INFO: Error: upload failed: error at "var/cache/apparmor/29ff1d9f.0/usr.sbin.tcpdump-RXsIDo"
 
Last edited:
please also share the output of ls -la of the affected directories within the container. Further, please download the corresponding sub-path where the error occurs via the single file restore of the previous snapshots and list its contents for comparison. Do all of the affected files have zero content size?
 
Last edited:
Unfortunately the tail of the error logs are a bit to short, could you either collect the full log (might be big though), or extend this to e.g. tail -n 3000
 
To make sure we have everything in one round, please run both commands and attach both output files.

Bash: 
PBS_LOG=debug vzdump 300 --storage pbs100G --pbs-change-detection-mode=metadata 2>&1 \
    | grep -vE "Connection: |frame=(Headers|Data)|append chunks list len" \
    | zstd -o client-debug.log.zst

If the resulting file is still too big to attach, trim it to the last 1000 lines without re-running the backup: zstdcat client-debug.log.zst | tail -n 1000 | zstd -o client-debug-tail.log.zst

A dump of the previous successful snapshot's metadata so we can check the on-disk offsets directly. Replace <SNAP> with the most recent successful snapshot of ct/300 before the failing run (something like ct/300/2026-04-26T14:00:00Z):

Bash: 
export PBS_REPOSITORY=root@pam@192.168.250.54:pbs_zfs1
NS=pve2
SNAP=<SNAP>

mkdir /tmp/pxar-inspect && cd /tmp/pxar-inspect
proxmox-backup-client restore "$SNAP" root.mpxar root.mpxar --ns "$NS"
proxmox-backup-client restore "$SNAP" root.ppxar root.ppxar --ns "$NS"
PBS_LOG=debug pxar list root.mpxar --payload-input root.ppxar 2>&1 \
    | grep PayloadOffset | gzip > pxar-offsets.log.gz

Attach client-debug.log.zst and pxar-offsets.log.gz here or send them to me via mail (my exact username here + @proxmox.com) and refernce this forum thread.
 
  • Like
Reactions: Chris
There are newer packages on the test repos with a preliminary fix for this that would need confirmation here.
To test ensure the test repo is enabled; i.e., PVE Node -> Repositories, then hit "Refresh" button under "Updates" after that and check for proxmox-backup-client version 4.1.12-1 as available update. Upgrade to that and retry your affected backups to a PBS storage.
 
With 4.1.12-1 on PBS and Client:

INFO: starting new backup job: vzdump 300 --mode stop --node pve2 --notification-mode notification-system --notes-template '{{guestname}}' --storage pbs100G --remove 0
INFO: Starting Backup of VM 300 (lxc)
INFO: Backup started at 2026-04-28 18:27:44
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: dev
INFO: including mount point rootfs ('/') in backup
INFO: creating Proxmox Backup Server archive 'ct/300/2026-04-28T16:27:44Z'
INFO: set max number of entries in memory for file-based backups to 1048576
INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp222473_300/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 300 --backup-time 1777393664 --change-detection-mode metadata --entries-max 1048576 --repository fritz@pam@192.xxx.yyy.54:pbs_zfs1 --ns pve2
INFO: Starting backup: [pve2]:ct/300/2026-04-28T16:27:44Z
INFO: Client name: pve2
INFO: Starting backup protocol: Tue Apr 28 18:27:44 2026
INFO: Downloading previous manifest (Thu Apr 23 20:00:57 2026)
INFO: Upload config file '/var/tmp/vzdumptmp222473_300/etc/vzdump/pct.conf' to 'fritz@pam@192.xxx.yyy.54:8007:pbs_zfs1' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'fritz@pam@192.xxx.yyy.54:8007:pbs_zfs1' as root.mpxar.didx
INFO: Using previous index as metadata reference for 'root.mpxar.didx'
INFO: re-encode: "nvidia_modprobe-jI7R38" previous archive payload offset 2783537890 not strictly greater than last seen 2784303267, treating as non-reusable
INFO: re-encode: "nvidia_modprobe-jJJfPP" previous archive payload offset 2783578173 not strictly greater than last seen 2784303267, treating as non-reusable
INFO: re-encode: "nvidia_modprobe-jJuu3e" previous archive payload offset 2783618456 not strictly greater than last seen 2784303267, treating as non-reusable
INFO: re-encode: "nvidia_modprobe-jQWd8A" previous archive payload offset 2783658739 not strictly greater than last seen 2784303267, treating as non-reusable
INFO: re-encode: "nvidia_modprobe-jbJ1FC" previous archive payload offset 2783699022 not strictly greater than last seen 2784303267, treating as non-reusable
INFO: re-encode: "nvidia_modprobe-kJAGqI" previous archive payload offset 2783739305 not strictly greater than last seen 2784303267, treating as non-reusable
INFO: re-encode: "nvidia_modprobe-kiDSuM" previous archive payload offset 2783779588 not strictly greater than last seen 2784303267, treating as non-reusable
INFO: re-encode: "nvidia_modprobe-ksSwKg" previous archive payload offset 2783819871 not strictly greater than last seen 2784303267, treating as non-reusable
INFO: re-encode: "nvidia_modprobe-l6QS7U" previous archive payload offset 2783860154 not strictly greater than last seen 2784303267, treating as non-reusable
INFO: re-encode: "nvidia_modprobe-lIN1rC" previous archive payload offset 2783900437 not strictly greater than last seen 2784303267, treating as non-reusable
INFO: re-encode: "nvidia_modprobe-lnzowf" previous archive payload offset 2783940720 not strictly greater than last seen 2784303267, treating as non-reusable
INFO: re-encode: "nvidia_modprobe-m17Ml3" previous archive payload offset 2783981003 not strictly greater than last seen 2784303267, treating as non-reusable
INFO: re-encode: "nvidia_modprobe-mKS3jV" previous archive payload offset 2784021286 not strictly greater than last seen 2784303267, treating as non-reusable
INFO: re-encode: "nvidia_modprobe-mKYUtK" previous archive payload offset 2784061569 not strictly greater than last seen 2784303267, treating as non-reusable
INFO: re-encode: "nvidia_modprobe-mVI0Sk" previous archive payload offset 2784101852 not strictly greater than last seen 2784303267, treating as non-reusable
INFO: re-encode: "nvidia_modprobe-mgPio4" previous archive payload offset 2784142135 not strictly greater than last seen 2784303267, treating as non-reusable
INFO: re-encode: "nvidia_modprobe-mqxTJW" previous archive payload offset 2784182418 not strictly greater than last seen 2784303267, treating as non-reusable
INFO: re-encode: "nvidia_modprobe-n0ft1B" previous archive payload offset 2784222701 not strictly greater than last seen 2784303267, treating as non-reusable
INFO: re-encode: "nvidia_modprobe-n4Qbw9" previous archive payload offset 2784262984 not strictly greater than last seen 2784303267, treating as non-reusable
INFO: re-encode: "nvidia_modprobe-nsce7o" previous archive payload offset 2784303267 not strictly greater than last seen 2784303267, treating as non-reusable
INFO: Change detection summary:
INFO: - 100784 total files (132 hardlinks)
INFO: - 71471 unchanged, reusable files with 5.307 GiB data
INFO: - 29181 changed or non-reusable files with 851.672 MiB data
INFO: - 138.716 MiB padding in 166 partially reused chunks
INFO: root.mpxar: had to backup 17.639 MiB of 17.639 MiB (compressed 3.558 MiB) in 19.55 s (average 923.955 KiB/s)
INFO: root.ppxar: reused 5.442 GiB from previous snapshot for unchanged files (2189 chunks)
INFO: root.ppxar: had to backup 439.775 MiB of 6.274 GiB (compressed 128.92 MiB) in 19.61 s (average 22.425 MiB/s)
INFO: root.ppxar: backup was done incrementally, reused 5.844 GiB (93.2%)
INFO: Duration: 20.16s
INFO: End Time: Tue Apr 28 18:28:04 2026
INFO: adding notes to backup
INFO: Finished Backup of VM 300 (00:00:21)
INFO: Backup finished at 2026-04-28 18:28:05
INFO: Backup job finished successfully
INFO: notified via target `mail-to-root`
TASK OK
 
You must log in or register to reply here.
Top Bottom
Back