Proxmox-backup-client (3.4.6): "unclosed encoder dropped" / "upload failed" during pxar backup (Following #170211 patches)

[yorilouz]

Hello everyone,

I am currently facing an issue with proxmox-backup-client (version 3.4.6) when backing up a directory. The backup fails during the upload process with "unclosed encoder dropped" and "upload failed" errors.

Here is the exact output/log I am getting:

Using previous index as metadata reference for 'files.mpxar.didx'
resource limit for open file handles low: 1024
unclosed encoder dropped
closed encoder dropped with state
unfinished encoder state dropped
files.ppxar: reused 547.938 MiB from previous snapshot for unchanged files (183 chunks)
files.ppxar: had to backup 66.367 MiB of 672.245 MiB (compressed 23.478 MiB) in 2.21 s (average 30.037 MiB/s)
files.ppxar: backup was done incrementally, reused 605.878 MiB (90.1%)
Error: upload failed: error at "world/region"

The command I am using is: nice -n 10 proxmox-backup-client backup files.pxar:/path/to/directory --backup-id my_backup_id --change-detection-mode=metadata

Troubleshooting & Context:

1. I saw thread #165596 and tried using prlimit to adjust the limits, but the backup still fails in the exact same way. Removing nice didn't change anything either.
2. In the past, I reported an issue where my backups were completing successfully but failing during restore (Thread #170211). Christian Ebner pushed some patches to address this exact issue: pxar: make payload reference offset checks stricter.
3. I tried running the backup with RUST_BACKTRACE=1 RUST_LOG=trace to catch the exact error right before the encoder drops, but I only get unknown stack frames.

As stated in the patch notes, this stricter check was implemented to "block the encoding of possible corrupt split pxar archives" and "narrow down the still lingering logical issue".

It seems the safeguard worked as intended on my end: the encoder caught an issue (most likely the offset desync) and dropped the state to prevent a corrupt backup, but now I obviously can't complete the backup at all.

I assume this means I am hitting the underlying "lingering logical issue" that was mentioned. Since I am consistently reproducing the behavior, is there any specific debug build, trace, or test I can run to help you guys track down the root cause?

Thanks in advance for your help!

 

[Chris]

Hello everyone,

I am currently facing an issue with proxmox-backup-client (version 3.4.6) when backing up a directory. The backup fails during the upload process with "unclosed encoder dropped" and "upload failed" errors.

Here is the exact output/log I am getting:

Using previous index as metadata reference for 'files.mpxar.didx'
resource limit for open file handles low: 1024
unclosed encoder dropped
closed encoder dropped with state
unfinished encoder state dropped
files.ppxar: reused 547.938 MiB from previous snapshot for unchanged files (183 chunks)
files.ppxar: had to backup 66.367 MiB of 672.245 MiB (compressed 23.478 MiB) in 2.21 s (average 30.037 MiB/s)
files.ppxar: backup was done incrementally, reused 605.878 MiB (90.1%)
Error: upload failed: error at "world/region"

Check the backup task log on the PBS side for possible further hints on where the issue is.

The command I am using is: nice -n 10 proxmox-backup-client backup files.pxar:/path/to/directory --backup-id my_backup_id --change-detection-mode=metadata

Troubleshooting & Context:

1. I saw thread #165596 and tried using prlimit to adjust the limits, but the backup still fails in the exact same way. Removing nice didn't change anything either.
2. In the past, I reported an issue where my backups were completing successfully but failing during restore (Thread #170211). Christian Ebner pushed some patches to address this exact issue: pxar: make payload reference offset checks stricter.

You are not running into the stricter check. While the patch got applied, it did not get packaged at all so far and internally I was not able to reproduce.

1. I tried running the backup with RUST_BACKTRACE=1 RUST_LOG=trace to catch the exact error right before the encoder drops, but I only get unknown stack frames.

As stated in the patch notes, this stricter check was implemented to "block the encoding of possible corrupt split pxar archives" and "narrow down the still lingering logical issue".

It seems the safeguard worked as intended on my end: the encoder caught an issue (most likely the offset desync) and dropped the state to prevent a corrupt backup, but now I obviously can't complete the backup at all.

I assume this means I am hitting the underlying "lingering logical issue" that was mentioned. Since I am consistently reproducing the behavior, is there any specific debug build, trace, or test I can run to help you guys track down the root cause?

I suspect there to be a different issue, check the PBS backup task log and systemd journal for errors.

 

[yorilouz]

Thank you for the quick reply and the clarification! My apologies for jumping to conclusions regarding the patch, it's good to know I'm not hitting that specific safeguard yet.

Following your advice, I checked the logs on the PBS server side, and it seems you are right, there is a different issue at play.

Here is the PBS Backup Task Log:

....
successfully closed dynamic index 2
backup ended and finish failed: backup ended but finished flag is not set.
removing unfinished backup
removing backup snapshot "/mnt/datastore/my-datastore/host/my_backup_id/2026-03-16T13:14:46Z"
TASK ERROR: backup ended but finished flag is not set.

And here is the output from journalctl at the exact same time:

proxmox-backup-proxy[2682]: TASK ERROR: backup ended but finished flag is not set.
proxmox-backup-proxy[2682]: error writing task result to the tasklog
proxmox-backup-proxy[2682]: error writing task result to the tasklog
proxmox-backup-api[138949]: error writing task result to the tasklog

Important context regarding the server health:
My datastore and system drives are perfectly healthy. I have many other backup tasks running very regularly on this exact same PBS server without any issues whatsoever. It is only this specific backup job that fails consistently every single time.

Given that this is completely isolated to a single backup task, could there be an issue with a specific corrupted task log file for this group, or perhaps something specific about the data/metadata in this directory causing the proxy to choke when writing the result?

Any pointers on what I should check next on the server side would be greatly appreciated!