Proxmox-backup-client (3.4.6): "unclosed encoder dropped" / "upload failed" during pxar backup (Following #170211 patches)

Y

yorilouz

New Member
Aug 24, 2025
13
1
3
Hello everyone,

I am currently facing an issue with proxmox-backup-client (version 3.4.6) when backing up a directory. The backup fails during the upload process with "unclosed encoder dropped" and "upload failed" errors.

Here is the exact output/log I am getting:
Code: 
Using previous index as metadata reference for 'files.mpxar.didx'
resource limit for open file handles low: 1024
unclosed encoder dropped
closed encoder dropped with state
unfinished encoder state dropped
files.ppxar: reused 547.938 MiB from previous snapshot for unchanged files (183 chunks)
files.ppxar: had to backup 66.367 MiB of 672.245 MiB (compressed 23.478 MiB) in 2.21 s (average 30.037 MiB/s)
files.ppxar: backup was done incrementally, reused 605.878 MiB (90.1%)
Error: upload failed: error at "world/region"

The command I am using is: nice -n 10 proxmox-backup-client backup files.pxar:/path/to/directory --backup-id my_backup_id --change-detection-mode=metadata

Troubleshooting & Context:
  1. I saw thread #165596 and tried using prlimit to adjust the limits, but the backup still fails in the exact same way. Removing nice didn't change anything either.
  2. In the past, I reported an issue where my backups were completing successfully but failing during restore (Thread #170211). Christian Ebner pushed some patches to address this exact issue: pxar: make payload reference offset checks stricter.
  3. I tried running the backup with RUST_BACKTRACE=1 RUST_LOG=trace to catch the exact error right before the encoder drops, but I only get unknown stack frames.
As stated in the patch notes, this stricter check was implemented to "block the encoding of possible corrupt split pxar archives" and "narrow down the still lingering logical issue".

It seems the safeguard worked as intended on my end: the encoder caught an issue (most likely the offset desync) and dropped the state to prevent a corrupt backup, but now I obviously can't complete the backup at all.

I assume this means I am hitting the underlying "lingering logical issue" that was mentioned. Since I am consistently reproducing the behavior, is there any specific debug build, trace, or test I can run to help you guys track down the root cause?

Thanks in advance for your help!
 
yorilouz said:
Hello everyone,

I am currently facing an issue with proxmox-backup-client (version 3.4.6) when backing up a directory. The backup fails during the upload process with "unclosed encoder dropped" and "upload failed" errors.

Here is the exact output/log I am getting:
Code: 
Using previous index as metadata reference for 'files.mpxar.didx'
resource limit for open file handles low: 1024
unclosed encoder dropped
closed encoder dropped with state
unfinished encoder state dropped
files.ppxar: reused 547.938 MiB from previous snapshot for unchanged files (183 chunks)
files.ppxar: had to backup 66.367 MiB of 672.245 MiB (compressed 23.478 MiB) in 2.21 s (average 30.037 MiB/s)
files.ppxar: backup was done incrementally, reused 605.878 MiB (90.1%)
Error: upload failed: error at "world/region"
Click to expand...

Check the backup task log on the PBS side for possible further hints on where the issue is.

yorilouz said:
The command I am using is: nice -n 10 proxmox-backup-client backup files.pxar:/path/to/directory --backup-id my_backup_id --change-detection-mode=metadata

Troubleshooting & Context:
  1. I saw thread #165596 and tried using prlimit to adjust the limits, but the backup still fails in the exact same way. Removing nice didn't change anything either.
  2. In the past, I reported an issue where my backups were completing successfully but failing during restore (Thread #170211). Christian Ebner pushed some patches to address this exact issue: pxar: make payload reference offset checks stricter.
Click to expand...

You are not running into the stricter check. While the patch got applied, it did not get packaged at all so far and internally I was not able to reproduce.

yorilouz said:
  1. I tried running the backup with RUST_BACKTRACE=1 RUST_LOG=trace to catch the exact error right before the encoder drops, but I only get unknown stack frames.
As stated in the patch notes, this stricter check was implemented to "block the encoding of possible corrupt split pxar archives" and "narrow down the still lingering logical issue".
Click to expand...



yorilouz said:
It seems the safeguard worked as intended on my end: the encoder caught an issue (most likely the offset desync) and dropped the state to prevent a corrupt backup, but now I obviously can't complete the backup at all.

I assume this means I am hitting the underlying "lingering logical issue" that was mentioned. Since I am consistently reproducing the behavior, is there any specific debug build, trace, or test I can run to help you guys track down the root cause?
Click to expand...

I suspect there to be a different issue, check the PBS backup task log and systemd journal for errors.
 
Thank you for the quick reply and the clarification! My apologies for jumping to conclusions regarding the patch, it's good to know I'm not hitting that specific safeguard yet.

Following your advice, I checked the logs on the PBS server side, and it seems you are right, there is a different issue at play.

Here is the PBS Backup Task Log:
Code: 
....
successfully closed dynamic index 2
backup ended and finish failed: backup ended but finished flag is not set.
removing unfinished backup
removing backup snapshot "/mnt/datastore/my-datastore/host/my_backup_id/2026-03-16T13:14:46Z"
TASK ERROR: backup ended but finished flag is not set.

And here is the output from journalctl at the exact same time:
Code: 
proxmox-backup-proxy[2682]: TASK ERROR: backup ended but finished flag is not set.
proxmox-backup-proxy[2682]: error writing task result to the tasklog
proxmox-backup-proxy[2682]: error writing task result to the tasklog
proxmox-backup-api[138949]: error writing task result to the tasklog

Important context regarding the server health:
My datastore and system drives are perfectly healthy. I have many other backup tasks running very regularly on this exact same PBS server without any issues whatsoever. It is only this specific backup job that fails consistently every single time.

Given that this is completely isolated to a single backup task, could there be an issue with a specific corrupted task log file for this group, or perhaps something specific about the data/metadata in this directory causing the proxy to choke when writing the result?

Any pointers on what I should check next on the server side would be greatly appreciated!
 
Last edited:
Please try to use the latest available version of the proxmox backup client, to exclude we are not chasing issues which might have already been fixed since. Same goes for the proxmox backup server. What version is running there?
 
Hi Christian,

Thanks for the suggestion. I went ahead and upgraded both the client and the server to the latest available versions in the 3.x repositories.
Here are the current versions:
  • Proxmox Backup Client: 3.4.7
  • Proxmox Backup Server: 3.4.8-3
Unfortunately, the issue persists. I ran the backup again and it still fails at the exact same point, producing the exact same logs on both sides.

Since upgrading to PBS 4.x is a major version bump (Debian 12 to 13) that requires planning and a proper maintenance window on my end, I would prefer to stick to the 3.x branch for now, as it is still actively maintained for bug fixes, unless I'm wrong or you know for a fact that this specific proxy/tasklog crash was identified and fixed exclusively in v4?

If we want to continue debugging this on 3.4.8, what can I do to help you pinpoint the issue?
 
Please post the output you get when invoking the proxmox-backup-client with PBS_LOG=debug env varialble set.

Furhter, can you specify if you are using the statically linked binary of the client or a regular dynamically linked build?
 
I am using the regular (dynamically linked) build, not the statically linked binary.

Regarding the logs, I didn't attach the full file because it is extremely verbose and may contains sensitive data like specific filenames that I would prefer not to post publicly.

Instead, I have included a truncated version of the logs below. It shows the beginning with the connection details and the exact moment the client crashes.

If you really need the complete logs to troubleshoot this, please let me know and I can send them to you via private message.

Here are the truncated logs:

Code: 
Starting backup: host/my_backup_id/2026-03-18T08:26:47Z   
Client name: my-host   
Starting backup protocol: Wed Mar 18 09:26:48 2026   
connecting to IP:PORT
connected to IP:PORT
flushed 210 bytes
parsed 3 headers
incoming body is content-length (232 bytes)
incoming body completed
pooling idle connection for ("https", IP:PORT)
reuse idle connection for ("https", IP:PORT)
flushed 547 bytes
parsed 3 headers
incoming body is empty
binding client connection
client connection bound
send frame=Settings { flags: (0x0), initial_window_size: 2147483646, max_frame_size: 4194304 }
Connection: received frame=Settings { flags: (0x0), initial_window_size: 33554432, max_frame_size: 4194304, max_header_list_size: 16777216 } peer=Client
Connection: send frame=Settings { flags: (0x1: ACK) } peer=Client
Connection: received frame=WindowUpdate { stream_id: StreamId(0), size_increment: 33488897 } peer=Client
Connection: send frame=WindowUpdate { stream_id: StreamId(0), size_increment: 2147418111 } peer=Client
Connection: send frame=Headers { stream_id: StreamId(1), flags: (0x5: END_HEADERS | END_STREAM) } peer=Client
Connection: received frame=Settings { flags: (0x1: ACK) } peer=Client
Connection: received settings ACK; applying Settings { flags: (0x0), initial_window_size: 2147483646, max_frame_size: 4194304 } peer=Client
Connection: received frame=Headers { stream_id: StreamId(1), flags: (0x4: END_HEADERS) } peer=Client
Connection: received frame=Data { stream_id: StreamId(1), flags: (0x1: END_STREAM) } peer=Client
Downloading previous manifest (Fri Feb 27 04:38:15 2026)   
Connection: send frame=Headers { stream_id: StreamId(3), flags: (0x5: END_HEADERS | END_STREAM) } peer=Client
Connection: received frame=Headers { stream_id: StreamId(3), flags: (0x4: END_HEADERS) } peer=Client
Connection: received frame=Data { stream_id: StreamId(3) } peer=Client
Connection: received frame=Data { stream_id: StreamId(3), flags: (0x1: END_STREAM) } peer=Client
Upload directory '/path/to/data/my_backup_id' to 'user@pbs@IP:PORT:my-datastore' as files.mpxar.didx   
Connection: send frame=Headers { stream_id: StreamId(5), flags: (0x5: END_HEADERS | END_STREAM) } peer=Client
Connection: received frame=Headers { stream_id: StreamId(5), flags: (0x4: END_HEADERS) } peer=Client
Connection: received frame=Data { stream_id: StreamId(5), flags: (0x1: END_STREAM) } peer=Client
connecting to IP:PORT
connected to IP:PORT
flushed 538 bytes
parsed 3 headers
incoming body is empty
binding client connection
client connection bound
send frame=Settings { flags: (0x0), initial_window_size: 2147483646, max_frame_size: 4194304 }
Connection: send frame=WindowUpdate { stream_id: StreamId(0), size_increment: 2147418111 } peer=Client
Connection: received frame=Settings { flags: (0x0), initial_window_size: 33554432, max_frame_size: 4194304, max_header_list_size: 16777216 } peer=Client
Connection: send frame=Settings { flags: (0x1: ACK) } peer=Client
Connection: send frame=Headers { stream_id: StreamId(1), flags: (0x5: END_HEADERS | END_STREAM) } peer=Client
Connection: received frame=Settings { flags: (0x1: ACK) } peer=Client
Connection: received settings ACK; applying Settings { flags: (0x0), initial_window_size: 2147483646, max_frame_size: 4194304 } peer=Client
Connection: received frame=WindowUpdate { stream_id: StreamId(0), size_increment: 33488897 } peer=Client
Connection: received frame=Headers { stream_id: StreamId(1), flags: (0x4: END_HEADERS) } peer=Client
Connection: received frame=Data { stream_id: StreamId(1) } peer=Client
Connection: received frame=Data { stream_id: StreamId(1), flags: (0x1: END_STREAM) } peer=Client
Connection: send frame=Headers { stream_id: StreamId(7), flags: (0x5: END_HEADERS | END_STREAM) } peer=Client
Connection: received frame=Headers { stream_id: StreamId(7), flags: (0x4: END_HEADERS) } peer=Client
Connection: received frame=Data { stream_id: StreamId(7) } peer=Client
Connection: received frame=Data { stream_id: StreamId(7) } peer=Client
Connection: received frame=Data { stream_id: StreamId(7), flags: (0x1: END_STREAM) } peer=Client
files.ppxar.didx: known chunks list length is 285
Using previous index as metadata reference for 'files.mpxar.didx'   
Connection: send frame=Headers { stream_id: StreamId(3), flags: (0x5: END_HEADERS | END_STREAM) } peer=Client
Connection: received frame=Headers { stream_id: StreamId(3), flags: (0x4: END_HEADERS) } peer=Client
Connection: received frame=Data { stream_id: StreamId(3), flags: (0x1: END_STREAM) } peer=Client
Connection: send frame=Headers { stream_id: StreamId(9), flags: (0x5: END_HEADERS | END_STREAM) } peer=Client
Connection: send frame=Headers { stream_id: StreamId(11), flags: (0x5: END_HEADERS | END_STREAM) } peer=Client
Connection: received frame=Headers { stream_id: StreamId(11), flags: (0x4: END_HEADERS) } peer=Client
Connection: received frame=Headers { stream_id: StreamId(9), flags: (0x4: END_HEADERS) } peer=Client
Connection: received frame=Data { stream_id: StreamId(11) } peer=Client
Connection: received frame=Data { stream_id: StreamId(9) } peer=Client
Connection: received frame=Data { stream_id: StreamId(11) } peer=Client
Connection: received frame=Data { stream_id: StreamId(11), flags: (0x1: END_STREAM) } peer=Client

.........

"world/poi/r.9.-7.mca"
re-encode: "r.9.-7.mca" not found in previous archive.
Got 0 cache entries to encode: reuse is false
Got 0 cache entries to encode: reuse is false
"world/region"
Got 0 cache entries to encode: reuse is false
Connection: received frame=Headers { stream_id: StreamId(119), flags: (0x4: END_HEADERS) } peer=Client
Connection: received frame=Data { stream_id: StreamId(119), flags: (0x1: END_STREAM) } peer=Client
Connection: received frame=Headers { stream_id: StreamId(117), flags: (0x4: END_HEADERS) } peer=Client
Connection: received frame=Data { stream_id: StreamId(117), flags: (0x1: END_STREAM) } peer=Client
Connection: received frame=Headers { stream_id: StreamId(115), flags: (0x4: END_HEADERS) } peer=Client
Connection: received frame=Data { stream_id: StreamId(115), flags: (0x1: END_STREAM) } peer=Client
Chunk size 1040789 below minimum chunk size   
Chunk at suggested boundary: 702998158, chunk size: 1073573   
Boundary 703039281 in future   
Chunk size 41123 below minimum chunk size   
Boundary 703264577 in future   
Chunk size 266419 below minimum chunk size   
Boundary 703334225 in future   
Chunk size 336067 below minimum chunk size   
Chunk size 430420 below minimum chunk size   
Boundary 703588408 in future   
Chunk size 590250 below minimum chunk size   
Chunk size 635461 below minimum chunk size   
Chunk size 688725 below minimum chunk size   
Boundary 703711612 in future   
Chunk size 713454 below minimum chunk size   
Chunk size 713470 below minimum chunk size   
Boundary 704055877 in future   
Chunk at suggested boundary: 704055877, chunk size: 1057719   
Chunk size 188566 below minimum chunk size   
Connection: send frame=Headers { stream_id: StreamId(121), flags: (0x4: END_HEADERS) } peer=Client
Connection: send frame=Headers { stream_id: StreamId(123), flags: (0x4: END_HEADERS) } peer=Client
Connection: send frame=Headers { stream_id: StreamId(125), flags: (0x4: END_HEADERS) } peer=Client
Connection: send frame=Data { stream_id: StreamId(121), flags: (0x1: END_STREAM) } peer=Client
Connection: send frame=Data { stream_id: StreamId(123), flags: (0x1: END_STREAM) } peer=Client
Connection: send frame=Data { stream_id: StreamId(125), flags: (0x1: END_STREAM) } peer=Client
Chunk size 205158 below minimum chunk size   
Chunk size 242179 below minimum chunk size   
Boundary 704367851 in future   
Chunk size 311974 below minimum chunk size   
Chunk size 311990 below minimum chunk size   
Chunk size 320341 below minimum chunk size   
Chunk size 353256 below minimum chunk size   
Chunk size 353272 below minimum chunk size   
Chunk size 390354 below minimum chunk size   
Chunk size 488809 below minimum chunk size   
Boundary 704581758 in future   
Chunk size 525881 below minimum chunk size   
Chunk size 591621 below minimum chunk size   
Chunk size 591637 below minimum chunk size   
Chunk size 673788 below minimum chunk size   
Boundary 704975578 in future   
Chunk size 919701 below minimum chunk size   
Boundary 705090461 in future   
Connection: send frame=Headers { stream_id: StreamId(127), flags: (0x4: END_HEADERS) } peer=Client
Connection: send frame=Data { stream_id: StreamId(127), flags: (0x1: END_STREAM) } peer=Client
Connection: received frame=Headers { stream_id: StreamId(125), flags: (0x4: END_HEADERS) } peer=Client
Connection: received frame=Data { stream_id: StreamId(125), flags: (0x1: END_STREAM) } peer=Client
Connection: received frame=Headers { stream_id: StreamId(123), flags: (0x4: END_HEADERS) } peer=Client
Connection: received frame=Data { stream_id: StreamId(123), flags: (0x1: END_STREAM) } peer=Client
Connection: received frame=Headers { stream_id: StreamId(121), flags: (0x4: END_HEADERS) } peer=Client
Connection: received frame=Data { stream_id: StreamId(121), flags: (0x1: END_STREAM) } peer=Client
Connection: received frame=Headers { stream_id: StreamId(127), flags: (0x4: END_HEADERS) } peer=Client
Connection: received frame=Data { stream_id: StreamId(127), flags: (0x1: END_STREAM) } peer=Client
unclosed encoder dropped
closed encoder dropped with state
unfinished encoder state dropped
Chunk size 1034584 below minimum chunk size   
Chunk at suggested boundary: 705143860, chunk size: 1087983   
Chunk size 73888 below minimum chunk size   
No suggested boundary, regular scan   
Connection: send frame=Headers { stream_id: StreamId(129), flags: (0x4: END_HEADERS) } peer=Client
Connection: send frame=Data { stream_id: StreamId(129), flags: (0x1: END_STREAM) } peer=Client
Connection: send frame=Headers { stream_id: StreamId(131), flags: (0x4: END_HEADERS) } peer=Client
Connection: send frame=Headers { stream_id: StreamId(133), flags: (0x4: END_HEADERS) } peer=Client
Connection: send frame=Data { stream_id: StreamId(131), flags: (0x1: END_STREAM) } peer=Client
Connection: send frame=Data { stream_id: StreamId(133), flags: (0x1: END_STREAM) } peer=Client
Connection: received frame=Headers { stream_id: StreamId(133), flags: (0x4: END_HEADERS) } peer=Client
Connection: received frame=Data { stream_id: StreamId(133), flags: (0x1: END_STREAM) } peer=Client
Connection: received frame=Headers { stream_id: StreamId(131), flags: (0x4: END_HEADERS) } peer=Client
Connection: received frame=Data { stream_id: StreamId(131), flags: (0x1: END_STREAM) } peer=Client
Connection: received frame=Headers { stream_id: StreamId(129), flags: (0x4: END_HEADERS) } peer=Client
Connection: received frame=Data { stream_id: StreamId(129), flags: (0x1: END_STREAM) } peer=Client
append chunks list len (43)
Connection: send frame=Headers { stream_id: StreamId(135), flags: (0x4: END_HEADERS) } peer=Client
Connection: send frame=Data { stream_id: StreamId(135), flags: (0x1: END_STREAM) } peer=Client
Connection: received frame=Headers { stream_id: StreamId(135), flags: (0x4: END_HEADERS) } peer=Client
Connection: received frame=Data { stream_id: StreamId(135), flags: (0x1: END_STREAM) } peer=Client
files.ppxar.didx: reused 547.938 MiB from previous snapshot for unchanged files (183 chunks)
files.ppxar.didx: had to backup 66.67 MiB of 672.548 MiB (compressed 23.525 MiB) in 2.70 s (average 24.704 MiB/s)
files.ppxar.didx: backup was done incrementally, reused 605.878 MiB (90.1%)
files.ppxar.didx: Reused 21 from 260 chunks.
files.ppxar.didx: Average chunk size was 2.587 MiB.
files.ppxar.didx: Average time per request: 10379 microseconds.
Connection: send frame=Headers { stream_id: StreamId(137), flags: (0x5: END_HEADERS | END_STREAM) } peer=Client
Connection: received frame=Headers { stream_id: StreamId(137), flags: (0x4: END_HEADERS) } peer=Client
Connection: received frame=Data { stream_id: StreamId(137), flags: (0x1: END_STREAM) } peer=Client
Error: upload failed: error at "world/region"
 
The high number of files might be the reason, and the error completely lacks the error context (need to check what's wrong there and improve this).

Did you try to bump entries-max already?
Code: 
--entries-max <integer> (default=1048576)
       Max number of entries to hold in memory.

Also, lowering the resource limits for open file handles with prlimit should not be necessary unless you are backing up a CIFS share where the server side enforces the lower limit.
 
  • Like
Reactions: yorilouz
Hi Christian,

That was exactly it! Increasing the --entries-max parameter to 2097152 solved the issue completely. The backup now completes successfully.

Since this directory contains ~1.4M files, the default limit was indeed the bottleneck. However, I have a quick follow-up question regarding this change:

Are there any significant downsides to doubling this limit (or increasing it further if my file count grows)? I assume it increases RAM usage on the client side, but is there a specific 'cost per entry' I should be aware of?

The final command that worked: proxmox-backup-client backup files.pxar:/path/to/directory --backup-id my_backup_id --change-detection-mode=metadata --entries-max 2097152

Thank you very much for your help! As you mentioned, a more explicit error message like 'max-entries limit reached' in the logs would definitely be a great addition for the future.
 
yorilouz said:
Are there any significant downsides to doubling this limit (or increasing it further if my file count grows)? I assume it increases RAM usage on the client side, but is there a specific 'cost per entry' I should be aware of?
Click to expand...
This is exactly the reason for this limit. The pbs-client needs to hold a list of directory entries and their stat metadata [0]. The limit is there to protect against excessive memory usage for environments where that is not acceptable (e.g. container backps in PVE).

[0] https://git.proxmox.com/?p=proxmox-...f8f2f8d68ceff1b53ca29ec7f0ce22b8;hb=HEAD#l680
 
Thanks for the technical details!

Looking at the code you shared (lines 681-687), it seems there is a specific error message: exceeded allowed number of file entries. However, in my case, I never saw this message in the logs. I only got unclosed encoder dropped and upload failed. Is it normal that the explicit error message didn't show up? It would have made the troubleshooting much faster!

Also, since I'm not an expert on the internal structure of these entries, do you have a rough idea of how much RAM the default limit (1,048,576 entries) represents? I just want to make sure that doubling it to 2M won't cause any OOM issues on my client machines.

Thanks again for your help!
 
yorilouz said:
However, in my case, I never saw this message in the logs. I only got unclosed encoder dropped and upload failed. Is it normal that the explicit error message didn't show up? It would have made the troubleshooting much faster!
Click to expand...
That is what I previously meant with the error message lacking context. This is not intentional needs improvement.

yorilouz said:
Also, since I'm not an expert on the internal structure of these entries, do you have a rough idea of how much RAM the default limit (1,048,576 entries) represents? I just want to make sure that doubling it to 2M won't cause any OOM issues on my client machines.
Click to expand...
Not from the top of my head, but you could check the max memory usage when running the client with the limit set, running into the limit. this will at least give you an estimate of what doubling might imply.
 
  • Like
Reactions: yorilouz
You must log in or register to reply here.
Top Bottom
Back