Proxmox Backup Client error - Unclosed encoder dropped - upload failed

N

nikhilbhalwankar

Active Member
Jun 8, 2019
11
0
41
41
Hi,
I completed full backup of filesystem worth 4 TB successfully using metadata change detection. AFter few days, I initiated backup again of the same filesystem which turned out to be incremental as expected. However, for jpg files of size more than 20 MB, I am randomly getting error - Unclosed encoder dropped. Upload failed for xyz.JPG.

This file is already existing in full backup. Also I tried manually opening this file from filesystem and it opens up correctly. This means the file is not corrupt. Still the backup client fails.
 
nikhilbhalwankar said:
Hi,
I completed full backup of filesystem worth 4 TB successfully using metadata change detection. AFter few days, I initiated backup again of the same filesystem which turned out to be incremental as expected. However, for jpg files of size more than 20 MB, I am randomly getting error - Unclosed encoder dropped. Upload failed for xyz.JPG.

This file is already existing in full backup. Also I tried manually opening this file from filesystem and it opens up correctly. This means the file is not corrupt. Still the backup client fails.
Click to expand...
Please share the full output of the client, there should be an additional line indicating the reason why the client fails.
 
Below is the error log. Total size of the filesystem is around 4 TB. It stucked at 1.45 TB.


Code: 
unclosed encoder dropped
closed encoder dropped with state
unfinished encoder state dropped
unfinished encoder state dropped
unfinished encoder state dropped
unfinished encoder state dropped
unfinished encoder state dropped
unfinished encoder state dropped
photolibrary01.ppxar: reused 1.451 TiB from previous snapshot for unchanged files (453468 chunks)
photolibrary01.ppxar: had to backup 83.891 MiB of 1.451 TiB (compressed 82.698 MiB) in 5674.08 s (average 15.14 KiB/s)
photolibrary01.ppxar: backup was done incrementally, reused 1.451 TiB (100.0%)
Error: upload failed: error at "/path_to_folder/DSC_0007.JPG"
 
What version of the proxmox-backup-client are you using? Please post the output of proxmox-backup-client version. Further, please share the command line parameters you are executing the client with.

nikhilbhalwankar said:
However, for jpg files of size more than 20 MB, I am randomly getting error - Unclosed encoder dropped. Upload failed for xyz.JPG.
Click to expand...
Does it always fail on the same file?

We had reports similar to this in the past, where the number of open file handles caused issues [0]. The lookahead cache used during metadata comparison in the client needs to hold open file handles, the cache capacity is set dynamically based on given limits on client invocation. Please try and see if explicitly reducing this limit helps in your case. You can achieve this by invoking with
Code: 
prlimit --nofile=1024:1024 proxmox-backup-client ...

[0] https://forum.proxmox.com/threads/b...e-pbc-with-metadata-backup.158634/post-727020
 
  • Like
Reactions: nikhilbhalwankar
Thanks Chris. It worked well. Below are the last few lines from the log.

Code: 
processed 1.452 TiB in 19m, uploaded 82.317 MiB
processed 1.452 TiB in 20m, uploaded 82.317 MiB
processed 1.452 TiB in 21m, uploaded 82.317 MiB
processed 1.452 TiB in 22m, uploaded 82.317 MiB
processed 1.452 TiB in 23m, uploaded 82.317 MiB
processed 1.452 TiB in 24m, uploaded 82.317 MiB
processed 1.452 TiB in 25m, uploaded 82.317 MiB
processed 2.038 TiB in 26m, uploaded 83.891 MiB
processed 3.74 TiB in 27m, uploaded 84.895 MiB
Change detection summary:
 - 446076 total files (0 hardlinks)
 - 445355 unchanged, reusable files with 3.944 TiB data
 - 721 changed or non-reusable files with 179.294 MiB data
 - 135.55 MiB padding in 69 partially reused chunks
photolibrary01.mpxar: had to backup 63.132 MiB of 63.132 MiB (compressed 11.443 MiB) in 1640.58 s (average 39.404 KiB/s)
photolibrary01.ppxar: reused 3.944 TiB from previous snapshot for unchanged files (1233779 chunks)
photolibrary01.ppxar: had to backup 85.899 MiB of 3.944 TiB (compressed 82.731 MiB) in 1640.39 s (average 53.621 KiB/s)
photolibrary01.ppxar: backup was done incrementally, reused 3.944 TiB (100.0%)
End Time: Tue Apr 29 15:43:00 2025
 
Glad that with the reduced nofile limit it works as expected for you. Out of curiosity, is the source content located on a CIFS or NFS share?
 
You must log in or register to reply here.
Top Bottom