Error restoring host backup: "unexpected offset" - mount works, restore fails

[yorilouz]

Hello,

I'm encountering a persistent issue when trying to restore a host backup made with proxmox-backup-client. While the backup verification process completes successfully and I can mount the backup without any apparent issues, the restore operation fails every time with a specific error.

Here is the error message I receive: Error: error extracting archive - encountered unexpected error during extraction: error reading pxar archive: unexpected offset 22538233598, smaller than already consumed payload 22538233614

This problem seems to affect only certain backups. However, once a backup for a specific host is affected, all subsequent backups for that same host also fail to restore with the same error.

I find it strange that the mount operation on the affected backup snapshots works perfectly fine. I can browse the file system and access the files, which forces me to manually copy the data over as a workaround instead of using the straightforward restore command.

Does anyone have an explanation for why mount would work while restore fails? This behavior seems inconsistent and makes me question the integrity of my backups, even when the verification process reports everything as "OK".

Here are the details of my setup:

• Proxmox Backup Server Version: 3.4.0
• Client: proxmox-backup-client (latest version)
• Backup Command: proxmox-backup-client backup files.pxar:/my/backup/path --backup-id myhost --repository myrepo --exclude=/backups --change-detection-mode=metadata
  

I'm trying to understand the root cause of this "unexpected offset" error. Has anyone encountered this before? Any insights into why this might be happening and how to permanently fix it would be greatly appreciated.

Thank you in advance for your help!

 

[Chris]

Hi,

Hello,

I'm encountering a persistent issue when trying to restore a host backup made with proxmox-backup-client. While the backup verification process completes successfully and I can mount the backup without any apparent issues, the restore operation fails every time with a specific error.

Here is the error message I receive: Error: error extracting archive - encountered unexpected error during extraction: error reading pxar archive: unexpected offset 22538233598, smaller than already consumed payload 22538233614

This problem seems to affect only certain backups. However, once a backup for a specific host is affected, all subsequent backups for that same host also fail to restore with the same error.

I find it strange that the mount operation on the affected backup snapshots works perfectly fine. I can browse the file system and access the files, which forces me to manually copy the data over as a workaround instead of using the straightforward restore command.

This indicates that the underlying backup snapshot is just fine, but there might be a bug in the sequential pxar decoder used for sequential restore operation.

Does anyone have an explanation for why mount would work while restore fails? This behavior seems inconsistent and makes me question the integrity of my backups, even when the verification process reports everything as "OK".

The mount command and single file restore use a slightly different random io pxar decoder, which does not seem to suffer from the same issue.

Here are the details of my setup:

• Proxmox Backup Server Version: 3.4.0
• Client: proxmox-backup-client (latest version)
• Backup Command: proxmox-backup-client backup files.pxar:/my/backup/path --backup-id myhost --repository myrepo --exclude=/backups --change-detection-mode=metadata

I'm trying to understand the root cause of this "unexpected offset" error. Has anyone encountered this before? Any insights into why this might be happening and how to permanently fix it would be greatly appreciated.

Please try restoring the snapshot with debug logs enabled by running PBS_LOG=debug proxmox-backup-client restore <snapshot> <archive-name> <retstore-target-path>. This should tell on which file the decoder fails during restore.

Also, download the raw ppxar and mpxar archive files via the WebUI and list the contents with debug information by invoking PXAR_LOG=debug pxar list <archive-name>.mpxar --payload-input <archive-name>.ppxar. This should dump all the archive contents including expected offsets in the payload archive.

Further, if possible, please provide us with the downloaded archive mpxar and ppxar files, so we can further investigate. Best is to share them via some private link, send to me via a direct message, thanks.

 

[Chris]

Thanks for sharing the logs via a private channel. While investigating, I recommend you create a new backup using the change detection mode data , so you break the chain of non-restorable backups. This will however re-read the full data contents and therefore take longer than the backups using change detection mode metadata.

 

[Chris]

If you can, please protect the last currently good and the first currently bad snapshot, so they don't get pruned.
Then sync them to a different namespace and try to create new backups with logging enabled using

PBS_LOG=debug proxmox-backup-client backup files.pxar:<source-path> --repository <your-repo> --ns <your-ns> --change-detection-mode=metadata 2>&1 | tee pbs-debug.log

.
Ideally one with the bad and one with the goodd snapshot as previous reference (by pruning the new and the bad one after creating the snapshot).
That should tell us in more details which files the client reuses and what offset ranges it calculates.

Can you also tell me how often backups were taken between 2025-08-14 22:48:41 and 2025-08-16 21:59:40? You stated that 2015-08-16 03:14:56 is still good and restorable.
From the pxar list with debug output logs of the snapshot created on 2025-08-24 09:10:16 you provided, we can see that there is an incorrectly encoded payload archive offset (last number, should always be greater than the previous one), which ultimately leads to the restore failing when using the sequential decoder.

-rw-r--r--   999/987       2025-08-14 22:48:41        0 "/world/region/r.-16.11.mca" 23262765092
Encountered padding of 18446744073709551600 bytes
-rw-r--r--   999/987       2025-08-16 21:59:40        0 "/world/region/r.-16.12.mca" 23262765092

Edit: Alos, you can attach the log outputs here in the forum, from the ones you provided offlist it does not seem that this contains sensitive information.

 

[yorilouz]

If you can, please protect the last currently good and the first currently bad snapshot, so they don't get pruned.
Then sync them to a different namespace and try to create new backups with logging enabled using

PBS_LOG=debug proxmox-backup-client backup files.pxar:<source-path> --repository <your-repo> --ns <your-ns> --change-detection-mode=metadata 2>&1 | tee pbs-debug.log

.
Ideally one with the bad and one with the goodd snapshot as previous reference (by pruning the new and the bad one after creating the snapshot).
That should tell us in more details which files the client reuses and what offset ranges it calculates.

How can I synchronize them to a different namespace to execute the tests you requested? Unfortunately, I don’t have the backup from August 24, 2025, at 9:10 AM, as it was pruned since our last conversation. However, I do have a failing backup that’s closer to the good one: August 17, 2025, at 3:14 AM. These backups are at a 24-hour interval, maybe it'll be easier for you to find the issue if backups are closer?

Can you also tell me how often backups were taken between 2025-08-14 22:48:41 and 2025-08-16 21:59:40? You stated that 2015-08-16 03:14:56 is still good and restorable.

Backups were taken on an hourly basis.

From the pxar list with debug output logs of the snapshot created on 2025-08-24 09:10:16 you provided, we can see that there is an incorrectly encoded payload archive offset (last number, should always be greater than the previous one), which ultimately leads to the restore failing when using the sequential decoder.

-rw-r--r--   999/987       2025-08-14 22:48:41        0 "/world/region/r.-16.11.mca" 23262765092
Encountered padding of 18446744073709551600 bytes
-rw-r--r--   999/987       2025-08-16 21:59:40        0 "/world/region/r.-16.12.mca" 23262765092

As evident from the restore logs I shared, the restore process was successfully completed without any reported errors. I attempted the restore process again today, and it also worked without any reported error from the restore command.

 

[Chris]

How can I synchronize them to a different namespace to execute the tests you requested?

Create a remote in Configuration > Remotes, using the PBS host as it's own remote by using localhost as host (and the fingerprint as found in Datastore > <your-datastore> > Summary > Show Connection Information). The create a new namespace via Datastore > <your-datastore> > Content > Add Namespace, create a sync job via Datastore > <your-datastore> > Sync Jobs > Add, selecting the reomte and target namespace you juist created and add a group filter to only sync the shnapshot group of interrest. After the sync, prune any snpshot not of interest the last good one and the first not good one.

Then use this namespace to create a new backup snapshot to it with the debug output enabled as mentioned above, followed by an immediate restore test (which should fail). Then repeat the same with the good snapshot as reference, by first pruning the other 2 snapshots first. The debug logs should tell us more about what condition leads to the incorrectly encoded payload offsets.

These backups are at a 24-hour interval, maybe it'll be easier for you to find the issue if backups are closer?

No, frequency does not matter, the question was rather if I could expect the debug log outputs of the 2 states you shared to be consecutive or not, which is however not the case, as you stated they were done hourly.

As evident from the restore logs I shared, the restore process was successfully completed without any reported errors. I attempted the restore process again today, and it also worked without any reported error from the restore command.

But this is not a restore, this is a dump of the metadata as encoded in the archive, the actual sequential restore will fail, as you mentioned in your initial post?

 

[yorilouz]

Create a remote in Configuration > Remotes, using the PBS host as it's own remote by using localhost as host (and the fingerprint as found in Datastore > <your-datastore> > Summary > Show Connection Information). The create a new namespace via Datastore > <your-datastore> > Content > Add Namespace, create a sync job via Datastore > <your-datastore> > Sync Jobs > Add, selecting the reomte and target namespace you juist created and add a group filter to only sync the shnapshot group of interrest. After the sync, prune any snpshot not of interest the last good one and the first not good one.

I’ve sent you the debug logs for the backups performed after the valid snapshot and after the failed one. Since the logs are quite verbose, I’ve opted to send them privately to you in case I’ve inadvertently omitted to remove any sensitive information.

But this is not a restore, this is a dump of the metadata as encoded in the archive, the actual sequential restore will fail, as you mentioned in your initial post?

Actually, sorry, I got confused. I thought you were talking about the backup from 2015-08-16 03:14:56.

 

[Chris]

Did you also double check that the restore fails (with debug log enabled) and if so at what file? Makes it easier to find a possible mismatch, thanks!

 

[yorilouz]

Yes, I've double-checked and just sent you all the log files for that backup.

 

[Chris]

Sorry if there was a misunderstanding!

What i meant for you do do is to create debug output for backup and restore of the NEW backup snapshots, based on the 2 previous states... Only by that we can directly compare the outputs. From the restore debug log we can learn were the corruption is and correlate that to the debug output during backup creation, showing what conditions lead to the incorrect encoding. Further, if any of the new snapshots does not fail during restore, it is of no interest for further investigation.

May I therefore kindly ask for you to repeat this again, but this time collect both backup and restore debug output for the new snapshot. Based on the data you provided so far I was not able to reproduce this issue or narrow it down to a particular code path.

 

[yorilouz]

Sorry for the misunderstanding on my end. I've now done what you asked.

I isolated the two previous backup states—the one that worked and the one that failed—in a new, separate namespace. Then, I created a new backup snapshot from each of them, using the current production data. Finally, I attempted to restore these newly created snapshots.

Surprisingly, both restores were successful.

I also tried restoring the latest backup from our production namespace, and that worked as well.

This has me wondering: is it possible that the production data has changed? The application is live, and data is modified over time. Perhaps the specific file that was causing the corruption has been altered or updated, which in turn resolved the restoration issue. I'm a bit confused because that's the only thing I can think of that has changed.

So, on one hand, this is good news because it means our recent backups are restorable. On the other hand, I'm still concerned about what caused the initial problem. It would be great if we could figure out the root cause to prevent this from happening again in the future.

Let me know what you think.

PS: I've sent you log files for backup and restore operations on backups isolated in a test namespace.

 

[Chris]

I also tried restoring the latest backup from our production namespace, and that worked as well.

Yes, the state of the source data as well as the previous backup snapshot used as reference must be in a specific state to trigger the exact code path leading to this encoding issue. In a highly dynamic environment which a lot of files and folders changing it is not surprising to me that you now are not able to reproduce. That is also why I was not able to reproduce the issue with the provided logs, since that state does not trigger the issue.

I've send a patch which should act as stop gap for the encoder issue (applied but not packaged yet at the time of writing), leading to backups fail if this is the case, see https://lore.proxmox.com/pbs-devel/20250827094633.239975-1-c.ebner@proxmox.com/T/. Will continue to try and reproduce given the information at hand.

Thanks a lot for your efforts!

 

[yorilouz]

Thank you very much for your support and for the attentive follow-up on this issue. I completely understand the difficulty in reproducing it in such a dynamic environment.

On my end, I will remain vigilant. If the problem occurs again, I won't hesitate to get back to you with as much information as possible to help find a permanent solution.