ZSTD backups always corrupted

[scintilla13]

In this forum, there are many cases of backup corruption related to ZSTD. The discussion has never gone deep.

All my backups on different simple XFS volumes are corrupted. This has never happened in the past with gzip or lzo.

Maybe this page could suggest that ZSTD is unsafe to be used:
https://github.com/facebook/zstd/issues/2852

 

[LnxBil]

In this forum, there are many cases of backup corruption related to ZSTD. The discussion has never gone deep.

Which one? I wasn't able to find such a discussion. Please provide a link.

All my backups on different simple XFS volumes are corrupted.

Could you please provide more information? for example log ouput from backup and the restore process.

 

[scintilla13]

Thanks for your answer.
Those are three forum examples:

https://forum.proxmox.com/threads/cant-restore-vm.97633/#post-437269
https://forum.proxmox.com/threads/problems-at-restoring-a-vm.93367/
https://forum.proxmox.com/threads/unpacking-vm-backup-in-ztsd-format-error.75706/

Of course, technology is stratified and complex and the fault could be elsewhere, anyway is curious that this occurs only with ZTSD compression.

I'm going to gather the rest of the data.

 

[LnxBil]

Thanks for your answer.
Those are three forum examples:

https://forum.proxmox.com/threads/cant-restore-vm.97633/#post-437269
https://forum.proxmox.com/threads/problems-at-restoring-a-vm.93367/
https://forum.proxmox.com/threads/unpacking-vm-backup-in-ztsd-format-error.75706/

Thank you and yes, there could be a thousand problems be responsible for the corruption, including zstd. Post mortem is nothing else you can do about it, therefore there was no further investigation.

Of course, technology is stratified and complex and the fault could be elsewhere, anyway is curious that this occurs only with ZTSD compression.

That could be the case. One would want hard numbers to back this up including a MWE (minimal working example) to replicate the issue. Best is to always check the backup that was written by extracting the image itself after backup. We do this regularely to check if the backup worked. In other backup mechanisms this is called validation and should always be done. There was also a feature request about this and one answer was to use the proxmox backup server, which already does this implicitely.

Maybe this page could suggest that ZSTD is unsafe to be used:
https://github.com/facebook/zstd/issues/2852

Of course this could be also the case here, as is with all other external compression tools. Maybe the proxmox staff can shed some light on that if they were able to replicate the problem. There is currently no bugreport about this in bugzilla that matches the problem exactly.

 

[fabian]

In this forum, there are many cases of backup corruption related to ZSTD. The discussion has never gone deep.

All my backups on different simple XFS volumes are corrupted. This has never happened in the past with gzip or lzo.

Maybe this page could suggest that ZSTD is unsafe to be used:
https://github.com/facebook/zstd/issues/2852

no, that page just says "it's possible to have a bug that results in corrupt zstd data" (and the same is true for any other data/compression format other than zstd+vma), not "there is a bug in zstd that causes corrupt archives".