Restore backup failure (zstd decompression)
- Thread starter zInvalid
- Start date
Can you guys please take a look at this thread and help? https://forum.proxmox.com/threads/c...ckup-with-clean-logs-can-t-restore-it.161337/ Seems to be similar problem.
An interesting observation, we were able to reproduce several times.
Scenario:
- two proxmox environments at remote locations from each other, making backups to NFS storage (Synology NASes)
- locations connected to each other with site-to-site VPN
- connectiong the "CIFS remote folder" (as it's called in Synology) share of the NAS containing the "dump" from one location to the NAS on the other location, allows direct filesystem access to the *.vma.zst files.
- copying the backup files directly between the NASes manually, using for example the "File station" utility of it, happens with about 30MB/s which is good enough for this occasional use case.
And what happens is:
- if I copy the *.log + *.vma.zst + *.vma.zst.notes from the "dump" directory of the source NAS to the "dump" directory of the target NAS, when the copy reaches about 99.7% stalls about half a minute, but apparently finishes without any error.
- trying to restore from this file on the target system will halt with corrupted error message, after the number of GB corresponding to the used space in the virtual disk to be restored (this is what made me thinking...)
- unzstd --test on the source system is without problem, but unzstd --test on the target fails, thus it's very likely that the curruption occurs by the end of the file transfer. md5sums don't match, of course.
But:
- if I copy the *.log + *.vma.zst + *.vma.zst.notes from the "dump" directory of the source NAS to some other "temp" directory manually created (in the same share) of the target NAS, copy doesn't stall at end, and also seems to finish without any error.
- once copy completes, move the files on the target NAS from "temp" to "dump"
- md5sum will be OK, and restore from the backup will be also OK!
Assumption:
- don't copy the backups directly to the "dump" directory of the target NAS, because proxmox seems to try to open/access the files while the copying is not yet finished, thus corruption occurs in the backup file
- copy to a "temp" dir first and after ensured that copy is done, move the files in "dump" directory. That will be fast as the files are in the same share/volume.
- I guess this could be true in other scenarios too. Not sure if this is how the handling of the backup files is supposed to be done, but if done like above, the behavior is consistent.
Scenario:
- two proxmox environments at remote locations from each other, making backups to NFS storage (Synology NASes)
- locations connected to each other with site-to-site VPN
- connectiong the "CIFS remote folder" (as it's called in Synology) share of the NAS containing the "dump" from one location to the NAS on the other location, allows direct filesystem access to the *.vma.zst files.
- copying the backup files directly between the NASes manually, using for example the "File station" utility of it, happens with about 30MB/s which is good enough for this occasional use case.
And what happens is:
- if I copy the *.log + *.vma.zst + *.vma.zst.notes from the "dump" directory of the source NAS to the "dump" directory of the target NAS, when the copy reaches about 99.7% stalls about half a minute, but apparently finishes without any error.
- trying to restore from this file on the target system will halt with corrupted error message, after the number of GB corresponding to the used space in the virtual disk to be restored (this is what made me thinking...)
- unzstd --test on the source system is without problem, but unzstd --test on the target fails, thus it's very likely that the curruption occurs by the end of the file transfer. md5sums don't match, of course.
But:
- if I copy the *.log + *.vma.zst + *.vma.zst.notes from the "dump" directory of the source NAS to some other "temp" directory manually created (in the same share) of the target NAS, copy doesn't stall at end, and also seems to finish without any error.
- once copy completes, move the files on the target NAS from "temp" to "dump"
- md5sum will be OK, and restore from the backup will be also OK!
Assumption:
- don't copy the backups directly to the "dump" directory of the target NAS, because proxmox seems to try to open/access the files while the copying is not yet finished, thus corruption occurs in the backup file
- copy to a "temp" dir first and after ensured that copy is done, move the files in "dump" directory. That will be fast as the files are in the same share/volume.
- I guess this could be true in other scenarios too. Not sure if this is how the handling of the backup files is supposed to be done, but if done like above, the behavior is consistent.
Last edited: