Abysmally slow restore from backup

_gabriel said:
That's a big patch ! your binary size is 187 MB where original is 6.8 MB put me off to try.
Click to expand...
Ah, good catch. That's the binary with debug info and without stripping. I noticed that there is a path: libproxmox-backup-qemu0-1.5.1/debian/libproxmox-backup-qemu0/usr/lib/libproxmox_backup_qemu.so.0 as well which seems to contain a binary of the size you describe, I was taking the one in libproxmox-backup-qemu0-1.5.1/target/release/libproxmox_backup_qemu.so

Here is the smaller [6.8 MB] version without debug info and stripped : https://notnullmakers.com/public/media/libproxmox_backup_qemu.so_09062025_mini
the SHA256 is: b6c27033e46b965d7b5ce8aaab1208fa76d6601697f72e8a87ba118ec2bed9f3

Again, of course I recommend you do a backup of the original /usr/lib/libproxmox_backup_qemu.so.0 binary and check e.g. with VirusTotal or other tools.

Thank you for spotting this. If somebody from the Proxmox team could clarify what is what that would be nice.
 
Last edited:
  • Like
Reactions: _gabriel
_gabriel said:
is local restore can be improved by your patch ?
I mean PBS installed alongside PVE where network is over localhost ?
Click to expand...
I don't think you will see a big jump in performance – the latency is usually much lower on the same machine. Of course you can try to launch PBS in a different namespace and add some latency on the veth connection between namespaces or you can probably just outright add the latency to the interface the PBS is using. This should give you an idea on what I mean: https://github.com/NOT-NULL-Makers/resources/blob/main/Installfest2024/krejzy-veci-s-SSH.md
 
Last edited:
  • Like
Reactions: Johannes S
_gabriel said:
That's a big patch ! your binary size is 187 MB where original is 6.8 MB put me off to try.
Click to expand...
I find it also highly off putting that he doesn't publish his source patch somewhere. This would allow to review the changes before applying instead of just hoping that no bad actor did a "man-in-the-middle-attack" on his webpage. Because one might trust @kaliszad ( he seems to be helpful and nice to be around with) but there might still be a risk he isn't even aware of or responsible (man-in-the-middle, a compromise on his server with the binary etc pp) for.


But even more off putting is that part of the community want to try it nontheless: I hope they use the patched binary only on test data and nothing important. Seriously folks, it's not like in Veeam or VMWare forums where a binary file from a staff account can be trusted to a certain extent (as long as the company isn't compromised).
On a open source software in contrast the only way to know whether you can trust the patch from a community member to have the source code.
If (like me) you are not much of a programmer you could still use it if the software allows reproducible builds (which sadly often enough is not the case although it's a lot better than five or ten years before): https://reproducible-builds.org/

Edit: I see now that @fabian actually already hinted on this. But I'm wondering now whether Proxmox developers already thought about providing support for reproducible builds of the software (I mean the part developed by Proxmox Server Solutions, for the rest they are depending on upstream (Debian, Ceph etc)?
 
Johannes S said:
Edit: I see now that @fabian actually already hinted on this. But I'm wondering now whether Proxmox developers already thought about providing support for reproducible builds of the software (I mean the part developed by Proxmox Server Solutions, for the rest they are depending on upstream (Debian, Ceph etc)?
Click to expand...

it would require some infrastructure changes that are on our todo list, but haven't been implemented yet. we do make use of the fact that most of our packages are (mostly) reproducible already when doing packaging changes :)
 
  • Like
Reactions: Johannes S
fabian said:
it would require some infrastructure changes that are on our todo list, but haven't been implemented yet. we do make use of the fact that most of our packages are (mostly) reproducible already when doing packaging changes :)
Click to expand...
That's great news, thanks for the insights. I know that such stuff isn't as easy as it sounds after watching the talks from the Debian guys on reproducible builds, so I highly appreciate that you have this on your agenda
 
Last edited:
fabian said:
your intentions are probably/hopefully good, but I can only strongly discourage everybody from installing random binary artifacts users post here.

@kaliszad if you want to get your changes reviewed/integrated/.., please submit them as patches:

https://pve.proxmox.com/wiki/Developer_Documentation#Preparing_Patches
https://pve.proxmox.com/wiki/Developer_Documentation#Software_License_and_Copyright

https://bugzilla.proxmox.com/show_bug.cgi?id=3163
Click to expand...
Yes, we might publish the patches if our client on which behalf we have done the changes agrees with that/ is interested in that and is willing to sponsor additional time.

I would also like to note that people didn't bother to do proper performance debugging since at least November 2020 (considering the posting of the bug by Nico Fellner) instead jumping to false conclusions. Even Proxmox support that our client contacted wasn't helpful in this regard instead looking for excuses and talking about a possible migration to http/3, which wasn't even the problem at all.

So for now you have a large binary with debug info and a small one without debug info and that's what people can use to do their own testing in their own testing infrastructure on testing data.
 
kaliszad said:
Ah, good catch. That's the binary with debug info and without stripping. I noticed that there is a path: libproxmox-backup-qemu0-1.5.1/debian/libproxmox-backup-qemu0/usr/lib/libproxmox_backup_qemu.so.0 as well which seems to contain a binary of the size you describe, I was taking the one in libproxmox-backup-qemu0-1.5.1/target/release/libproxmox_backup_qemu.so

Here is the smaller [6.8 MB] version without debug info and stripped : https://notnullmakers.com/public/media/libproxmox_backup_qemu.so_09062025_mini
the SHA256 is: b6c27033e46b965d7b5ce8aaab1208fa76d6601697f72e8a87ba118ec2bed9f3

Again, of course I recommend you do a backup of the original /usr/lib/libproxmox_backup_qemu.so.0 binary and check e.g. with VirusTotal or other tools.

Thank you for spotting this. If somebody from the Proxmox team could clarify what is what that would be nice.
Click to expand...

Applied the patch and didn't see any difference. Maybe I installed it wrong?
Bash: 
cp /usr/lib/libproxmox_backup_qemu.so.0 proxmox_backup.bak
wget https://notnullmakers.com/public/media/libproxmox_backup_qemu.so_09062025_mini
cp libproxmox_backup_qemu.so_09062025_mini /usr/lib/libproxmox_backup_qemu.so.0

Also verified the sha1 was the same after install
1749654743793.png

Pre-install
1749654213782.png

Post-install
1749654279340.png

I've confirmed the storage can write at over 3.5GB/s, so no issue there.
 
PwrBank said:
Applied the patch and didn't see any difference. Maybe I installed it wrong?
Bash: 
cp /usr/lib/libproxmox_backup_qemu.so.0 proxmox_backup.bak
wget https://notnullmakers.com/public/media/libproxmox_backup_qemu.so_09062025_mini
cp libproxmox_backup_qemu.so_09062025_mini /usr/lib/libproxmox_backup_qemu.so.0

Also verified the sha1 was the same after install
View attachment 87020

Pre-install
View attachment 87018

Post-install
View attachment 87019

I've confirmed the storage can write at over 3.5GB/s, so no issue there.
Click to expand...
I am at a conference until Friday so I have no chance to look into this in detail. The speedup should be visible if your backups are on a remote Proxmox Backup Server and you restore from the Proxmox Virtual Environment. That will trigger the qmrestore tool which will under the hood call pbs-restore.
 
You are seriously using a "patch" from some third party on the internet without any ability to verify that it doesn't contain malicious code or a backdoor? I hope you don't do this on systems with real life data or production loads. I'm also hoping I'm not somehow using your services
 
Last edited:
Johannes S said:
You are seriously using a "patch" from some third party on the internet without any ability to verify that it doesn't contain malicious code or a backdoor? I hope you don't do this on systems with real life data or production loads. I also hope I'm not somehow using your services
Click to expand...

Good thing I'm running this in a test environment, yeesh.
kaliszad said:
I am at a conference until Friday so I have no chance to look into this in detail. The speedup should be visible if your backups are on a remote Proxmox Backup Server and you restore from the Proxmox Virtual Environment. That will trigger the qmrestore tool which will under the hood call pbs-restore.
Click to expand...
Yup. A PVE server and a PBS connected over a 25GbE switch.
 
  • Like
Reactions: Johannes S
You must log in or register to reply here.
Top Bottom