Terrible restore speed due to index verification

[AraToken]

Hello,

while restoring a vm backup over the pbs i noticed that the restore took a huge amount of time, multiple hours in fact which would be catastrophic for a productive system that needs to be restored in case of sudden crashes. So i tried to look into the possible reasons as of why the restore takes so long.
The VM itself uses a disk around 300GB which of course is not really small. The datastore on the pbs is a zfs using multiple HDDs. Replacing them with SSDs would make the whole thing run faster, i know that too. The server is also connected with a gigabit to the network. Upgrading to 10G would be better aswell. So far all points that make sense to me.

However, what kinda bugs me is the process of the restore itself. While reading the backup from the pbs already took around 2 hours, after reaching 100% read progress he immediately started with "downloading and verifying backup index" which again took 2 more hours until finally finishing the restore job.
And that's what does not make any sense to me.

My question is now: where does this verification come from? Is that something that the PVE server does on its own or does it come from the PBS aswell?
I also want to clarify that the PBS used is currently a test system, which means there are as for now no verify jobs configured and all backups on it are "verify state none". Is it possible that PVE/PBS does not restore a machine from a backup that never got verified? If so: is there a way to force the restore even without verification?

Or is there maybe a completely different reason for this behaviour?
Thank you in advance for answers!

Current PVE version: 7.2-1
Current PBS version: 2.0-7

 

[shanreich]

PBS client has a built-in benchmark tool that you can use to check your maximum throughput - you can run it via proxmox-backup-client benchmark. You should run this on the instance where you want to restore the backup as well as the PBS server itself. This might give an indication for why it is taking so long. Usually the CPU is the bottleneck since it needs to decompress the Backup or the Storage, because it cannot output data fast enough for the CPU. The important factor for storage is the read speed for random access, do you know how fast your storage is for random access scenarios?

There currently is no way to skip the verification process, since it is an inherently important part of restoring the backup. It wouldn't be much use if you restored a faulty backup and a lot can happen during the transit.

 

[AraToken]

Thank you for your response!
I was not aware of the benchmark tool but will definetely use that in the future. Thank you!

So verification is definetely needed, no matter if the backup itself got verified on the PBS beforehand because the PVE checks for errors during the transport, correct? In that case i assume it's just unfortunate that it took so long and we definetely need to look into hardware and network upgrades for the future.

 

[fabian]

could you post the full restore log please?

 

[AraToken]

Hi fabian!

Here's the restore log from the PVE as attachment.

Edit: During the restore i noticed some steps taking especially long, for example:

progress 26% (read 41875931136 bytes, zeroes = 5% (2306867200 bytes), duration 632 sec)
progress 27% (read 43486543872 bytes, zeroes = 5% (2327838720 bytes), duration 1595 sec)

 

[fabian]

like I expected - it doesn't start over, your VM has two disks (and each of those is represented on the PBS side as an index file).

the question now is where your bottle neck is - even just watching with something like atop on both ends during a restore might help to shed some light.