Restore extremly slow

[Stefan_Malte_Schumacher]

Hello,

I am running PVE7.0 connect via 10Gbits to BackupServer 2.0. In my experiences restoring volumes ist extremely fast. Even VM on thin_volumes were restored in a matter of minutes. Now I am trying to restore a VM (thin, 500GB) and it takes forever. This is the output so far, I will post the final result later this evening or even tomorrow.
Of course there are only two questions to ask: Why is it so slow and how do i fix it?

Task viewer: VM 117 - Restore

OutputStatus

Stop
WARNING: You have not turned on protection against thin pools running out of space.
WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
Logical volume "vm-117-disk-1" created.
WARNING: Sum of all thin volume sizes (<5.15 TiB) exceeds the size of thin pool hw_raid/hw_thinpool and the size of whole volume group (<4.37 TiB).
new volume ID is 'hw_thinpool:vm-117-disk-1'
restore proxmox backup image: /usr/bin/pbs-restore --repository 1x1consult@pbs@pbs01.inet.de:1x1consult vm/116/2021-08-11T22:03:03Z drive-scsi0.img.fidx /dev/hw_raid/vm-117-disk-1 --verbose --format raw --skip-zero
connecting to repository '1x1consult@pbs@pbs01.inet.de:1x1consult'
open block backend for target '/dev/hw_raid/vm-117-disk-1'
starting to restore snapshot 'vm/116/2021-08-11T22:03:03Z'
download and verify backup index
progress 1% (read 5368709120 bytes, zeroes = 63% (3388997632 bytes), duration 42 sec)
progress 2% (read 10737418240 bytes, zeroes = 37% (4051697664 bytes), duration 616 sec)
progress 3% (read 16106127360 bytes, zeroes = 27% (4458545152 bytes), duration 963 sec)
progress 4% (read 21474836480 bytes, zeroes = 34% (7507804160 bytes), duration 1028 sec)
progress 5% (read 26843545600 bytes, zeroes = 47% (12691963904 bytes), duration 1031 sec)
progress 6% (read 32212254720 bytes, zeroes = 43% (14008975360 bytes), duration 1275 sec)

Yours sincerely
Stefan

 

[dietmar]

Hello,

I am running PVE7.0 connect via 10Gbits to BackupServer 2.0.

What kind of storage do you use at the backup server?

 

[Stefan_Malte_Schumacher]

WARNING: You have not turned on protection against thin pools running out of space.
WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
Logical volume "vm-117-disk-1" created.
WARNING: Sum of all thin volume sizes (<5.15 TiB) exceeds the size of thin pool hw_raid/hw_thinpool and the size of whole volume group (<4.37 TiB).
new volume ID is 'hw_thinpool:vm-117-disk-1'
restore proxmox backup image: /usr/bin/pbs-restore --repository 1x1consult@pbs@pbs01.inet.de:1x1consult vm/116/2021-08-11T22:03:03Z drive-scsi0.img.fidx /dev/hw_raid/vm-117-disk-1 --verbose --format raw --skip-zero
connecting to repository '1x1consult@pbs@pbs01.inet.de:1x1consult'
open block backend for target '/dev/hw_raid/vm-117-disk-1'
starting to restore snapshot 'vm/116/2021-08-11T22:03:03Z'
download and verify backup index
progress 1% (read 5368709120 bytes, zeroes = 63% (3388997632 bytes), duration 42 sec)
progress 2% (read 10737418240 bytes, zeroes = 37% (4051697664 bytes), duration 616 sec)
progress 3% (read 16106127360 bytes, zeroes = 27% (4458545152 bytes), duration 963 sec)
progress 4% (read 21474836480 bytes, zeroes = 34% (7507804160 bytes), duration 1028 sec)
progress 5% (read 26843545600 bytes, zeroes = 47% (12691963904 bytes), duration 1031 sec)
progress 6% (read 32212254720 bytes, zeroes = 43% (14008975360 bytes), duration 1275 sec)
progress 7% (read 37580963840 bytes, zeroes = 38% (14378074112 bytes), duration 1543 sec)
progress 8% (read 42949672960 bytes, zeroes = 36% (15783165952 bytes), duration 2010 sec)
progress 9% (read 48318382080 bytes, zeroes = 43% (21135097856 bytes), duration 2010 sec)
progress 10% (read 53687091200 bytes, zeroes = 49% (26495418368 bytes), duration 2010 sec)
progress 11% (read 59055800320 bytes, zeroes = 53% (31851544576 bytes), duration 2010 sec)
progress 12% (read 64424509440 bytes, zeroes = 57% (37211865088 bytes), duration 2010 sec)
progress 13% (read 69793218560 bytes, zeroes = 60% (42567991296 bytes), duration 2010 sec)

restore image complete (bytes=536870912000, duration=2014.28s, speed=254.19MB/s)
rescan volumes...
VM 117 add unreferenced volume 'hw_thinpool:vm-117-disk-0' as 'unused0' to config
TASK OK
----------------------------------------------
I shortened the log output to increase the readability of this posting. "Duration 2010 sec" was repeated 93 times.
I will ask my provider what storage they use and and post the result.

 

[Stefan_Malte_Schumacher]

I had a memory malfunction: The Backup is not running on dedicated storage unit but on a normal supermicro 19"-Server with a hardware RAID5 consistings of ST2400MM0129 -Drives (Seagate Enterprise Performance 10K 2,4TB, 12Gbits SAS). To my knowledge I am the only one using it at the moment (Proxmox and the Backup Server are "new" at my ISP). The Restore should be faster, shouldn't it? vm117 is a thin volume and uses only 11GB of 493GB. Why would it take more than a few seconds to restore it?

 

[JamesT]

I'm having a somewhat similar issue. Did you find out what was wrong @Stefan_Malte_Schumacher ?

In my case, 4 VMs restored very quickly and the last one which is bigger started off OK then ground to a halt.
https://forum.proxmox.com/threads/restore-from-pbs1-0-5-to-pve7-0-11-ground-to-halt-very-slow.95116/

 

[Stefan_Malte_Schumacher]

No, unfortunately not, but I did not really have time for more restore attempts. I also would very much like to see this issue resolved.
Of one thing I am sure: Considering the hardware and network connection I am using the restore should have been much faster and has been much faster before on Backup Server 1.