Abysmally slow restore from backup

[damarrin]

Hello,

My woes with my HP Microserver Gen8 continue (here's my previous thread, now thankfully solved... mostly), as I'm now getting terrible performance with backups.

I'm restoring backups to lvm-data that's located on a 7200rpm HDD. The SATA controller is this:

00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller (rev 05)

The backups are stored on PBS connected with gigabit ethernet. The restore starts fine, with write performance around 20-30 MB/s, which is what I'd expect. After a few minutes, write speed falls to single megabytes, at times as low as a few kilobytes, as measured by iotop.

I have ipire in a vm (with virtio-scsi-single and iothread and aio=threads set as per the other thread) as a router and pihole in a container as a DNS server running on this machine as well. Ipfire continues working, but pihole running in the container becomes unresponsive and I basically get no internet.

As a result, a backup that took 5 minutes to make (my PBS is running on an ancient Core 2 Quad machine with a couple of WD 7200 HDDs and everything is super fine there) will need an hour to restore on this machine. To boot, after it reports 100% restored, it sits frozen for another 10 minutes before TASK OK appears. During all this time, IO delay is very high (60-90%). Presumably, it continues writing to the HDD from cache. Once that's done, VMs and containers become responsive again.

I guess it must be a problem with how the SATA controller is configured by Proxmox. I will be very grateful for ideas I could try to make it perform well. Thanks.

 

[tonci]

Hello , this is my post regarding this subject: https://forum.proxmox.com/threads/backup-speed-vs-restore-speed.106873/#post-466581
My general conclusion was:
- backup speed is very satisfactory - 90% saturation of 1G net work and this is considered expected ... no matter backing up 1 VM or more VMs from more hosts
- restore speed of 1VM goes down to 20%+ (cca 200Mbs+ ) , BUT ... if we restore more VMs in the same time (no matter restoring to the same host or to more hosts in cluster) newtork link wil be satureted up to 80-90% and no vm-restore affects/slows down any other ..
Conclusion: PBS can serve out 4 concurrent restores occupying 1G link at 90% , BUT when restoring one VM restore speed cannot get faster but 200+Mbs
Literaly, when restoring 4 VMs , PBS reads 4 times more chunks in the same time than when restoring 1 VM ... with the same sata 7200 disks.

How come that raid10 (4 x 2T sata 7200rpm) is behaving very slow when restoring 1VM, but suddenly very fast when restoring 4 VMs at the same time , saturating whole bandwidth so that we can say that bottleneck is network itself and not slow sata disks trying to read many little chunks ?

Involving metadata special cache(disks) did not help much ...

At the end , we are facing system that is (much) faster in writting than reading which is not that usual

I think there is still space for design-improvement ... or I'm very wrong regarding this experience and description?

 

[rontex]

I use the latest version of Proxmox 8.1.4 and server backup 3.1. I encountered a similar problem.
On a machine with virtual machines I use 6 nvme in raid 6 mdadm, on a backup server 36 SAS raidz2.
Also the biggest problem is the low recovery speed of one virtual machine (network speed is approximately 40Mb/sec).
But if you run 6 simultaneous virtual machines, you can get a speed of 200 MB/sec. Network between backup
server and cluster - 10G. Network latency does not exceed 3ms.
Is there a solution to the problem with the low recovery speed of one VM?

 

[lucius_the]

I believe there are some serious issues in how PBS is working. I was testing a restore from PBS to PVE today (10 Gbps LAN link / low-latency, NVME is used on both sides -> restore speed is barely 60 MB/s). Backup is faster than this. Verify is faster than this. Tested with proxmox-backup-client benchmark on both sides, everything should be faster. storage is not the bottleneck, CPU is not the bottleneck, network also. Sp I have no explanation besides: something is wrong with PBS. PBS restore speed is not acceptable. I will try Veeam B&R on same hardware and see what I'll get.