Extremely slow & inefficient restore from block device backup

[hilli97]

Hi Guys,
We have a large (4TB) VM backed up to a bare metal PBS server running on NVME drives.
When we map a backup on the PVE host from PBS using proxmox-backup-client to restore specific files/folders, we encounter large network utilisation with extremely low efficiency.
IE When restoring 26GB of files, we'll get ~12-20MB/s disk write on PVE with network transfer from PBS of ~200MB/s.
In the end we saw roughly 354GB move across the network between the two servers for only 26GB of actual data restored.
These servers were doing nothing else during this test, so the usage was strictly the restore.

Is this normal? Is there any way to increase the efficiency? Or is this inherent behaviour in restoring individual files from a block device backup?

Cheers in advance

 

[l.leahu-vladucu]

Hello hili97! Just to check: are you sure that you are using MB/s for both speeds, and not Mbit/s? This would partially explain the large speed difference.

However, either way, if you are only restoring specific files/folders, there can be some inefficiencies if the files are very small. PBS currently uses 4 MB chunks to improve speed, which is great if you are transferring large files, but can cause more network traffic for small files. While PBS also uses a cache trying to prevent this, the cache might not be large enough and might get invalidated too often in your case, leading to the issues you are describing. In other words, PBS tries to do its best to avoid such situations, but depending on what you are trying to restore, this can lead to a worst case scenario where multiple blocks are copied multiple times.

So my question is: are you trying to restore a lot of very small files?

 

[hilli97]

Heya,
Yep all MB/s in capitals not bits, used the same metric to make it easier.
I think you might have hit the nail on the head there. There are 278k files in that 26GB, Which would make the theoretical average file size of 90kb.
I guess that is significantly increasing the transfer size if it has to transfer 4MB chunks for each one.

Curiously though, i created a new block storage device on the VM that was only 40GB and put the 26GB of files on there just to see if there was much difference.
We have almost 90% efficiency on transfer with that one from the PBS so much much better. It could be a combination of (as you say) the small file size/large chunk size, and the fragmentation of the very large drive size (4TB). I wonder if there is much caching of the chunks on the client end, that way if it reads a 4MB chunk for a 90kb file, the contents of the remaining chunk might be used for the next file on a less fragmented drive. Would go a long way to explaining why the smaller block device was so much more efficient, it could keep alot of the unused chunks in memory and not re-download them till the restore is complete...

I did Check and the fragmentation on the 4TB block device was only 0.3% so not enough to make a difference imo.