[SOLVED] Restore is so slow on remote servers

[Skyfay]

Hey,

My setup consists of having PVE at location 1 and PBS at location 2. The backups are working fine but today I had to do a restore:

restore image complete (bytes=42949672960, duration=6725.02s, speed=6.09MB/s)

(2h for a 40GB disk)

The latency of the servers is approx. 12 ms and one side has a 10 Gbit/s and the other a 1 Gbit/s connection.
I have found the following article:

J

Thread 'Slow restore performance'

Jun 3, 2024

Currently have proxmox 8.2 and PBS 3.2, backup of a 1.65TiB takes around 1 hour (which is pretty fast).

But restore took us 14 hours....which is very long.

You can find attached image of a PBS benchmark:


Machine specs:

i9-10940X CPU @ 3.30GHz CPU @ 3.3GHz
128G DDR4 3000MT/s
4TiB P2 NVMe PCIe SSD
Upload speed is ~60MB/s

Below you can find fio random read benchmark:

sudo fio --filename=/dev/nvme0n1 --direct=1 --rw=randread --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --readonly
...

• JustThat
• Replies: 29
• Forum: Proxmox Backup: Installation and configuration

So is it really the case that this setup does not work with a remote PBS because the restore simply takes forever?

 

[Chris]

The latency of the servers is approx. 12 ms and one side has a 10 Gbit/s and the other a 1 Gbit/s connection.

I assume these bandwidth values are the maximum link speed, but what about the real world throughput you expect from the connection? Did you already try to benchmark your connection speed with e.g. iperf? Further, did you also benchmark your PBS storage? What storage layout are you using for the datastore backing storage?

 

[Skyfay]

I assume these bandwidth values are the maximum link speed, but what about the real world throughput you expect from the connection? Did you already try to benchmark your connection speed with e.g. iperf?

Okay the connection was over a relay

Accepted connection from 100.100.90.1, port 47010
[  5] local 100.100.70.60 port 5201 connected to 100.100.90.1 port 47012
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  2.59 MBytes  21.7 Mbits/sec                 
[  5]   1.00-2.00   sec  2.54 MBytes  21.4 Mbits/sec                 
[  5]   2.00-3.00   sec  3.01 MBytes  25.3 Mbits/sec                 
[  5]   3.00-4.00   sec  2.64 MBytes  22.2 Mbits/sec                 
[  5]   4.00-5.00   sec  2.35 MBytes  19.7 Mbits/sec                 
[  5]   5.00-6.00   sec  2.38 MBytes  20.0 Mbits/sec                 
[  5]   6.00-7.00   sec  2.44 MBytes  20.4 Mbits/sec                 
[  5]   7.00-8.00   sec  2.29 MBytes  19.2 Mbits/sec                 
[  5]   8.00-9.00   sec  2.45 MBytes  20.5 Mbits/sec                 
[  5]   9.00-10.00  sec  2.35 MBytes  19.7 Mbits/sec                 
[  5]  10.00-10.05  sec   140 KBytes  23.1 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.05  sec  25.2 MBytes  21.0 Mbits/sec

So here is the new test without:

Connecting to host 100.100.70.60, port 5201
[  5] local 100.100.90.1 port 45498 connected to 100.100.70.60 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   100 MBytes   840 Mbits/sec    0   3.54 MBytes       
[  5]   1.00-2.00   sec   106 MBytes   891 Mbits/sec    0   3.54 MBytes       
[  5]   2.00-3.00   sec   105 MBytes   881 Mbits/sec    0   3.54 MBytes       
[  5]   3.00-4.00   sec   106 MBytes   891 Mbits/sec    0   3.54 MBytes       
[  5]   4.00-5.00   sec   105 MBytes   881 Mbits/sec    0   3.54 MBytes       
[  5]   5.00-6.00   sec   105 MBytes   881 Mbits/sec    0   3.54 MBytes       
[  5]   6.00-7.00   sec   106 MBytes   891 Mbits/sec    0   3.54 MBytes       
[  5]   7.00-8.00   sec   105 MBytes   881 Mbits/sec    0   3.54 MBytes       
[  5]   8.00-9.00   sec   106 MBytes   891 Mbits/sec    0   3.54 MBytes       
[  5]   9.00-10.00  sec   106 MBytes   891 Mbits/sec    0   3.54 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.03 GBytes   882 Mbits/sec    0             sender
[  5]   0.00-10.03  sec  1.03 GBytes   879 Mbits/sec                  receiver

The restore is also much faster now:
restore image complete (bytes=48318382080, duration=179.54s, speed=256.65MB/s)
Seems like this was the problem.